{
 "id": "8.3",
 "title": "Restore",
 "department": "Ops and infrastructure",
 "department_slug": "ops-and-infrastructure",
 "direction": null,
 "branded": false,
 "status": "draft",
 "owner": "Param",
 "last_verified": "2026-10-26",
 "holds": "Backups, restore steps, the restore test",
 "urls": {
  "html": "https://handbook.joinsquareup.com/ops-and-infrastructure/restore",
  "markdown": "https://handbook.joinsquareup.com/ops-and-infrastructure/restore.md",
  "json": "https://handbook.joinsquareup.com/ops-and-infrastructure/restore.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "Backups exist for every site, so a restore is a routine",
  "points": [
   {
    "icon": "clock",
    "text": "Production keeps point-in-time recovery for 7 days and 14 daily backups."
   },
   {
    "icon": "box",
    "text": "A nightly export at 21:30 UTC lands in a separate bucket that survives deleting the instance."
   },
   {
    "icon": "check",
    "text": "The 25 Oct restore test passed: every brand's row counts matched live production."
   },
   {
    "icon": "warn",
    "text": "After a restore run the connector ping, and never reconnect Arata."
   },
   {
    "icon": "flag",
    "text": "The monthly test is due, and no later run is recorded."
   }
  ],
  "stat": {
   "value": "7",
   "label": "Days of point-in-time recovery"
  },
  "decide": "Who runs the monthly restore test, and on which day?",
  "mood": "waiting"
 },
 "sources": [
  "brew/.wt-prod/docs/production.md:35-64",
  "brew origin/main d6154b3: tools/promote.sh:54-62, db/connector-ping.mjs:1-24, db/partner-copy.js:1-11, db/wipe-cli.js:18-20, db/brand-wipe.js:236-238, docs/production.md:108-110, db/reset-jams.mjs:4",
  "memory brew-storage, brew-prod-backend, never-disconnect-arata-meta"
 ],
 "open_questions": [
  "**Who runs the monthly restore test, and on which day?** Proposal: Param, on the first Monday of each month, starting 2 Nov 2026, with Claude doing the steps and Param checking the counts.",
  "**Is staging data worth backing up?** The partners now live on production. Proposal: treat staging as rebuildable, keep only the bucket versioning, and say so here.",
  "**Is 7 days of point-in-time recovery enough?** Proposal: keep 7 days during the design partnership and revisit when the first partner pays."
 ],
 "parts": {
  "Answer": "This page says what backups exist for each site, how to bring back the whole production database, one brand or one file, and how the monthly restore test is run, so a restore is a known routine rather than a first attempt under pressure.",
  "Today": "### What is kept\n\n| Site | Kind | Kept for | Source |\n|---|---|---|---|\n| Production database | Point-in-time recovery | 7 days, to the second, into a new instance | [src: brew/.wt-prod/docs/production.md:37] |\n| Production database | Automated backups | daily, 14 kept | [src: brew/.wt-prod/docs/production.md:38] |\n| Production database | On-demand backup before every promote | until deleted | [src: brew/tools/promote.sh:54-62] |\n| Production database | Nightly logical export, Cloud Scheduler `brew-prod-nightly-export` at 21:30 UTC, to `gs://brew-prod-backups/nightly/brew.sql.gz` | separate bucket, survives deleting the instance | [src: brew/.wt-prod/docs/production.md:40] |\n| Production files | `gs://brew-prod` versioning plus 7-day soft delete | old versions until removed | [src: brew/.wt-prod/docs/production.md:41] |\n| Staging files | `gs://brew-prototype` versioning; noncurrent versions deleted after 30 days | 30 days | [src: memory brew-storage] |\n| Staging records | Every row also mirrored to the bucket while `OBJECTS_MIRROR=1` | as the bucket | [src: brew/deploy.sh:30-32] |\n| Staging jam and brand resets | A copy in `_deleted/jams-<time>/` or `_deleted/brand-<slug>-<time>/` unless `--no-archive` | until removed | [src: brew/db/reset-jams.mjs:4] [src: brew/db/wipe-cli.js:18-19] |\n| Production brand reset or deletion | No `_deleted/` copy; the database backups hold the brand until they age out | 7 days of logs, 14 daily backups | [src: brew/db/wipe-cli.js:18-20] [src: brew/db/brand-wipe.js:236-238] |\n\n- Staging's Cloud SQL backup settings are not written down anywhere we read. [src: brew/.wt-prod/docs/production.md:11]\n- The demo has no backups recorded; its brands come again from staging with `tools/demo-sync.sh`. [src: brew/tools/demo-sync.sh:1-7]\n\n### The restore test\n\n- 25 Oct 2026: the export was imported into a scratch database on `brew-staging`; every brand's row and part counts matched live production. [src: brew/.wt-prod/docs/production.md:43]\n- It is to be repeated monthly. No later run is recorded. [src: brew/.wt-prod/docs/production.md:43]\n\n### Sign-ins after a restore\n\nA whole-database restore also brings back old sign-ins. Arata's tokens do not rotate, so this is safe for Arata, but `db/connector-ping.mjs` must be run afterwards, and Arata is never reconnected. [src: brew/.wt-prod/docs/production.md:50] [src: memory never-disconnect-arata-meta]",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| Point-in-time recovery, daily backups, nightly export, versioned files | v1 | Stays |\n| Monthly restore test, logged with its date and result | v1 | New |\n| A one-brand restore job written and tested, from `db/partner-copy.js` | v1 | New |\n| Staging backups written down, or staging declared rebuildable | v1 | New |\n| Point-in-time window longer than 7 days once partners pay | v2 | Changes, Param's call |",
  "How we do it": "### The whole production database to a moment\n\n1. Clone, about 20 to 40 minutes:\n\n```bash\ngcloud sql instances clone brew-prod brew-prod-restore --point-in-time=<RFC3339>\n```\n\n2. Check the clone. Prefer copying the damaged rows back. Only if that is not possible, point `BREW_PROD_DATABASE_URL` at the clone and promote.\n3. Run `db/connector-ping.mjs arata-in` through the `brew-prod-ping` job. Never reconnect Arata.\n\n[src: brew/.wt-prod/docs/production.md:47-50]\n\n### One brand\n\n1. Clone to a moment before the damage, as above.\n2. Copy that brand's `objects` and `doc_parts` rows from the clone with a small one-off SQL job, modelled on `db/partner-copy.js` minus its rule that keeps production's own rows. Keep the current `private/vault/*` rows unless told otherwise.\n3. Bring the brand's files in `gs://brew-prod/brands/<slug>/` back from object versions.\n\n[src: brew/.wt-prod/docs/production.md:52-55]\n\n### Files only\n\n```bash\ngcloud storage ls -a gs://brew-prod/brands/<slug>/<path>\ngcloud storage cp gs://brew-prod/brands/<slug>/<path>#<generation> gs://brew-prod/brands/<slug>/<path>\n```\n\n[src: brew/.wt-prod/docs/production.md:57]\n\n### Monthly restore test\n\n1. `gcloud scheduler jobs run brew-prod-nightly-export`, then wait for the EXPORT operation.\n2. Give the `brew-staging` Cloud SQL service account read access on `gs://brew-prod-backups` for the duration.\n3. `gcloud sql databases create restore_drill --instance=brew-staging`\n4. `gcloud sql import sql brew-staging gs://brew-prod-backups/nightly/brew.sql.gz --database=restore_drill`\n5. Count rows per brand in both databases with a one-off job inside the VPC, compare, then drop `restore_drill` and remove the temporary access.\n6. Write the date and result in Numbers below.\n\n[src: brew/.wt-prod/docs/production.md:59-64]",
  "Numbers": "| Metric | Definition | Where measured |\n|---|---|---|\n| Last restore test | Date and pass or fail of the monthly test | This page; 25 Oct 2026, passed [src: brew/.wt-prod/docs/production.md:43] |\n| Restore time | Minutes from clone start to a checked clone | Proposed; production.md estimates 20 to 40 |\n| Nightly export age | Hours since the last successful export | Cloud Monitoring alert \"export failed\" [src: brew/.wt-prod/docs/production.md:68] |",
  "Open": "1. **Who runs the monthly restore test, and on which day?** Proposal: Param, on the first Monday of each month, starting 2 Nov 2026, with Claude doing the steps and Param checking the counts.\n2. **Is staging data worth backing up?** The partners now live on production. Proposal: treat staging as rebuildable, keep only the bucket versioning, and say so here.\n3. **Is 7 days of point-in-time recovery enough?** Proposal: keep 7 days during the design partnership and revisit when the first partner pays."
 }
}