Answer
Code reaches staging with ./deploy.sh any time it is tested, and reaches production only through tools/promote.sh, which moves staging's exact image after a backup, migrations, a smoke test and a clean gate, and rolls back by itself if any step fails.
Today
CI on every push
- GitHub Actions workflow
testruns the same suitedeploy.shruns,AI_MOCK=1 node --test test/*.test.js, on every push and pull request, on Node 22 with Postgres 17 binaries. First green run: https://github.com/wulfor24/brew/actions/runs/40286754218. brew/.github/workflows/test.yml:1-37 GitHub Actions run 40286754218 - It deploys nothing and
deploy.shdoes not wait for it; the deploy still runs the suite itself. brew/.github/workflows/test.yml:9-37 brew/deploy.sh:13-16
Staging deploy (./deploy.sh)
- Refuses uncommitted code. brew/deploy.sh:7
node --checkon the server, db, reader, connectors, pdf and public JS. brew/deploy.sh:8- Checks the folders the image needs exist. brew/deploy.sh:10
- Runs the free suite; one retry after 20 s; names failing tests. brew/deploy.sh:13-16
gcloud run deploy shopos-feed --source .with the staging env and secrets, labelledgit-sha. brew/deploy.sh:39-41- Keeps the newest 3 revisions; rollback is
update-traffic --to-revisions=<rev>=100. brew/deploy.sh:47-50 - Curls the landing, Cozy, Arata and
/api/config. brew/deploy.sh:51
Deploys never wait for background work: SIGTERM stops new Plays, flushes jams and workspaces, and leaves runs unfinished; the next instance resumes them. Never re-add a wait. brew/CLAUDE.md memory brew-pro-switch-and-flush
Production promote (tools/promote.sh)
| Step | What | Fails how | Source |
|---|---|---|---|
| 0 | Read staging's live revision, its image digest and git-sha; the sha must be on origin/main; the image must be pinned to a digest | stops, nothing changed | brew/tools/promote.sh:15-24 |
| 0b | tools/promote.sh <image@sha256:…> promotes an earlier staging image: it must be one a staging revision ran, and production is labelled with that revision's own commit |
stops | git 45b6a06 |
| lock | gs://brew-prod-backups/locks/promote.lock, created only if absent; a lock over 2 hours old is taken over |
stops | brew/tools/promote.sh:28-38 |
| 1/8 backup | On-demand Cloud SQL backup, waits for that exact backup | stops | brew/tools/promote.sh:54-62 |
| 2/8 migrations | Job brew-prod-migrate runs db/migrate.js; migrations only add |
stops, production unchanged | brew/tools/promote.sh:64-68 |
| 3/8 web | Deploys brew-web with no traffic, tag candidate |
stops, traffic unchanged | brew/tools/promote.sh:70-82 |
| 4/8 gate | Job brew-prod-clean from inside the VPC: smoke test plus clean gate on the candidate |
stops, traffic unchanged | brew/tools/promote.sh:84-90 |
| 5/8 worker | Deploys brew-worker, gives that revision the traffic by name, checks it, runs the gate on the worker's settings | rolls back | brew/tools/promote.sh:92-107 |
| 6/8 traffic | Moves web traffic to the candidate by name, never "latest" | rolls back | brew/tools/promote.sh:109-110 |
| 7/8 checks | /, /healthz/ready, /api/config, /privacy, /arata-in, /b/arata-in/seed.js through the load balancer, 5 tries each |
rolls back | brew/tools/promote.sh:112-118 |
| 8/8 | Prints the sha and revision now serving | brew/tools/promote.sh:119 |
Last promote: 26 Oct, a559de6 (the 26 Oct jam fixes, jam product truth and Higgsfield safety) went to production as brew-web-00036-sam and brew-worker-00018-wxg; the backup, clean gate and load-balancer checks passed. tools/promote.sh run, Param, 26 Oct git a559de6
A failed rollback prints ROLLBACK FAILED with what to restore by hand. brew/tools/promote.sh:48-52
The clean gate
db/prod-clean.mjs is read only and exits 1 on any finding. It checks each service's ROLE, ingress, image, service account, that migrations do not run on boot, no demo, mock, open-site or dev sign-in settings, the production bucket and URL, production's own secrets and sign-in, which revision holds the lease, and that the database has no test brands (Cozy, Zenant, adpage, questions), test keys, archives, demo sandboxes or simulated connector data, and every built brand's page. brew/db/prod-clean.mjs:1-40 brew/.wt-prod/docs/production.md
Param's kept exceptions: the Pro plan sheet without billing, Marketplaces "Coming soon", the catalog-photo fallback and Delete my data as an email request. They are code, not data. memory prod-clean-only
Backups and restore
- Point-in-time recovery 7 days; automated backups daily, 14 kept; an on-demand backup before every promote; a nightly logical export at 21:30 UTC to
gs://brew-prod-backups/nightly/brew.sql.gz;gs://brew-prodversioned with 7-day soft delete. brew/.wt-prod/docs/production.md - The restore test passed on 25 Oct: every brand's row and part counts matched. Repeat monthly. brew/.wt-prod/docs/production.md
Demo
Now on origin/main and not yet on production: d6154b3, the staff brief buttons, the Postgres reset and delete tools with migration 010, the customer copy cleanup and CI. Its staging deploy was running on 26 Oct. The next promote runs migration 010, which adds the data_deletions table; the brew-prod-ops job is then made once by hand. git d6154b3 brew/docs/production.md:95-106 deploy state 26 Oct
tools/deploy-demo.sh deploys brew-demo with DEMO_MODE=1, all Flash, one instance, and runs no tests. brew/tools/deploy-demo.sh
Tomorrow
| Item | Version | Tag |
|---|---|---|
Promote only by tools/promote.sh, same image, clean gate |
v1 | Stays |
| A record of each promote (sha, revision, who, when) in the handbook changelog | v1 | New |
| Tests in the demo deploy | v1 | Changes |
| CI on every push and pull request | v1 | Stays |
deploy.sh stops when the commit's CI run is red |
v1 | New, proposed |
| A canary share of traffic before the full switch | v2 | New |
| Per-change release notes for partners | v2 | New |
How we do it
- Staging, after a tested change, from a clean tree of origin/main (see Parallel sessions):
git pull --rebase && git push origin HEAD:main
./deploy.sh
- Confirm the live staging app serves your change by grepping the served
app.jsfor a new identifier, not the deploy log. memory brew-parallel-sessions-deploy - Production, from a clean checkout of origin/main once staging runs the commit:
tools/promote.sh
- Never edit
tools/promote.shwhile it runs; bash reads it as it goes. One promote at a time. brew/.wt-prod/docs/production.md - In zsh, write gcloud flags out in full; zsh does not word-split
$NET. memory brew-prod-backend - Restore one brand: clone to a moment before the damage, copy its
objectsanddoc_partsrows back, keep currentprivate/vault/*, restore files from object versions. Never reconnect Arata. brew/.wt-prod/docs/production.md
gcloud sql instances clone brew-prod brew-prod-restore --point-in-time=<RFC3339>
- Remove one-off
OPS_*env vars after an ops job; check the live env after anyone else deploys. brew/docs/handover/engineering/07-deploy-and-ops.md
Numbers
| Metric | Definition | Where measured |
|---|---|---|
| Promote success rate | Promotes that reach 8/8 over all started | Proposed: a log line per run in gs://brew-prod-backups/locks/ |
| Rollbacks | Promotes that rolled back | same |
| Staging deploys per day | Revisions created on shopos-feed | gcloud run revisions list (about 120 in 22 hours on 16 Oct) memory brew-audit-2026-10-15 |
| Restore test | Pass or fail, monthly | brew/docs/production.md |
Open
- Who may run a promote? Proposal: Param, or a session Param asks in chat, one at a time.
- Is there a release freeze during a partner session? Proposal: no promotes during a booked partner call; staging stays free.
- What does production run now? Memory says 0285592 on 25 Oct. Proposal: print the serving sha on
/api/configso this page can read it.