---
id: "5.13"
title: "Release and promote"
dept: engineering
status: draft
owner: Param
branded: false
last_verified: "2026-10-26"
sources:
  - "brew origin/main: deploy.sh:1-55; tools/promote.sh:1-120; tools/prod-env.sh; db/prod-clean.mjs:1-40; tools/deploy-demo.sh; CLAUDE.md 'Release rhythm'; server.js:735-738, 1066-1081"
  - "brew/.wt-prod/docs/production.md (Releasing, Backups, Restoring, Alerts)"
  - "brew origin/main d6154b3: .github/workflows/test.yml:1-37, db/migrations/010_data_deletions.sql, docs/production.md:95-106; commits fccf3cb, d6154b3; GitHub Actions run 40286754218"
  - "memory brew-prod-backend, prod-clean-only, always-deploy-to-test, brew-pro-switch-and-flush, brew-parallel-sessions-deploy, brew-qa-sweep-2026-10-19, brew-wipe-data"
---

# Release and promote

## Answer

Code reaches staging with `./deploy.sh` any time it is tested, and reaches production only through `tools/promote.sh`, which moves staging's exact image after a backup, migrations, a smoke test and a clean gate, and rolls back by itself if any step fails.

## Today

### CI on every push

- GitHub Actions workflow `test` runs the same suite `deploy.sh` runs, `AI_MOCK=1 node --test test/*.test.js`, on every push and pull request, on Node 22 with Postgres 17 binaries. First green run: https://github.com/wulfor24/brew/actions/runs/40286754218. [src: brew/.github/workflows/test.yml:1-37] [src: GitHub Actions run 40286754218]
- It deploys nothing and `deploy.sh` does not wait for it; the deploy still runs the suite itself. [src: brew/.github/workflows/test.yml:9-37] [src: brew/deploy.sh:13-16]

### Staging deploy (`./deploy.sh`)

1. Refuses uncommitted code. [src: brew/deploy.sh:7]
2. `node --check` on the server, db, reader, connectors, pdf and public JS. [src: brew/deploy.sh:8]
3. Checks the folders the image needs exist. [src: brew/deploy.sh:10]
4. Runs the free suite; one retry after 20 s; names failing tests. [src: brew/deploy.sh:13-16]
5. `gcloud run deploy shopos-feed --source .` with the staging env and secrets, labelled `git-sha`. [src: brew/deploy.sh:39-41]
6. Keeps the newest 3 revisions; rollback is `update-traffic --to-revisions=<rev>=100`. [src: brew/deploy.sh:47-50]
7. Curls the landing, Cozy, Arata and `/api/config`. [src: brew/deploy.sh:51]

Deploys never wait for background work: SIGTERM stops new Plays, flushes jams and workspaces, and leaves runs unfinished; the next instance resumes them. Never re-add a wait. [src: brew/CLAUDE.md] [src: memory brew-pro-switch-and-flush]

### Production promote (`tools/promote.sh`)

| Step | What | Fails how | Source |
|---|---|---|---|
| 0 | Read staging's live revision, its image digest and git-sha; the sha must be on origin/main; the image must be pinned to a digest | stops, nothing changed | [src: brew/tools/promote.sh:15-24] |
| 0b | `tools/promote.sh <image@sha256:…>` promotes an earlier staging image: it must be one a staging revision ran, and production is labelled with that revision's own commit | stops | [src: git 45b6a06] |
| lock | `gs://brew-prod-backups/locks/promote.lock`, created only if absent; a lock over 2 hours old is taken over | stops | [src: brew/tools/promote.sh:28-38] |
| 1/8 backup | On-demand Cloud SQL backup, waits for that exact backup | stops | [src: brew/tools/promote.sh:54-62] |
| 2/8 migrations | Job `brew-prod-migrate` runs `db/migrate.js`; migrations only add | stops, production unchanged | [src: brew/tools/promote.sh:64-68] |
| 3/8 web | Deploys brew-web with no traffic, tag `candidate` | stops, traffic unchanged | [src: brew/tools/promote.sh:70-82] |
| 4/8 gate | Job `brew-prod-clean` from inside the VPC: smoke test plus clean gate on the candidate | stops, traffic unchanged | [src: brew/tools/promote.sh:84-90] |
| 5/8 worker | Deploys brew-worker, gives that revision the traffic by name, checks it, runs the gate on the worker's settings | rolls back | [src: brew/tools/promote.sh:92-107] |
| 6/8 traffic | Moves web traffic to the candidate by name, never "latest" | rolls back | [src: brew/tools/promote.sh:109-110] |
| 7/8 checks | `/`, `/healthz/ready`, `/api/config`, `/privacy`, `/arata-in`, `/b/arata-in/seed.js` through the load balancer, 5 tries each | rolls back | [src: brew/tools/promote.sh:112-118] |
| 8/8 | Prints the sha and revision now serving | | [src: brew/tools/promote.sh:119] |

Last promote: 26 Oct, a559de6 (the 26 Oct jam fixes, jam product truth and Higgsfield safety) went to production as brew-web-00036-sam and brew-worker-00018-wxg; the backup, clean gate and load-balancer checks passed. [src: tools/promote.sh run, Param, 26 Oct] [src: git a559de6]

A failed rollback prints `ROLLBACK FAILED` with what to restore by hand. [src: brew/tools/promote.sh:48-52]

### The clean gate

`db/prod-clean.mjs` is read only and exits 1 on any finding. It checks each service's ROLE, ingress, image, service account, that migrations do not run on boot, no demo, mock, open-site or dev sign-in settings, the production bucket and URL, production's own secrets and sign-in, which revision holds the lease, and that the database has no test brands (Cozy, Zenant, adpage, questions), test keys, archives, demo sandboxes or simulated connector data, and every built brand's page. [src: brew/db/prod-clean.mjs:1-40] [src: brew/.wt-prod/docs/production.md]

Param's kept exceptions: the Pro plan sheet without billing, Marketplaces "Coming soon", the catalog-photo fallback and Delete my data as an email request. They are code, not data. [src: memory prod-clean-only]

### Backups and restore

- Point-in-time recovery 7 days; automated backups daily, 14 kept; an on-demand backup before every promote; a nightly logical export at 21:30 UTC to `gs://brew-prod-backups/nightly/brew.sql.gz`; `gs://brew-prod` versioned with 7-day soft delete. [src: brew/.wt-prod/docs/production.md]
- The restore test passed on 25 Oct: every brand's row and part counts matched. Repeat monthly. [src: brew/.wt-prod/docs/production.md]

### Demo

Now on origin/main and not yet on production: d6154b3, the staff brief buttons, the Postgres reset and delete tools with migration 010, the customer copy cleanup and CI. Its staging deploy was running on 26 Oct. The next promote runs migration 010, which adds the `data_deletions` table; the `brew-prod-ops` job is then made once by hand. [src: git d6154b3] [src: brew/docs/production.md:95-106] [src: deploy state 26 Oct]

`tools/deploy-demo.sh` deploys brew-demo with `DEMO_MODE=1`, all Flash, one instance, and runs no tests. [src: brew/tools/deploy-demo.sh]

## Tomorrow

| Item | Version | Tag |
|---|---|---|
| Promote only by `tools/promote.sh`, same image, clean gate | v1 | Stays |
| A record of each promote (sha, revision, who, when) in the handbook changelog | v1 | New |
| Tests in the demo deploy | v1 | Changes |
| CI on every push and pull request | v1 | Stays |
| `deploy.sh` stops when the commit's CI run is red | v1 | New, proposed |
| A canary share of traffic before the full switch | v2 | New |
| Per-change release notes for partners | v2 | New |

## How we do it

1. Staging, after a tested change, from a clean tree of origin/main (see Parallel sessions):

```bash
git pull --rebase && git push origin HEAD:main
./deploy.sh
```

2. Confirm the live staging app serves your change by grepping the served `app.js` for a new identifier, not the deploy log. [src: memory brew-parallel-sessions-deploy]
3. Production, from a clean checkout of origin/main once staging runs the commit:

```bash
tools/promote.sh
```

4. Never edit `tools/promote.sh` while it runs; bash reads it as it goes. One promote at a time. [src: brew/.wt-prod/docs/production.md]
5. In zsh, write gcloud flags out in full; zsh does not word-split `$NET`. [src: memory brew-prod-backend]
6. Restore one brand: clone to a moment before the damage, copy its `objects` and `doc_parts` rows back, keep current `private/vault/*`, restore files from object versions. Never reconnect Arata. [src: brew/.wt-prod/docs/production.md]

```bash
gcloud sql instances clone brew-prod brew-prod-restore --point-in-time=<RFC3339>
```

7. Remove one-off `OPS_*` env vars after an ops job; check the live env after anyone else deploys. [src: brew/docs/handover/engineering/07-deploy-and-ops.md]

## Numbers

| Metric | Definition | Where measured |
|---|---|---|
| Promote success rate | Promotes that reach 8/8 over all started | Proposed: a log line per run in `gs://brew-prod-backups/locks/` |
| Rollbacks | Promotes that rolled back | same |
| Staging deploys per day | Revisions created on shopos-feed | `gcloud run revisions list` (about 120 in 22 hours on 16 Oct) [src: memory brew-audit-2026-10-15] |
| Restore test | Pass or fail, monthly | `brew/docs/production.md` |

## Open

1. **Who may run a promote?** Proposal: Param, or a session Param asks in chat, one at a time.
2. **Is there a release freeze during a partner session?** Proposal: no promotes during a booked partner call; staging stays free.
3. **What does production run now?** Memory says 0285592 on 25 Oct. Proposal: print the serving sha on `/api/config` so this page can read it.