Snapshots

Staging takes every tested change; production only by promote

Waiting on you
  • ./deploy.sh puts a tested change on staging right away, without asking.
  • tools/promote.sh ships staging's exact image after a backup and a clean gate.
  • If anything fails after traffic changes, promote puts production back by itself.
  • Only one promote at a time; a lock older than 2 hours is taken over.
  • Never edit promote.sh while it runs; that once caused a rollback.
8Steps printed during a promote
Your call

How often should production be promoted, and should staging smoke checks fail a deploy?

Ops and infrastructure · 8.2

Deploy and promote

DraftLast checked Mon Oct 26Owner Param

Answer

Code goes to staging with ./deploy.sh the moment it is tested, and to production only with tools/promote.sh, which ships staging's exact image after a backup and a clean gate and puts production back by itself if anything fails; this page is the operator's checklist for both.

Today

Staging deploy Production promote Demo deploy
Command ./deploy.sh tools/promote.sh tools/deploy-demo.sh brew/deploy.sh:7-53 brew/tools/promote.sh:16-119 brew/tools/deploy-demo.sh:9-26
Builds an image yes, --source . no, reuses staging's image by digest yes brew/deploy.sh:7-53 brew/tools/promote.sh:16-119 brew/tools/deploy-demo.sh:9-26
Refuses a dirty tree yes no tree check; refuses a staging sha not on origin/main yes brew/deploy.sh:7-53 brew/tools/promote.sh:16-119 brew/tools/deploy-demo.sh:9-26
Runs the test suite yes, one retry after 20 s no, it relies on staging no brew/deploy.sh:7-53 brew/tools/promote.sh:16-119 brew/tools/deploy-demo.sh:9-26
Backup first no on-demand Cloud SQL backup, waited for no brew/deploy.sh:7-53 brew/tools/promote.sh:16-119 brew/tools/deploy-demo.sh:9-26
Gate none db/prod-clean.mjs on the candidate and the worker none brew/deploy.sh:7-53 brew/tools/promote.sh:16-119 brew/tools/deploy-demo.sh:9-26
Rollback by hand, newest 3 revisions kept by itself on any failure after a change by hand, newest 3 kept brew/deploy.sh:7-53 brew/tools/promote.sh:16-119 brew/tools/deploy-demo.sh:9-26
Smoke checks curls 4 staging pages, never fails the deploy 6 pages through the load balancer, 5 tries each none brew/deploy.sh:7-53 brew/tools/promote.sh:16-119 brew/tools/deploy-demo.sh:9-26

Sources: brew/deploy.sh:7-16 brew/deploy.sh:39-53 brew/tools/promote.sh:16-22 brew/tools/promote.sh:54-119 brew/tools/deploy-demo.sh:9-26

Rules in force

  • Deploy after every tested change without asking; Param tests on the live staging URL. Still ask before anything that spends money or touches real ads or stores. memory always-deploy-to-test
  • A deploy never waits for background work. The old instance stops starting Plays and saves; the new one resumes. Never add a wait back. brew/CLAUDE.md memory brew-pro-switch-and-flush
  • Nothing dummy or from our testing reaches production; the clean gate refuses a promote on any finding. memory prod-clean-only brew/db/prod-clean.mjs:1-6
  • One promote at a time, through the lock gs://brew-prod-backups/locks/promote.lock; a lock older than 2 hours is taken over. brew/tools/promote.sh:25-36
  • Never edit promote.sh while it runs; bash reads it as it goes and that once caused a rollback. brew/.wt-prod/docs/production.md:33 memory brew-prod-backend
  • After a staging deploy that ran on the bucket, run the brew-import job. brew/deploy.sh:30-32 memory brew-storage
  • Model choice is set by environment at deploy time: staging defaults to Gemini, Pro for Plays and Flash for the rest. brew/deploy.sh:23-29 brew/deploy.sh:41

Tomorrow

Item Version Tag
Staging on every tested change, production only by promote.sh v1 Stays
Staging smoke checks fail the deploy when a page is not 200 v1 Changes
A short release note per promote, written into the handbook changelog v1 New
A second person able to promote v2 New, after onboarding

How we do it

Staging

  1. Commit only your lines, push, then deploy from a clean worktree of origin/main. memory brew-parallel-sessions-deploy
cd /Users/param/Documents/ChatGPT/brew
git pull --rebase && git push origin HEAD:main
git worktree add --detach "$TMPDIR/brew-deploy" origin/main
ln -s /Users/param/Documents/ChatGPT/brew/node_modules "$TMPDIR/brew-deploy/node_modules"
cd "$TMPDIR/brew-deploy" && BRANDS_DIR=/Users/param/Documents/ChatGPT/brew/.brands ./deploy.sh
  1. Confirm the live site serves your commit: grep the served app.js for a new identifier and check the revision's git-sha label descends from your commit. memory brew-qa-sweep-2026-10-19
  2. After a change on the pack or card path, time the seed, not just the status code:
curl -s -o /dev/null -w '%{time_total}\n' https://brew.joinsquareup.com/b/cozysnugglet-shop/seed.js
  1. Roll back by hand if needed:
gcloud run services update-traffic shopos-feed --to-revisions=<rev>=100 --region=asia-south1

Production

  1. Staging must run the commit you want, built from origin/main, and have been tested there.
  2. From a clean checkout of origin/main:
tools/promote.sh
  1. Read the eight steps as they print. Any STOP: line means production traffic did not move or was put back. ROLLBACK FAILED means put the named revision back by hand with update-traffic. brew/tools/promote.sh:49-52
  2. If the lock is held and no promote is running: gcloud storage rm gs://brew-prod-backups/locks/promote.lock. brew/tools/promote.sh:32
  3. After the promote, open feed.joinsquareup.com and one partner brand signed in.

Full step detail lives in Release and promote.

Numbers

Metric Definition Where measured
Promotes per week Runs of promote.sh that reach 8/8 promote output; Proposed: log each in the changelog
Promote rollbacks Runs that print "rolling back" same, Proposed
Staging deploys per day New shopos-feed revisions gcloud run revisions list, Proposed
Time from staging to production Hours between a commit's staging deploy and its promote Proposed

Open

  1. How often should production be promoted? No rhythm is written down; Next carries one pending promote of the 26 Oct jam fixes. Proposal: promote at most once a day, after Param has walked Arata and one other partner on staging.
  2. Should staging smoke checks fail the deploy? Today they only print codes. Proposal: yes, fail on any non-200 from / or /api/config.
Sources
  • brew origin/main a559de6: deploy.sh:5-53, tools/promote.sh:1-119, tools/prod-env.sh:1-13, tools/deploy-demo.sh:7-26, db/prod-clean.mjs:1-6, CLAUDE.md 'Release rhythm'
  • brew/.wt-prod/docs/production.md:23-33
  • memory always-deploy-to-test, prod-clean-only, brew-parallel-sessions-deploy, brew-qa-sweep-2026-10-19, brew-prod-backend, brew-pro-switch-and-flush, brew-storage
  • handbook content/engineering/release-and-promote.md, content/engineering/parallel-sessions.md

Source file content/ops-and-infrastructure/deploy-and-promote.md · For agents: Markdown · JSON