{
 "id": "5.1",
 "title": "Architecture",
 "department": "Engineering",
 "department_slug": "engineering",
 "direction": null,
 "branded": false,
 "status": "draft",
 "owner": "Param",
 "last_verified": "Mon Oct 26",
 "holds": "Services, web and worker roles, queue, leases, storage, diagram",
 "urls": {
  "html": "https://handbook.joinsquareup.com/engineering/architecture",
  "markdown": "https://handbook.joinsquareup.com/engineering/architecture.md",
  "json": "https://handbook.joinsquareup.com/engineering/architecture.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "One Node codebase, two production services, and exactly one paid background worker",
  "points": [
   {
    "icon": "gear",
    "text": "Production runs brew-web and brew-worker on Cloud Run over Cloud SQL Postgres."
   },
   {
    "icon": "lock",
    "text": "Only one worker holds the background lease, so paid work never runs twice."
   },
   {
    "icon": "box",
    "text": "Staging is a single instance, and the demo uses a bucket only."
   },
   {
    "icon": "chart",
    "text": "Web scales from 2 to 10 instances behind one load balancer."
   },
   {
    "icon": "warn",
    "text": "Alerts fire when the worker loses its lease or jobs wait over 10 minutes."
   }
  ],
  "stat": {
   "value": "160",
   "label": "Postgres connection alert level"
  },
  "decide": "When do we run more than one worker? Proposal: when over 3 brands wait 30 minutes.",
  "mood": "solid"
 },
 "sources": [
  "brew origin/main b91e364: server.js:460-480, 520, 735-738, 1066-1168",
  "brew origin/main: roles.js, jobs.js, db/index.js, objects.js, shared-doc.js, bus.js, presence.js, counters.js, sessions.js, build-events.js, instance.js, moved.js, retire.js",
  "brew origin/main: db/migrations/001-009",
  "brew origin/main: deploy.sh:39, tools/prod-env.sh:6-9, tools/promote.sh",
  "brew/.wt-prod/docs/production.md",
  "brew origin/main: docs/handover/engineering/01-architecture.md (state of 15 Oct, corrected here)",
  "memory brew-prod-backend, brew-storage, brew-gemini-provider, brew-pro-switch-and-flush"
 ],
 "open_questions": [
  "**When do we run more than one worker?** Proposal: stay on one worker until a build queue of more than 3 brands waits over 30 minutes, then build per-brand leases.",
  "**When does staging stop mirroring to the bucket?** Proposal: turn `OBJECTS_MIRROR` and `OBJECTS_READTHROUGH` off on 25 Nov 2026 if no rollback to the bucket happened.",
  "**Should the demo move to Postgres?** Proposal: yes, in v2, on its own small database, so the demo tests the same storage as production."
 ],
 "parts": {
  "Answer": "Brew is one Node 22 codebase that runs as two production services (web and worker) on Cloud Run over Cloud SQL Postgres, plus a single-instance staging and a bucket-only demo, and this page settles where each piece of work runs and why only one worker may do paid background work.",
  "Today": "### The shape\n\n```\n                    Cloudflare DNS (feed, brew, demo  ->  34.54.65.239)\n                                       |\n                 Google HTTPS load balancer \"squareup-urlmap\"\n          (Cloud Armor brew-prod-armor on the production backend only)\n        |                              |                               |\n feed.joinsquareup.com         brew.joinsquareup.com          demo.joinsquareup.com\n PRODUCTION                    STAGING                        DEMO (DEMO_MODE=1)\n        |                              |                               |\n Cloud Run brew-web            Cloud Run shopos-feed           Cloud Run brew-demo\n ROLE=web, 2..10 inst,         ROLE=web,worker, exactly 1      exactly 1 instance,\n session affinity,             instance, CPU always on         bucket storage only\n ingress: LB only                      |                               |\n        |  jobs (Postgres)             |                               |\n        v                              |                               |\n Cloud Run brew-worker                 |                               |\n ROLE=worker, exactly 1,               |                               |\n internal, holds the                   |                               |\n background lease                      |                               |\n        |                              |                               |\n Cloud SQL brew-prod           Cloud SQL brew-staging          gs://brew-prototype-demo\n (Postgres 17, HA,             (172.30.0.3)\n 172.30.0.4)                           |\n        |                              |\n gs://brew-prod                gs://brew-prototype\n (pictures, video, PDFs,       (binaries, plus a mirror of\n zips; versioned)              every row while it is the way back)\n\n Outside: Vertex AI (Gemini text, Gemini image, Veo, TTS, embeddings), Anthropic (switch back),\n OpenAI (image fallback, transcription), treg (SERP, Ad Library, AI answers, social, marketplaces),\n Apify (quick commerce, Indian marketplaces), Higgsfield and Sloosh (Studio and finals), Resend (team email),\n Shopify, Meta, Google Ads, GA4, Klaviyo, Judge.me, Slack (read-only connectors), headless Chromium (in the image)\n```\n\nSources for the diagram: [src: brew/.wt-prod/docs/production.md] [src: brew/tools/promote.sh:70-107] [src: brew/deploy.sh:39] [src: brew/tools/deploy-demo.sh] [src: memory brew-prod-backend]\n\n### Services\n\n| Piece | What it is | Source |\n|---|---|---|\n| brew-web | Production web: ROLE=web, 2 to 10 instances, 2 vCPU, 2 GiB, concurrency 80, session affinity, ingress internal and load balancer only, pool of 5 Postgres connections | [src: brew/tools/promote.sh:79-80] |\n| brew-worker | Production worker: ROLE=worker, exactly 1 instance, 2 vCPU, 4 GiB, ingress internal, pool of 10 | [src: brew/tools/promote.sh:94-95] |\n| shopos-feed | Staging: both roles in one instance, min 1 max 1, 2 vCPU, 4 GiB, timeout 3600 s, concurrency 200, STORAGE=pg | [src: brew/deploy.sh:39-40] |\n| brew-demo | Demo: DEMO_MODE=1, one instance, all Gemini Flash, bucket storage | [src: brew/tools/deploy-demo.sh] |\n| Runtime identity | Production runs as brew-prod-runtime; staging, the demo and ops jobs run as brew-runtime and cannot read production secrets or data | [src: brew/tools/prod-env.sh:5-6] |\n| Image | node:22-bookworm-slim with ffmpeg, chromium and fonts; five npm dependencies: @anthropic-ai/sdk, @imgly/background-removal-node, date-holidays, pg, puppeteer-core | [src: brew/Dockerfile] [src: brew/package.json] |\n\n### Roles, the lease and the queue\n\n- `ROLE` is `web`, `worker` or both (the default, used locally and on staging). [src: brew/roles.js:1-9]\n- On Postgres one worker holds the background lease: a session advisory lock (key 727275) on a connection of its own. If that connection drops, the worker exits and Cloud Run restarts it. [src: brew/roles.js:18-32]\n- Only the lease holder runs timers, resumes builds and pays for watchers. Every timer is wrapped in `leaderOnly`. [src: brew/roles.js:39] [src: brew/server.js:520]\n- A web-only instance never runs builder work itself. A JS Proxy turns every builder method, other than local reads such as `status`, `list` and `events`, into a job the worker runs. Clicks wait up to 90 s (PDFs 180 s); nudges such as `seen`, `heal` and `bookAfter` are fire and forget. [src: brew/server.js:464-480]\n- The job queue is the Postgres `jobs` table, claimed with `FOR UPDATE SKIP LOCKED`, leased for 120 s with a heartbeat, retried with backoff, at most one open job per key. Queues are `interactive`, `slow` and `batch`. Delivery is at least once, so every handler must be safe to run twice. [src: brew/jobs.js:1-8] [src: brew/db/migrations/002_jobs_counters_events.sql]\n- Without Postgres (demo, local, tests) a job runs at once in-process. [src: brew/jobs.js:7]\n\n### How instances talk\n\n| Need | Mechanism | Source |\n|---|---|---|\n| A stored object changed | `objects` trigger sends `pg_notify('objects', key|version)`; other instances drop their cache | [src: brew/db/migrations/007_objects_src_notify.sql] |\n| New cards, a connector read again | `bus` table plus NOTIFY, kept a day | [src: brew/bus.js:1-3] |\n| Build progress on any instance | `build_events` table plus NOTIFY, kept 3 days | [src: brew/build-events.js:1-4] |\n| Who is in a jam | `jam_presence` unlogged table, read every 2 s | [src: brew/presence.js:1-3] |\n| Day caps across instances | `counters` table, written within 1 s, pulled every 2 s; `take()` for strict caps | [src: brew/counters.js:1-5] |\n| Sign out everywhere | `sessions` and `session_revocations` plus NOTIFY | [src: brew/sessions.js:1-4] |\n| Two people editing one jam | `shared-doc.js`: each instance writes only the parts it changed; the other side's new list items merge in | [src: brew/shared-doc.js:1-5] |\n| One paid Studio make per board | `make_claims` rows taken in one transaction | [src: brew/jam2/make-claim.js:1-4] [src: brew/db/migrations/009_make_claims.sql] |\n\n### Request path\n\n`serve()` tries handlers in this order: the health probe, a cross-site refusal, demo routes, the Shopify app, auth, the closed-store gate, the moved-brand redirect, Sloosh, jam mail links, Slack, connectors, the team gate, memory, cadence moves, brand routes, brand files, the public Brand Book, the mic, the feed service (which mounts the jam, Studio, briefs and the Brand Book API), then `x-brand` routes, `/research/` and `/gen/` media, static files and pages. [src: brew/server.js:1110-1168]\n\nThere is no route registry: each module matches its own paths with regexes. [src: brew/docs/handover/engineering/04-api-routes.md]\n\n### Restarts and deploys\n\n- SIGTERM runs one shutdown path: no new move starts, jams and workspaces flush, and runs are left unfinished so the next instance resumes them. [src: brew/server.js:1066-1081] [src: brew/CLAUDE.md]\n- At boot the lease holder runs `builder.resumeAll()` and `finals.reconcileAll()`, so builds, rewrites and paid finals carry on and are never paid twice. [src: brew/server.js:735-738]\n- `retire.js` and `control/live.json` still keep one live revision on bucket deployments; production sets `RETIRE_OFF=1` because the lease does that job there. [src: brew/retire.js:1-10] [src: brew/tools/prod-env.sh:9]\n\n### AI providers\n\n- `ai.js` gives one call shape for Claude or Gemini; `gemini-client.js` maps Claude names to Gemini models. [src: brew/ai.js:1-3] [src: brew/gemini-client.js:1-5]\n- Today staging and production run `AI_PROVIDER=gemini`: Plays on `gemini-3.1-pro-preview`, everything else on `gemini-3.5-flash`. [src: brew/deploy.sh:39] [src: brew/tools/prod-env.sh:9] [src: brew/moves-engine.js:17]\n- Draft media: Vertex Gemini image, then OpenAI; video Veo 3.1 fast then Veo 3.1; voice Gemini TTS then OpenAI. [src: brew/media.js:1-7]",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| Web and worker split, one lease holder, Postgres queue | v1 | Stays |\n| One worker for all brands. Per-brand build leases so several workers can build at once | v2 | New |\n| A route table that declares each route's auth level, checked by one test | v2 | New |\n| Staging on the bucket mirror (`OBJECTS_MIRROR=1`, `OBJECTS_READTHROUGH=1`) as the way back | v2 | Goes, once Postgres has run a month without a rollback |\n| Demo on the bucket with one instance | v2 | Changes: move to Postgres so one storage path is tested |\n| `retire.js` and `control/live.json` | v3 | Goes, once every site is on Postgres |\n| The feed service as one 348 KB file mounting the jam and Studio | v3 | Changes: split by area (feed, jam, Studio, brief) |",
  "How we do it": "1. Anything that spends money in the background must run under `leaderOnly(...)` or `whenLeader(...)`. A web instance must never start paid work. [src: brew/roles.js:39]\n2. Anything a click needs from the builder goes through the builder proxy; add a method to `BUILD_LOCAL` only if it is a pure read. [src: brew/server.js:468]\n3. A new job handler must be safe to run twice. Paid calls carry their own idempotency keys. [src: brew/jobs.js:4-5]\n4. A new piece of shared state across instances uses Postgres (a table, NOTIFY, or `counters`), never process memory.\n5. Check what the worker is doing on production:\n\n```bash\ngcloud run jobs execute brew-ops --region=asia-south1 --wait   # runs db/status.mjs: Arata's connections, table sizes\n```",
  "Numbers": "| Metric | Definition | Where measured |\n|---|---|---|\n| Lease holder | Which worker revision holds the background lease | `db/prod-clean.mjs` `EXPECT_WORKER_REV`; alert \"worker lost its lease\" [src: brew/.wt-prod/docs/production.md] |\n| Job queue wait | Age of the oldest queued job; alert past 10 minutes | `alerts.js` [src: brew/.wt-prod/docs/production.md] |\n| Web instances | Between 2 and 10 | Cloud Run brew-web [src: brew/tools/promote.sh:80] |\n| Postgres connections | Instances times pool size; alert above 160 | Cloud Monitoring [src: brew/.wt-prod/docs/production.md] |",
  "Open": "1. **When do we run more than one worker?** Proposal: stay on one worker until a build queue of more than 3 brands waits over 30 minutes, then build per-brand leases.\n2. **When does staging stop mirroring to the bucket?** Proposal: turn `OBJECTS_MIRROR` and `OBJECTS_READTHROUGH` off on 25 Nov 2026 if no rollback to the bucket happened.\n3. **Should the demo move to Postgres?** Proposal: yes, in v2, on its own small database, so the demo tests the same storage as production."
 }
}