{
 "id": "5.19",
 "title": "Performance and reliability",
 "department": "Engineering",
 "department_slug": "engineering",
 "direction": null,
 "branded": false,
 "status": "draft",
 "owner": "Param",
 "last_verified": "Mon Oct 26",
 "holds": "Page speed, load test results, caches, timeouts, what we promise on uptime",
 "urls": {
  "html": "https://handbook.joinsquareup.com/engineering/performance-and-reliability",
  "markdown": "https://handbook.joinsquareup.com/engineering/performance-and-reliability.md",
  "json": "https://handbook.joinsquareup.com/engineering/performance-and-reliability.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "Fast today, with no written uptime promise to partners yet",
  "points": [
   {
    "icon": "chart",
    "text": "Load test on production: p50 9 ms, p95 20 ms, p99 259 ms, 0 errors."
   },
   {
    "icon": "rocket",
    "text": "The signed in feed went from 2,160 ms warm to 192 ms."
   },
   {
    "icon": "gear",
    "text": "Caches, change notices and a boot warm up keep pages quick."
   },
   {
    "icon": "warn",
    "text": "First load after a release is still slow, noted and not measured."
   },
   {
    "icon": "clock",
    "text": "Proposed target: feed under 500 ms at p95."
   }
  ],
  "stat": {
   "value": "259",
   "label": "p99 milliseconds under load"
  },
  "decide": "What uptime do we promise partners? Proposal: 99.5% a month, as a target, not a guarantee.",
  "mood": "waiting"
 },
 "sources": [
  "memory brew-prod-backend (Phase 4 speed numbers, load test, Armor, CDN), brew-qa-round2-2026-10-24 (seed speed, feed cache), brew-qa-sweep-2026-10-19 (claims check speed)",
  "brew/.wt-prod/docs/production.md (alerts, backups)",
  "brew origin/main: server.js:965, 1056, 1093-1113, 464-480; db/index.js:3-14; jobs.js:13; treg.js:8-28; http-zip.js:1-6; brand-pack.js:173-190; feed-service.js:1835, 2597; counters.js:1-5; alerts.js:5-12; tools/promote.sh:79-80, 94-95, 112-118; deploy.sh:39"
 ],
 "open_questions": [
  "**What uptime do we promise design partners?** Proposal: 99.5% a month for the feed, measured by the `/healthz/ready` check, stated in the partner terms as a target, not a guarantee.",
  "**What page speed is good enough?** Proposal: feed under 500 ms at p95 and brand page first paint under 2 s on a phone, both checked after each promote.",
  "**Should we load test again before more brands join?** Proposal: yes, at 3 times the 25 Oct rate (about 90 requests a second) before the next onboarding wave."
 ],
 "parts": {
  "Answer": "This page records how fast Brew answers, what the load test showed, which caches and timeouts keep it fast and safe, and what we watch; there is no written uptime promise to partners yet, and the proposed one is in Open.",
  "Today": "### Measured speed\n\n| Measurement | Result | When and where | Source |\n|---|---|---|---|\n| Signed-in feed, cold, on a local Arata copy | 876 ms → 238 ms after Phase 4 | 25 Oct, local | [src: memory brew-prod-backend] |\n| Signed-in feed, warm | 2,160 ms → 192 ms | same | same |\n| Signed-in feed, repeat | 55 ms | same | same |\n| Load test through the production load balancer | p50 9 ms, p95 20 ms, p99 259 ms, 0 errors at about 30 requests a second | 25 Oct, production | same |\n| Brand pages before the claims-check fix | 17 to 50 s on staging; fixed by memoising the catalog normalisation | 19 Oct | [src: memory brew-qa-sweep-2026-10-19] |\n| First load after a release | Still slow (noted, not measured) | 25 Oct QA on production | [src: memory brew-qa-round2-2026-10-24] |\n\n### What keeps it fast\n\n| Mechanism | What it does | Source |\n|---|---|---|\n| Pack read cache | Version-checked: a pack is reused until a file it is built from changes (`BUILT_FROM`); other files do not rebuild it | [src: brew/brand-pack.js:173-190] [src: memory brew-qa-round2-2026-10-24] |\n| Change notices | Postgres NOTIFY on every object write, so instances drop only what changed | [src: brew/db/migrations/007_objects_src_notify.sql] |\n| Feed cache | Only viewer, connection or Pro changes bump a workspace's revision (`STATE_SIG`); a stale rebuild waits 4 s | [src: brew/feed-service.js:1835, 2597] [src: memory brew-qa-round2-2026-10-24] |\n| Warm at boot | Web instances build brand packs in memory before the startup probe says ready (at most `WARM_READY_MS`, 90 s); warm-up never starts paid work | [src: brew/server.js:1056, 1113] [src: memory brew-prod-backend] |\n| Compression | Brotli or gzip on text over 1 KB, big answers compressed off the main thread | [src: brew/http-zip.js:1-6] |\n| treg cache | By signal: 30 days, 7 days or 1 day; identical calls in flight share one answer; failures never cached | [src: brew/treg.js:12-27] |\n| Background refresh | A refresh of research, live-watch or daily facts never makes a page wait; the page gets the pack in hand | [src: git 0fc7cfc] |\n| CDN | Off on production: session affinity made it useless | [src: memory brew-prod-backend] |\n\n### Timeouts and limits\n\n| Setting | Value | Source |\n|---|---|---|\n| Cloud Run request timeout | 3,600 s on every service | [src: brew/deploy.sh:39] [src: brew/tools/promote.sh:79, 94] |\n| Concurrency | 80 per production instance, 200 on staging | same |\n| Startup probe | `/healthz/ready` every 3 s, 40 failures allowed | same |\n| Postgres statement timeout | 30 s (`PG_STATEMENT_MS`); connect timeout 10 s | [src: brew/db/index.js:12-14] |\n| Pool per instance | web 5, worker 10 (staging 10) | [src: brew/db/index.js:3-4] [src: brew/tools/promote.sh:80, 95] |\n| Job lease | 120 s, heartbeat every 40 s | [src: brew/jobs.js:13, 33] |\n| A click waiting on the worker | 90 s; PDFs 180 s; dropped if nobody waits any more | [src: brew/server.js:476-479] |\n| Read stage deadlines | 40 s to 240 s per stage | [src: brew/reader/index.js:35] |\n| Cloud Armor | 3,000 and 6,000 requests a minute rules | [src: memory brew-prod-backend] |\n| Slow log | `[slow]` when an answer takes over 1.5 s (`SLOW_MS`); `[seed]` when the seed takes over 800 ms; every answer sends `Server-Timing` with total and database time | [src: brew/server.js:965, 1093-1107] |\n\n### Reliability\n\n- Deploys and crashes: runs resume from what was saved; paid work carries keys so it is never paid twice. [src: brew/server.js:735-738] [src: brew/CLAUDE.md]\n- The worker exits the moment its lease connection drops, so two workers never run the same paid work. [src: brew/roles.js:25-30]\n- Day caps are shared across instances; two instances can overshoot a cap only by what they spend inside a 2-second window. [src: brew/counters.js:1-5]\n- Production database: regional HA, point-in-time recovery 7 days, daily backups kept 14, nightly export, restore test passed 25 Oct. [src: brew/.wt-prod/docs/production.md]\n- Every promote rolls back by itself if checks through the load balancer fail. [src: brew/tools/promote.sh:112-118]\n\n### What we watch\n\n- Cloud Monitoring emails param@ and hello@: uptime checks on `/`, `/healthz/ready`, `/api/config`; server errors, many slow requests, worker lost its lease, worker not running, a release job failed, database disk over 85%, connections over 160, CPU over 80%, export failed, unhandled errors, site down. [src: brew/.wt-prod/docs/production.md]\n- The app's own alerts, once a day each: Vertex billing and quota, the AI pause, treg balance under $10, a 5xx rate of 5% or more (at least 20 answers on production), Sloosh, worker down (no lease for 4 minutes), a job waiting over 10 minutes. [src: brew/alerts.js:5-12] [src: brew/tools/prod-env.sh:9] [src: brew/.wt-prod/docs/production.md]\n\n### Uptime promise\n\nNone is written down for partners or in the Terms. [src: brew/legal-pages.js] No uptime figure for production has been measured or recorded.",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| The caches, timeouts and alerts above | v1 | Stays |\n| A written uptime target for design partners | v1 | New (see Open) |\n| A monthly uptime number from the Cloud Monitoring checks | v1 | New |\n| Faster first load after a release (warm before traffic moves) | v1 | Changes |\n| A load test before each partner onboarding wave | v2 | New |\n| Per-brand build leases so one brand's build never queues another's | v2 | New |",
  "How we do it": "1. When pages feel slow, read `Server-Timing` on the answer, then search the logs:\n\n```bash\ngcloud logging read 'textPayload:\"[slow]\" OR textPayload:\"[seed]\"' --project=squareup-492617 --freshness=1h --limit=50\n```\n\n2. After any change on the pack or card path, time the seed on staging, not just the status code:\n\n```bash\ncurl -s -o /dev/null -w '%{time_total}\\n' https://brew.joinsquareup.com/b/cozysnugglet-shop/seed.js\n```\n\n[src: memory brew-qa-sweep-2026-10-19]\n\n3. Before blaming the code for thin cards, check the treg balance. [src: brew/docs/handover/engineering/07-deploy-and-ops.md]\n4. Repeat the restore test monthly (steps in Release and promote). [src: brew/.wt-prod/docs/production.md]",
  "Numbers": "| Metric | Definition | Where measured |\n|---|---|---|\n| Feed p95 | 95th percentile server time of `GET /api/tenants/:t/feed` | `Server-Timing` and request logs (Proposed target: under 500 ms) |\n| Load balancer p99 | 99th percentile at the load balancer | load test; 259 ms on 25 Oct |\n| Uptime | Share of minutes the `/healthz/ready` check passes, per month | Cloud Monitoring uptime check (Proposed) |\n| 5xx rate | 5xx answers over all answers in 10 minutes | `alerts.js`; alert at 5% |\n| Slow answers | `[slow]` lines per hour | logs |",
  "Open": "1. **What uptime do we promise design partners?** Proposal: 99.5% a month for the feed, measured by the `/healthz/ready` check, stated in the partner terms as a target, not a guarantee.\n2. **What page speed is good enough?** Proposal: feed under 500 ms at p95 and brand page first paint under 2 s on a phone, both checked after each promote.\n3. **Should we load test again before more brands join?** Proposal: yes, at 3 times the 25 Oct rate (about 90 requests a second) before the next onboarding wave."
 }
}