This page records how fast Brew answers, what the load test showed, which caches and timeouts keep it fast and safe, and what we watch; there is no written uptime promise to partners yet, and the proposed one is in Open.
Today
Measured speed
Measurement
Result
When and where
Source
Signed-in feed, cold, on a local Arata copy
876 ms → 238 ms after Phase 4
25 Oct, local
memory brew-prod-backend
Signed-in feed, warm
2,160 ms → 192 ms
same
same
Signed-in feed, repeat
55 ms
same
same
Load test through the production load balancer
p50 9 ms, p95 20 ms, p99 259 ms, 0 errors at about 30 requests a second
25 Oct, production
same
Brand pages before the claims-check fix
17 to 50 s on staging; fixed by memoising the catalog normalisation
19 Oct
memory brew-qa-sweep-2026-10-19
First load after a release
Still slow (noted, not measured)
25 Oct QA on production
memory brew-qa-round2-2026-10-24
What keeps it fast
Mechanism
What it does
Source
Pack read cache
Version-checked: a pack is reused until a file it is built from changes (BUILT_FROM); other files do not rebuild it
Web instances build brand packs in memory before the startup probe says ready (at most WARM_READY_MS, 90 s); warm-up never starts paid work
brew/server.js:1056, 1113memory brew-prod-backend
Compression
Brotli or gzip on text over 1 KB, big answers compressed off the main thread
brew/http-zip.js:1-6
treg cache
By signal: 30 days, 7 days or 1 day; identical calls in flight share one answer; failures never cached
brew/treg.js:12-27
Background refresh
A refresh of research, live-watch or daily facts never makes a page wait; the page gets the pack in hand
git 0fc7cfc
CDN
Off on production: session affinity made it useless
memory brew-prod-backend
Timeouts and limits
Setting
Value
Source
Cloud Run request timeout
3,600 s on every service
brew/deploy.sh:39brew/tools/promote.sh:79, 94
Concurrency
80 per production instance, 200 on staging
same
Startup probe
/healthz/ready every 3 s, 40 failures allowed
same
Postgres statement timeout
30 s (PG_STATEMENT_MS); connect timeout 10 s
brew/db/index.js:12-14
Pool per instance
web 5, worker 10 (staging 10)
brew/db/index.js:3-4brew/tools/promote.sh:80, 95
Job lease
120 s, heartbeat every 40 s
brew/jobs.js:13, 33
A click waiting on the worker
90 s; PDFs 180 s; dropped if nobody waits any more
brew/server.js:476-479
Read stage deadlines
40 s to 240 s per stage
brew/reader/index.js:35
Cloud Armor
3,000 and 6,000 requests a minute rules
memory brew-prod-backend
Slow log
[slow] when an answer takes over 1.5 s (SLOW_MS); [seed] when the seed takes over 800 ms; every answer sends Server-Timing with total and database time
brew/server.js:965, 1093-1107
Reliability
Deploys and crashes: runs resume from what was saved; paid work carries keys so it is never paid twice. brew/server.js:735-738brew/CLAUDE.md
The worker exits the moment its lease connection drops, so two workers never run the same paid work. brew/roles.js:25-30
Day caps are shared across instances; two instances can overshoot a cap only by what they spend inside a 2-second window. brew/counters.js:1-5
Production database: regional HA, point-in-time recovery 7 days, daily backups kept 14, nightly export, restore test passed 25 Oct. brew/.wt-prod/docs/production.md
Every promote rolls back by itself if checks through the load balancer fail. brew/tools/promote.sh:112-118
What we watch
Cloud Monitoring emails param@ and hello@: uptime checks on /, /healthz/ready, /api/config; server errors, many slow requests, worker lost its lease, worker not running, a release job failed, database disk over 85%, connections over 160, CPU over 80%, export failed, unhandled errors, site down. brew/.wt-prod/docs/production.md
The app's own alerts, once a day each: Vertex billing and quota, the AI pause, treg balance under $10, a 5xx rate of 5% or more (at least 20 answers on production), Sloosh, worker down (no lease for 4 minutes), a job waiting over 10 minutes. brew/alerts.js:5-12brew/tools/prod-env.sh:9brew/.wt-prod/docs/production.md
Uptime promise
None is written down for partners or in the Terms. brew/legal-pages.js No uptime figure for production has been measured or recorded.
Tomorrow
Item
Version
Tag
The caches, timeouts and alerts above
v1
Stays
A written uptime target for design partners
v1
New (see Open)
A monthly uptime number from the Cloud Monitoring checks
v1
New
Faster first load after a release (warm before traffic moves)
v1
Changes
A load test before each partner onboarding wave
v2
New
Per-brand build leases so one brand's build never queues another's
v2
New
How we do it
When pages feel slow, read Server-Timing on the answer, then search the logs:
gcloud logging read 'textPayload:"[slow]" OR textPayload:"[seed]"' --project=squareup-492617 --freshness=1h --limit=50
After any change on the pack or card path, time the seed on staging, not just the status code:
Before blaming the code for thin cards, check the treg balance. brew/docs/handover/engineering/07-deploy-and-ops.md
Repeat the restore test monthly (steps in Release and promote). brew/.wt-prod/docs/production.md
Numbers
Metric
Definition
Where measured
Feed p95
95th percentile server time of GET /api/tenants/:t/feed
Server-Timing and request logs (Proposed target: under 500 ms)
Load balancer p99
99th percentile at the load balancer
load test; 259 ms on 25 Oct
Uptime
Share of minutes the /healthz/ready check passes, per month
Cloud Monitoring uptime check (Proposed)
5xx rate
5xx answers over all answers in 10 minutes
alerts.js; alert at 5%
Slow answers
[slow] lines per hour
logs
Open
What uptime do we promise design partners? Proposal: 99.5% a month for the feed, measured by the /healthz/ready check, stated in the partner terms as a target, not a guarantee.
What page speed is good enough? Proposal: feed under 500 ms at p95 and brand page first paint under 2 s on a phone, both checked after each promote.
Should we load test again before more brands join? Proposal: yes, at 3 times the 25 Oct rate (about 90 requests a second) before the next onboarding wave.
