---
id: "5.19"
title: "Performance and reliability"
dept: engineering
status: draft
owner: Param
branded: false
last_verified: 2026-10-26
sources:
  - "memory brew-prod-backend (Phase 4 speed numbers, load test, Armor, CDN), brew-qa-round2-2026-10-24 (seed speed, feed cache), brew-qa-sweep-2026-10-19 (claims check speed)"
  - "brew/.wt-prod/docs/production.md (alerts, backups)"
  - "brew origin/main: server.js:965, 1056, 1093-1113, 464-480; db/index.js:3-14; jobs.js:13; treg.js:8-28; http-zip.js:1-6; brand-pack.js:173-190; feed-service.js:1835, 2597; counters.js:1-5; alerts.js:5-12; tools/promote.sh:79-80, 94-95, 112-118; deploy.sh:39"
---

# Performance and reliability

## Answer

This page records how fast Brew answers, what the load test showed, which caches and timeouts keep it fast and safe, and what we watch; there is no written uptime promise to partners yet, and the proposed one is in Open.

## Today

### Measured speed

| Measurement | Result | When and where | Source |
|---|---|---|---|
| Signed-in feed, cold, on a local Arata copy | 876 ms → 238 ms after Phase 4 | 25 Oct, local | [src: memory brew-prod-backend] |
| Signed-in feed, warm | 2,160 ms → 192 ms | same | same |
| Signed-in feed, repeat | 55 ms | same | same |
| Load test through the production load balancer | p50 9 ms, p95 20 ms, p99 259 ms, 0 errors at about 30 requests a second | 25 Oct, production | same |
| Brand pages before the claims-check fix | 17 to 50 s on staging; fixed by memoising the catalog normalisation | 19 Oct | [src: memory brew-qa-sweep-2026-10-19] |
| First load after a release | Still slow (noted, not measured) | 25 Oct QA on production | [src: memory brew-qa-round2-2026-10-24] |

### What keeps it fast

| Mechanism | What it does | Source |
|---|---|---|
| Pack read cache | Version-checked: a pack is reused until a file it is built from changes (`BUILT_FROM`); other files do not rebuild it | [src: brew/brand-pack.js:173-190] [src: memory brew-qa-round2-2026-10-24] |
| Change notices | Postgres NOTIFY on every object write, so instances drop only what changed | [src: brew/db/migrations/007_objects_src_notify.sql] |
| Feed cache | Only viewer, connection or Pro changes bump a workspace's revision (`STATE_SIG`); a stale rebuild waits 4 s | [src: brew/feed-service.js:1835, 2597] [src: memory brew-qa-round2-2026-10-24] |
| Warm at boot | Web instances build brand packs in memory before the startup probe says ready (at most `WARM_READY_MS`, 90 s); warm-up never starts paid work | [src: brew/server.js:1056, 1113] [src: memory brew-prod-backend] |
| Compression | Brotli or gzip on text over 1 KB, big answers compressed off the main thread | [src: brew/http-zip.js:1-6] |
| treg cache | By signal: 30 days, 7 days or 1 day; identical calls in flight share one answer; failures never cached | [src: brew/treg.js:12-27] |
| Background refresh | A refresh of research, live-watch or daily facts never makes a page wait; the page gets the pack in hand | [src: git 0fc7cfc] |
| CDN | Off on production: session affinity made it useless | [src: memory brew-prod-backend] |

### Timeouts and limits

| Setting | Value | Source |
|---|---|---|
| Cloud Run request timeout | 3,600 s on every service | [src: brew/deploy.sh:39] [src: brew/tools/promote.sh:79, 94] |
| Concurrency | 80 per production instance, 200 on staging | same |
| Startup probe | `/healthz/ready` every 3 s, 40 failures allowed | same |
| Postgres statement timeout | 30 s (`PG_STATEMENT_MS`); connect timeout 10 s | [src: brew/db/index.js:12-14] |
| Pool per instance | web 5, worker 10 (staging 10) | [src: brew/db/index.js:3-4] [src: brew/tools/promote.sh:80, 95] |
| Job lease | 120 s, heartbeat every 40 s | [src: brew/jobs.js:13, 33] |
| A click waiting on the worker | 90 s; PDFs 180 s; dropped if nobody waits any more | [src: brew/server.js:476-479] |
| Read stage deadlines | 40 s to 240 s per stage | [src: brew/reader/index.js:35] |
| Cloud Armor | 3,000 and 6,000 requests a minute rules | [src: memory brew-prod-backend] |
| Slow log | `[slow]` when an answer takes over 1.5 s (`SLOW_MS`); `[seed]` when the seed takes over 800 ms; every answer sends `Server-Timing` with total and database time | [src: brew/server.js:965, 1093-1107] |

### Reliability

- Deploys and crashes: runs resume from what was saved; paid work carries keys so it is never paid twice. [src: brew/server.js:735-738] [src: brew/CLAUDE.md]
- The worker exits the moment its lease connection drops, so two workers never run the same paid work. [src: brew/roles.js:25-30]
- Day caps are shared across instances; two instances can overshoot a cap only by what they spend inside a 2-second window. [src: brew/counters.js:1-5]
- Production database: regional HA, point-in-time recovery 7 days, daily backups kept 14, nightly export, restore test passed 25 Oct. [src: brew/.wt-prod/docs/production.md]
- Every promote rolls back by itself if checks through the load balancer fail. [src: brew/tools/promote.sh:112-118]

### What we watch

- Cloud Monitoring emails param@ and hello@: uptime checks on `/`, `/healthz/ready`, `/api/config`; server errors, many slow requests, worker lost its lease, worker not running, a release job failed, database disk over 85%, connections over 160, CPU over 80%, export failed, unhandled errors, site down. [src: brew/.wt-prod/docs/production.md]
- The app's own alerts, once a day each: Vertex billing and quota, the AI pause, treg balance under $10, a 5xx rate of 5% or more (at least 20 answers on production), Sloosh, worker down (no lease for 4 minutes), a job waiting over 10 minutes. [src: brew/alerts.js:5-12] [src: brew/tools/prod-env.sh:9] [src: brew/.wt-prod/docs/production.md]

### Uptime promise

None is written down for partners or in the Terms. [src: brew/legal-pages.js] No uptime figure for production has been measured or recorded.

## Tomorrow

| Item | Version | Tag |
|---|---|---|
| The caches, timeouts and alerts above | v1 | Stays |
| A written uptime target for design partners | v1 | New (see Open) |
| A monthly uptime number from the Cloud Monitoring checks | v1 | New |
| Faster first load after a release (warm before traffic moves) | v1 | Changes |
| A load test before each partner onboarding wave | v2 | New |
| Per-brand build leases so one brand's build never queues another's | v2 | New |

## How we do it

1. When pages feel slow, read `Server-Timing` on the answer, then search the logs:

```bash
gcloud logging read 'textPayload:"[slow]" OR textPayload:"[seed]"' --project=squareup-492617 --freshness=1h --limit=50
```

2. After any change on the pack or card path, time the seed on staging, not just the status code:

```bash
curl -s -o /dev/null -w '%{time_total}\n' https://brew.joinsquareup.com/b/cozysnugglet-shop/seed.js
```

[src: memory brew-qa-sweep-2026-10-19]

3. Before blaming the code for thin cards, check the treg balance. [src: brew/docs/handover/engineering/07-deploy-and-ops.md]
4. Repeat the restore test monthly (steps in Release and promote). [src: brew/.wt-prod/docs/production.md]

## Numbers

| Metric | Definition | Where measured |
|---|---|---|
| Feed p95 | 95th percentile server time of `GET /api/tenants/:t/feed` | `Server-Timing` and request logs (Proposed target: under 500 ms) |
| Load balancer p99 | 99th percentile at the load balancer | load test; 259 ms on 25 Oct |
| Uptime | Share of minutes the `/healthz/ready` check passes, per month | Cloud Monitoring uptime check (Proposed) |
| 5xx rate | 5xx answers over all answers in 10 minutes | `alerts.js`; alert at 5% |
| Slow answers | `[slow]` lines per hour | logs |

## Open

1. **What uptime do we promise design partners?** Proposal: 99.5% a month for the feed, measured by the `/healthz/ready` check, stated in the partner terms as a target, not a guarantee.
2. **What page speed is good enough?** Proposal: feed under 500 ms at p95 and brand page first paint under 2 s on a phone, both checked after each promote.
3. **Should we load test again before more brands join?** Proposal: yes, at 3 times the 25 Oct rate (about 90 requests a second) before the next onboarding wave.