Answer
Two layers watch Brew, Cloud Monitoring from outside and the app's own alerts from inside, and both email param@ and hello@; this page lists every check, what it means when it fires and the first thing to do, and leaves who answers out of hours to Param.
Today
Cloud Monitoring, production
| Check | Kind | Source |
|---|---|---|
/, /healthz/ready, /api/config |
Uptime checks | brew/.wt-prod/docs/production.md:68 |
| Server errors, many slow requests, unhandled errors, site down | Alert policies, partly from log metrics [slow] and [unhandled] |
brew/.wt-prod/docs/production.md:68 memory brew-prod-backend |
| Worker lost its lease, worker not running | Alert policies | same |
| A release job failed | Alert policy on brew-prod-migrate and brew-prod-clean |
same |
| Database disk over 85%, connections over 160, CPU over 80% | Alert policies | brew/.wt-prod/docs/production.md:68 |
| Nightly export failed | Alert policy | same |
| Recipients | param@ and hello@ | same |
Staging and the demo have no Cloud Monitoring checks recorded. brew/.wt-prod/docs/production.md:66-69
The app's own alerts, alerts.js
| Alert | Fires when | First thing to do | Source |
|---|---|---|---|
| Vertex billing | A Vertex call fails on billing or credits | Check billing and credits for squareup-492617 | brew/alerts.js:4 brew/alerts.js:40 |
| Vertex quota | 5 or more 429 or quota errors in 10 minutes | Check the quota page, or slow background jobs | brew/alerts.js:5 brew/alerts.js:41 |
| AI pause | A billing failure paused every AI call for 15 minutes | Fix billing; it resumes by itself | brew/alerts.js:6 brew/alerts.js:42 |
| treg low | treg team balance under $10 | Top up team squareup on treg; check the balance live first | brew/alerts.js:7 memory treg-usage-rules |
| 5xx | 5% or more of 10 minutes' answers were 5xx, once there were at least 50; 20 on production | Cloud Run logs, severity ERROR | brew/alerts.js:8 brew/tools/prod-env.sh:9 |
| Sloosh | Finals cannot be made; they fall back to Gemini | Reconnect at /api/ops/sloosh/connect |
brew/alerts.js:9-10 brew/alerts.js:44 |
| Worker down | No worker holds the background lease for about 4 minutes | Look at brew-worker in Cloud Run |
brew/server.js:731 brew/alerts.js:46 |
| Queue stuck | An interactive job waited over 10 minutes | Look at brew-worker |
brew/server.js:732 brew/alerts.js:47 |
- Each alert is sent at most once a day, and the subject names the site: production, staging or the service name. brew/alerts.js:58-66 brew/alerts.js:72
- The inbox is
ALERT_EMAIL, default hello@joinsquareup.com. brew/alerts.js:56 - Alerts are off locally, under
AI_MOCKand on the demo. Production setsSLOOSH_CHECK=0, so the Sloosh alert does not run there. brew/alerts.js:11 brew/tools/prod-env.sh:9
Release checks
Every promote checks six pages through the load balancer after traffic moves and rolls back if any fails. brew/tools/promote.sh:112-118
Tomorrow
| Item | Version | Tag |
|---|---|---|
| Cloud Monitoring on production plus the app's alerts | v1 | Stays |
One uptime check on staging's /healthz/ready |
v1 | New |
| A named person on call each week | v1 | New, see Open |
| Alerts to a Slack channel as well as email | v2 | New |
| A monthly uptime number per site | v1 | New |
How we do it
- When an alert email arrives, read the subject: which site, which alert.
- Do the first thing in the table above.
- For "site down", run the DNS checks in DNS and domains before anything else. memory brew-dns-local
- Read the logs:
gcloud logging read 'resource.labels.service_name="brew-web" AND severity>=ERROR' --project=squareup-492617 --freshness=1h --limit=50
gcloud logging read 'textPayload:"[slow]" OR textPayload:"[unhandled]"' --project=squareup-492617 --freshness=1h --limit=50
- If a partner is affected, open an incident, see Incidents.
Numbers
| Metric | Definition | Where measured |
|---|---|---|
| Uptime | Share of minutes /healthz/ready passes, per month |
Cloud Monitoring uptime check, Proposed |
| Alerts per week | App and Cloud Monitoring alert emails | hello@ inbox, Proposed |
| Time to first look | Minutes from an alert email to someone acting on it | Proposed, logged per incident |
Open
- Who reads hello@ and who answers out of hours? No source names anyone. Proposal: Param is on call every week, with Arvind as backup, until a second engineer joins; hello@ forwards to Param's phone; any alert gets a first look within 2 hours from 9 am to 10 pm IST.
- Should staging have an uptime check? Proposal: yes, one check on
/healthz/ready, emailing hello@ only. - Should the Sloosh check run on production? It is off there because Sloosh is not connected on production. Proposal: turn it on the day Sloosh is connected there.