{
 "id": "8.7",
 "title": "Monitoring and alerts",
 "department": "Ops and infrastructure",
 "department_slug": "ops-and-infrastructure",
 "direction": null,
 "branded": false,
 "status": "draft",
 "owner": "Param",
 "last_verified": "Mon Oct 26",
 "holds": "Uptime checks, alert policies, who gets them",
 "urls": {
  "html": "https://handbook.joinsquareup.com/ops-and-infrastructure/monitoring-and-alerts",
  "markdown": "https://handbook.joinsquareup.com/ops-and-infrastructure/monitoring-and-alerts.md",
  "json": "https://handbook.joinsquareup.com/ops-and-infrastructure/monitoring-and-alerts.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "Two layers watch Brew and email you; nobody is on call",
  "points": [
   {
    "icon": "eye",
    "text": "Cloud Monitoring checks production from outside; the app's own alerts watch from inside."
   },
   {
    "icon": "chat",
    "text": "Both layers email param@ and hello@, and each app alert sends once a day."
   },
   {
    "icon": "warn",
    "text": "Staging and the demo have no Cloud Monitoring checks recorded."
   },
   {
    "icon": "check",
    "text": "Every promote checks six pages and rolls back if any fails."
   },
   {
    "icon": "clock",
    "text": "Proposed: a named person on call each week, with a first look within 2 hours."
   }
  ],
  "stat": {
   "value": "85",
   "label": "Percent disk use that alerts"
  },
  "decide": "Who reads hello@ and answers out of hours?",
  "mood": "waiting"
 },
 "sources": [
  "brew/.wt-prod/docs/production.md:66-69",
  "brew origin/main a559de6: alerts.js:1-47, alerts.js:54-90, server.js:731-732, tools/prod-env.sh:9, tools/promote.sh:112-118",
  "memory brew-prod-backend, brew-dns-local, treg-usage-rules",
  "handbook content/engineering/performance-and-reliability.md"
 ],
 "open_questions": [
  "**Who reads hello@ and who answers out of hours?** No source names anyone. Proposal: Param is on call every week, with Arvind as backup, until a second engineer joins; hello@ forwards to Param's phone; any alert gets a first look within 2 hours from 9 am to 10 pm IST.",
  "**Should staging have an uptime check?** Proposal: yes, one check on `/healthz/ready`, emailing hello@ only.",
  "**Should the Sloosh check run on production?** It is off there because Sloosh is not connected on production. Proposal: turn it on the day Sloosh is connected there."
 ],
 "parts": {
  "Answer": "Two layers watch Brew, Cloud Monitoring from outside and the app's own alerts from inside, and both email param@ and hello@; this page lists every check, what it means when it fires and the first thing to do, and leaves who answers out of hours to Param.",
  "Today": "### Cloud Monitoring, production\n\n| Check | Kind | Source |\n|---|---|---|\n| `/`, `/healthz/ready`, `/api/config` | Uptime checks | [src: brew/.wt-prod/docs/production.md:68] |\n| Server errors, many slow requests, unhandled errors, site down | Alert policies, partly from log metrics `[slow]` and `[unhandled]` | [src: brew/.wt-prod/docs/production.md:68] [src: memory brew-prod-backend] |\n| Worker lost its lease, worker not running | Alert policies | same |\n| A release job failed | Alert policy on `brew-prod-migrate` and `brew-prod-clean` | same |\n| Database disk over 85%, connections over 160, CPU over 80% | Alert policies | [src: brew/.wt-prod/docs/production.md:68] |\n| Nightly export failed | Alert policy | same |\n| Recipients | param@ and hello@ | same |\n\nStaging and the demo have no Cloud Monitoring checks recorded. [src: brew/.wt-prod/docs/production.md:66-69]\n\n### The app's own alerts, `alerts.js`\n\n| Alert | Fires when | First thing to do | Source |\n|---|---|---|---|\n| Vertex billing | A Vertex call fails on billing or credits | Check billing and credits for squareup-492617 | [src: brew/alerts.js:4] [src: brew/alerts.js:40] |\n| Vertex quota | 5 or more 429 or quota errors in 10 minutes | Check the quota page, or slow background jobs | [src: brew/alerts.js:5] [src: brew/alerts.js:41] |\n| AI pause | A billing failure paused every AI call for 15 minutes | Fix billing; it resumes by itself | [src: brew/alerts.js:6] [src: brew/alerts.js:42] |\n| treg low | treg team balance under $10 | Top up team squareup on treg; check the balance live first | [src: brew/alerts.js:7] [src: memory treg-usage-rules] |\n| 5xx | 5% or more of 10 minutes' answers were 5xx, once there were at least 50; 20 on production | Cloud Run logs, severity ERROR | [src: brew/alerts.js:8] [src: brew/tools/prod-env.sh:9] |\n| Sloosh | Finals cannot be made; they fall back to Gemini | Reconnect at `/api/ops/sloosh/connect` | [src: brew/alerts.js:9-10] [src: brew/alerts.js:44] |\n| Worker down | No worker holds the background lease for about 4 minutes | Look at `brew-worker` in Cloud Run | [src: brew/server.js:731] [src: brew/alerts.js:46] |\n| Queue stuck | An interactive job waited over 10 minutes | Look at `brew-worker` | [src: brew/server.js:732] [src: brew/alerts.js:47] |\n\n- Each alert is sent at most once a day, and the subject names the site: production, staging or the service name. [src: brew/alerts.js:58-66] [src: brew/alerts.js:72]\n- The inbox is `ALERT_EMAIL`, default hello@joinsquareup.com. [src: brew/alerts.js:56]\n- Alerts are off locally, under `AI_MOCK` and on the demo. Production sets `SLOOSH_CHECK=0`, so the Sloosh alert does not run there. [src: brew/alerts.js:11] [src: brew/tools/prod-env.sh:9]\n\n### Release checks\n\nEvery promote checks six pages through the load balancer after traffic moves and rolls back if any fails. [src: brew/tools/promote.sh:112-118]",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| Cloud Monitoring on production plus the app's alerts | v1 | Stays |\n| One uptime check on staging's `/healthz/ready` | v1 | New |\n| A named person on call each week | v1 | New, see Open |\n| Alerts to a Slack channel as well as email | v2 | New |\n| A monthly uptime number per site | v1 | New |",
  "How we do it": "1. When an alert email arrives, read the subject: which site, which alert.\n2. Do the first thing in the table above.\n3. For \"site down\", run the DNS checks in [DNS and domains](/ops-and-infrastructure/dns-and-domains) before anything else. [src: memory brew-dns-local]\n4. Read the logs:\n\n```bash\ngcloud logging read 'resource.labels.service_name=\"brew-web\" AND severity>=ERROR' --project=squareup-492617 --freshness=1h --limit=50\ngcloud logging read 'textPayload:\"[slow]\" OR textPayload:\"[unhandled]\"' --project=squareup-492617 --freshness=1h --limit=50\n```\n\n5. If a partner is affected, open an incident, see [Incidents](/ops-and-infrastructure/incidents).",
  "Numbers": "| Metric | Definition | Where measured |\n|---|---|---|\n| Uptime | Share of minutes `/healthz/ready` passes, per month | Cloud Monitoring uptime check, Proposed |\n| Alerts per week | App and Cloud Monitoring alert emails | hello@ inbox, Proposed |\n| Time to first look | Minutes from an alert email to someone acting on it | Proposed, logged per incident |",
  "Open": "1. **Who reads hello@ and who answers out of hours?** No source names anyone. Proposal: Param is on call every week, with Arvind as backup, until a second engineer joins; hello@ forwards to Param's phone; any alert gets a first look within 2 hours from 9 am to 10 pm IST.\n2. **Should staging have an uptime check?** Proposal: yes, one check on `/healthz/ready`, emailing hello@ only.\n3. **Should the Sloosh check run on production?** It is off there because Sloosh is not connected on production. Proposal: turn it on the day Sloosh is connected there."
 }
}