---
id: "8.7"
title: "Monitoring and alerts"
dept: ops-and-infrastructure
status: draft
owner: Param
branded: false
last_verified: 2026-10-26
sources:
  - "brew/.wt-prod/docs/production.md:66-69"
  - "brew origin/main a559de6: alerts.js:1-47, alerts.js:54-90, server.js:731-732, tools/prod-env.sh:9, tools/promote.sh:112-118"
  - "memory brew-prod-backend, brew-dns-local, treg-usage-rules"
  - "handbook content/engineering/performance-and-reliability.md"
---

# Monitoring and alerts

## Answer

Two layers watch Brew, Cloud Monitoring from outside and the app's own alerts from inside, and both email param@ and hello@; this page lists every check, what it means when it fires and the first thing to do, and leaves who answers out of hours to Param.

## Today

### Cloud Monitoring, production

| Check | Kind | Source |
|---|---|---|
| `/`, `/healthz/ready`, `/api/config` | Uptime checks | [src: brew/.wt-prod/docs/production.md:68] |
| Server errors, many slow requests, unhandled errors, site down | Alert policies, partly from log metrics `[slow]` and `[unhandled]` | [src: brew/.wt-prod/docs/production.md:68] [src: memory brew-prod-backend] |
| Worker lost its lease, worker not running | Alert policies | same |
| A release job failed | Alert policy on `brew-prod-migrate` and `brew-prod-clean` | same |
| Database disk over 85%, connections over 160, CPU over 80% | Alert policies | [src: brew/.wt-prod/docs/production.md:68] |
| Nightly export failed | Alert policy | same |
| Recipients | param@ and hello@ | same |

Staging and the demo have no Cloud Monitoring checks recorded. [src: brew/.wt-prod/docs/production.md:66-69]

### The app's own alerts, `alerts.js`

| Alert | Fires when | First thing to do | Source |
|---|---|---|---|
| Vertex billing | A Vertex call fails on billing or credits | Check billing and credits for squareup-492617 | [src: brew/alerts.js:4] [src: brew/alerts.js:40] |
| Vertex quota | 5 or more 429 or quota errors in 10 minutes | Check the quota page, or slow background jobs | [src: brew/alerts.js:5] [src: brew/alerts.js:41] |
| AI pause | A billing failure paused every AI call for 15 minutes | Fix billing; it resumes by itself | [src: brew/alerts.js:6] [src: brew/alerts.js:42] |
| treg low | treg team balance under $10 | Top up team squareup on treg; check the balance live first | [src: brew/alerts.js:7] [src: memory treg-usage-rules] |
| 5xx | 5% or more of 10 minutes' answers were 5xx, once there were at least 50; 20 on production | Cloud Run logs, severity ERROR | [src: brew/alerts.js:8] [src: brew/tools/prod-env.sh:9] |
| Sloosh | Finals cannot be made; they fall back to Gemini | Reconnect at `/api/ops/sloosh/connect` | [src: brew/alerts.js:9-10] [src: brew/alerts.js:44] |
| Worker down | No worker holds the background lease for about 4 minutes | Look at `brew-worker` in Cloud Run | [src: brew/server.js:731] [src: brew/alerts.js:46] |
| Queue stuck | An interactive job waited over 10 minutes | Look at `brew-worker` | [src: brew/server.js:732] [src: brew/alerts.js:47] |

- Each alert is sent at most once a day, and the subject names the site: production, staging or the service name. [src: brew/alerts.js:58-66] [src: brew/alerts.js:72]
- The inbox is `ALERT_EMAIL`, default hello@joinsquareup.com. [src: brew/alerts.js:56]
- Alerts are off locally, under `AI_MOCK` and on the demo. Production sets `SLOOSH_CHECK=0`, so the Sloosh alert does not run there. [src: brew/alerts.js:11] [src: brew/tools/prod-env.sh:9]

### Release checks

Every promote checks six pages through the load balancer after traffic moves and rolls back if any fails. [src: brew/tools/promote.sh:112-118]

## Tomorrow

| Item | Version | Tag |
|---|---|---|
| Cloud Monitoring on production plus the app's alerts | v1 | Stays |
| One uptime check on staging's `/healthz/ready` | v1 | New |
| A named person on call each week | v1 | New, see Open |
| Alerts to a Slack channel as well as email | v2 | New |
| A monthly uptime number per site | v1 | New |

## How we do it

1. When an alert email arrives, read the subject: which site, which alert.
2. Do the first thing in the table above.
3. For "site down", run the DNS checks in [DNS and domains](/ops-and-infrastructure/dns-and-domains) before anything else. [src: memory brew-dns-local]
4. Read the logs:

```bash
gcloud logging read 'resource.labels.service_name="brew-web" AND severity>=ERROR' --project=squareup-492617 --freshness=1h --limit=50
gcloud logging read 'textPayload:"[slow]" OR textPayload:"[unhandled]"' --project=squareup-492617 --freshness=1h --limit=50
```

5. If a partner is affected, open an incident, see [Incidents](/ops-and-infrastructure/incidents).

## Numbers

| Metric | Definition | Where measured |
|---|---|---|
| Uptime | Share of minutes `/healthz/ready` passes, per month | Cloud Monitoring uptime check, Proposed |
| Alerts per week | App and Cloud Monitoring alert emails | hello@ inbox, Proposed |
| Time to first look | Minutes from an alert email to someone acting on it | Proposed, logged per incident |

## Open

1. **Who reads hello@ and who answers out of hours?** No source names anyone. Proposal: Param is on call every week, with Arvind as backup, until a second engineer joins; hello@ forwards to Param's phone; any alert gets a first look within 2 hours from 9 am to 10 pm IST.
2. **Should staging have an uptime check?** Proposal: yes, one check on `/healthz/ready`, emailing hello@ only.
3. **Should the Sloosh check run on production?** It is off there because Sloosh is not connected on production. Proposal: turn it on the day Sloosh is connected there.