Answer
We have no written incident process yet, so this page records the incidents our notes already describe and proposes how to call one, who does what and the postmortem template, for Param to accept.
Today
No process is written
No note, script or document names an incident lead, a severity scale, a status channel or a postmortem habit. memory brew-prod-backend memory brew-parallel-sessions-deploy
Incidents our notes describe
| Date | What happened | Effect | Fix and lesson | Source |
|---|---|---|---|---|
| 15 Oct | AI credit ran out in the middle of Arata's rewrite | Rewrite stopped part way | Resume bug fixed; hold deploys during long paid runs | memory brew-arata-rewrite-2026-10-15 |
| 16 Oct | A 15 ml variant out of stock matched the 200 ml product and marked 54 of Arata's Plays | Wrong "out of date" marks | Matcher fixed in 3a2d86e, marks healed; check matchers on real variant data | memory brew-move-freshness |
| 16 Oct | Cozy's Plays were rewritten about 116 times in 3 hours after connecting | Wasted AI spend | Settle wait, one rewrite per set of accounts, per-brand day shares, alerts | memory brew-connect-refresh |
| 19 Oct | A session deployed from a local main without pushed fixes | Fixes vanished from staging | Deploy from a fresh worktree of origin/main; check the served hash | memory brew-qa-sweep-2026-10-19 |
| 19 Oct | "Site not working" on Param's Mac | None; local DNS | Three DNS checks before calling an outage | memory brew-dns-local |
| 20 Oct | Anthropic key out of credit at 07:50 IST | Live switched to Gemini | Gemini Pro for Plays, Flash for the rest | memory brew-pro-switch-and-flush |
| about 23 Oct | treg key returned 401 on every call | Reads fell back or failed | New agent key brew-live, old version disabled |
memory brew-apify-watch |
| 25 Oct | After one rollback, production's worker ran the morning's code all day | Old background code on production | promote.sh gives the new worker its traffic by name |
brew/tools/promote.sh:97-98 memory brew-prod-backend |
| 25 Oct | A network blip on the Mac running promote.sh rolled back a healthy release |
Release undone | Each check now gets 5 tries | brew/tools/promote.sh:113 memory brew-prod-backend |
| 25 Oct | promote.sh was edited while it ran |
Rollback | Never edit it while it runs | memory brew-prod-backend |
Tomorrow
| Item | Version | Tag |
|---|---|---|
| A written way to call an incident, roles and a postmortem template | v1 | New, see Open |
| Every incident logged in the table above within a day | v1 | New |
| A status note to affected design partners | v1 | New |
| A status page for partners | v2 | New |
How we do it
Proposed, until Param accepts it.
- Call it when a design partner cannot use the product, data looks wrong to a partner, money is being spent that should not be, or a sign-in or secret may have leaked. Anyone may call it.
- Severity.
- Sev 1: production down, partner data wrong or exposed, a protected connection lost, or spend running away. Data seen outside its team or a leaked key also follows the level 1 breach runbook in incident response, which sets the 72 hour notice.
- Sev 2: one feature broken for a partner, or staging down during a demo.
- Sev 3: anything else worth a record.
- Roles. One lead decides and talks to partners; one fixer works the problem. With three people, the lead is Param and the fixer is whoever is free, often a Claude session he directs.
- First 15 minutes. Run the DNS checks; read the alert; if a release caused it, roll back with
update-traffic; never disconnect Arata; never wipe partner data to fix it. - Tell partners for Sev 1 within an hour, in plain words: what they see, what we are doing, when we next write.
- Postmortem within 2 working days for Sev 1 and 2, in this template:
## <date> <one-line title>
- Severity:
- Lead / fixer:
- Started, noticed, fixed: <times, IST>
- Who was affected and how:
- What happened:
- Why it happened:
- What we changed in code, with the test or guard that stops it coming back:
- What we told partners:
- Add a row to the table above and fix the cause in code for every brand, with a test. memory brew-qa-sweep-2026-10-19
Numbers
| Metric | Definition | Where measured |
|---|---|---|
| Incidents per month | Rows added to the table, by severity | This page, Proposed |
| Time to fix | Minutes from noticed to fixed, Sev 1 and 2 | Postmortems, Proposed |
| Repeat incidents | Incidents whose cause matches an earlier row | This page, Proposed; target 0 |
Open
- Accept the proposed process above? Proposal: accept as written; Param is the lead for every Sev 1 until a second engineer joins.
- Where do we talk during an incident? Proposal: a Slack channel
#incidentsin the joinsquareup workspace, plus a phone call for Sev 1. - What do we promise partners about incidents? Proposal: for Sev 1, a message within an hour and a short written postmortem within 2 working days; nothing about uptime until we measure it.