{
 "id": "8.8",
 "title": "Incidents",
 "department": "Ops and infrastructure",
 "department_slug": "ops-and-infrastructure",
 "direction": null,
 "branded": false,
 "status": "needs-you",
 "owner": "Param",
 "last_verified": "Mon Oct 26",
 "holds": "Calling one, roles, the postmortem template",
 "urls": {
  "html": "https://handbook.joinsquareup.com/ops-and-infrastructure/incidents",
  "markdown": "https://handbook.joinsquareup.com/ops-and-infrastructure/incidents.md",
  "json": "https://handbook.joinsquareup.com/ops-and-infrastructure/incidents.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "No incident process yet; here is one to accept",
  "points": [
   {
    "icon": "book",
    "text": "Ten past incidents from 15 to 25 Oct are logged, each with its lesson."
   },
   {
    "icon": "warn",
    "text": "No note names an incident lead, a severity scale or a postmortem habit."
   },
   {
    "icon": "flag",
    "text": "Proposed: three severity levels, with Param as lead for every Sev 1."
   },
   {
    "icon": "clock",
    "text": "Tell affected partners within an hour for Sev 1; postmortem within 2 working days."
   },
   {
    "icon": "lock",
    "text": "First 15 minutes: never disconnect Arata, and never wipe partner data to fix it."
   }
  ],
  "stat": {
   "value": "54",
   "label": "Plays wrongly marked 16 Oct"
  },
  "decide": "Accept the proposed incident process, and where do we talk during one?",
  "mood": "waiting"
 },
 "sources": [
  "memory brew-arata-rewrite-2026-10-15 (index line), brew-move-freshness, brew-connect-refresh, brew-qa-sweep-2026-10-19, brew-dns-local, brew-pro-switch-and-flush, brew-apify-watch, brew-prod-backend, brew-parallel-sessions-deploy",
  "brew origin/main a559de6: tools/promote.sh:97-98, tools/promote.sh:113"
 ],
 "open_questions": [
  "**Accept the proposed process above?** Proposal: accept as written; Param is the lead for every Sev 1 until a second engineer joins.",
  "**Where do we talk during an incident?** Proposal: a Slack channel `#incidents` in the joinsquareup workspace, plus a phone call for Sev 1.",
  "**What do we promise partners about incidents?** Proposal: for Sev 1, a message within an hour and a short written postmortem within 2 working days; nothing about uptime until we measure it."
 ],
 "parts": {
  "Answer": "We have no written incident process yet, so this page records the incidents our notes already describe and proposes how to call one, who does what and the postmortem template, for Param to accept.",
  "Today": "### No process is written\n\nNo note, script or document names an incident lead, a severity scale, a status channel or a postmortem habit. [src: memory brew-prod-backend] [src: memory brew-parallel-sessions-deploy]\n\n### Incidents our notes describe\n\n| Date | What happened | Effect | Fix and lesson | Source |\n|---|---|---|---|---|\n| 15 Oct | AI credit ran out in the middle of Arata's rewrite | Rewrite stopped part way | Resume bug fixed; hold deploys during long paid runs | [src: memory brew-arata-rewrite-2026-10-15] |\n| 16 Oct | A 15 ml variant out of stock matched the 200 ml product and marked 54 of Arata's Plays | Wrong \"out of date\" marks | Matcher fixed in 3a2d86e, marks healed; check matchers on real variant data | [src: memory brew-move-freshness] |\n| 16 Oct | Cozy's Plays were rewritten about 116 times in 3 hours after connecting | Wasted AI spend | Settle wait, one rewrite per set of accounts, per-brand day shares, alerts | [src: memory brew-connect-refresh] |\n| 19 Oct | A session deployed from a local main without pushed fixes | Fixes vanished from staging | Deploy from a fresh worktree of origin/main; check the served hash | [src: memory brew-qa-sweep-2026-10-19] |\n| 19 Oct | \"Site not working\" on Param's Mac | None; local DNS | Three DNS checks before calling an outage | [src: memory brew-dns-local] |\n| 20 Oct | Anthropic key out of credit at 07:50 IST | Live switched to Gemini | Gemini Pro for Plays, Flash for the rest | [src: memory brew-pro-switch-and-flush] |\n| about 23 Oct | treg key returned 401 on every call | Reads fell back or failed | New agent key `brew-live`, old version disabled | [src: memory brew-apify-watch] |\n| 25 Oct | After one rollback, production's worker ran the morning's code all day | Old background code on production | `promote.sh` gives the new worker its traffic by name | [src: brew/tools/promote.sh:97-98] [src: memory brew-prod-backend] |\n| 25 Oct | A network blip on the Mac running `promote.sh` rolled back a healthy release | Release undone | Each check now gets 5 tries | [src: brew/tools/promote.sh:113] [src: memory brew-prod-backend] |\n| 25 Oct | `promote.sh` was edited while it ran | Rollback | Never edit it while it runs | [src: memory brew-prod-backend] |",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| A written way to call an incident, roles and a postmortem template | v1 | New, see Open |\n| Every incident logged in the table above within a day | v1 | New |\n| A status note to affected design partners | v1 | New |\n| A status page for partners | v2 | New |",
  "How we do it": "Proposed, until Param accepts it.\n\n1. **Call it** when a design partner cannot use the product, data looks wrong to a partner, money is being spent that should not be, or a sign-in or secret may have leaked. Anyone may call it.\n2. **Severity.**\n   - Sev 1: production down, partner data wrong or exposed, a protected connection lost, or spend running away. Data seen outside its team or a leaked key also follows the level 1 breach runbook in [incident response](/security-privacy-and-legal/incident-response), which sets the 72 hour notice.\n   - Sev 2: one feature broken for a partner, or staging down during a demo.\n   - Sev 3: anything else worth a record.\n3. **Roles.** One lead decides and talks to partners; one fixer works the problem. With three people, the lead is Param and the fixer is whoever is free, often a Claude session he directs.\n4. **First 15 minutes.** Run the DNS checks; read the alert; if a release caused it, roll back with `update-traffic`; never disconnect Arata; never wipe partner data to fix it.\n5. **Tell partners** for Sev 1 within an hour, in plain words: what they see, what we are doing, when we next write.\n6. **Postmortem** within 2 working days for Sev 1 and 2, in this template:\n\n```markdown",
  "Numbers": "| Metric | Definition | Where measured |\n|---|---|---|\n| Incidents per month | Rows added to the table, by severity | This page, Proposed |\n| Time to fix | Minutes from noticed to fixed, Sev 1 and 2 | Postmortems, Proposed |\n| Repeat incidents | Incidents whose cause matches an earlier row | This page, Proposed; target 0 |",
  "Open": "1. **Accept the proposed process above?** Proposal: accept as written; Param is the lead for every Sev 1 until a second engineer joins.\n2. **Where do we talk during an incident?** Proposal: a Slack channel `#incidents` in the joinsquareup workspace, plus a phone call for Sev 1.\n3. **What do we promise partners about incidents?** Proposal: for Sev 1, a message within an hour and a short written postmortem within 2 working days; nothing about uptime until we measure it."
 }
}