{
 "id": "9.10",
 "title": "Incident response",
 "department": "Security, privacy and legal",
 "department_slug": "security-privacy-and-legal",
 "direction": null,
 "branded": false,
 "status": "needs-you",
 "owner": "Param",
 "last_verified": "2026-10-26",
 "holds": "Breach steps, who to tell, timelines",
 "urls": {
  "html": "https://handbook.joinsquareup.com/security-privacy-and-legal/incident-response",
  "markdown": "https://handbook.joinsquareup.com/security-privacy-and-legal/incident-response.md",
  "json": "https://handbook.joinsquareup.com/security-privacy-and-legal/incident-response.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "No breach runbook exists, so we drafted one with 72 hours",
  "points": [
   {
    "icon": "warn",
    "text": "Alerts exist, but no runbook, on call rota or list of whom to tell."
   },
   {
    "icon": "flag",
    "text": "Three levels proposed; level 1 is data seen outside its team or a leaked key."
   },
   {
    "icon": "clock",
    "text": "Level 1: team told in 1 hour, brands in 72 hours, write up in 7 days."
   },
   {
    "icon": "lock",
    "text": "Evidence is exported before logs roll off, and leaked keys are rotated and promoted."
   },
   {
    "icon": "check",
    "text": "No breach has happened; past incidents were a bad deploy and rollbacks."
   }
  ],
  "stat": {
   "value": "72",
   "label": "hours to tell affected brands"
  },
  "decide": "Promise brands 72 hours in the DPA, and rotate the 28 Sep keys now?",
  "mood": "urgent"
 },
 "sources": [
  "brew/docs/production.md:31, 45-57, 112-115",
  "brew/alerts.js:1-13",
  "brew/sessions.js:30-32",
  "brew/connectors/protected.js:1-4",
  "brew/legal-pages.js:5",
  "memory brew-prod-backend, brew-move-freshness, brew-qa-sweep-2026-10-19, never-disconnect-arata-meta, squareup-tracking-structure"
 ],
 "open_questions": [
  "**Who is on call?** Proposal: Param in the day, Arvind as backup; one phone number each in the team channel.",
  "**Do we promise brands 72 hours?** Proposal: yes, in the DPA, after the lawyer confirms Indian law needs nothing shorter.",
  "**A security@ address?** Proposal: yes, forwarding to Param and hello@, named on the privacy page.",
  "**Rotate the 28 Sep keys now?** Proposal: yes, before any brand asks about security."
 ],
 "parts": {
  "Answer": "No incident or breach runbook exists; this document proposes one, the steps, who to tell and by when, and Param and a lawyer must confirm the notice times before we promise them to any brand.",
  "Today": "| What exists | Source |\n|---|---|\n| Cloud Monitoring emails param@ and hello@ on errors, slow requests, a lost worker lease, failed releases, disk, connections, CPU, export failure and site down | [src: brew/docs/production.md:112-115] |\n| The app emails hello@ once a day per issue: Vertex billing or quota, the AI pause, treg low, 5xx rate, Sloosh down | [src: brew/alerts.js:1-13] |\n| A failed release rolls itself back; a failed rollback prints `ROLLBACK FAILED` with what to restore | [src: brew/docs/production.md:31] |\n| Restore steps for the whole database, one brand or files | [src: brew/docs/production.md:45-57] |\n| A root admin can end every session of one email | [src: brew/sessions.js:30-32] |\n| Arata's connections are protected in code against an accidental disconnect | [src: brew/connectors/protected.js:1-4] [src: memory never-disconnect-arata-meta] |\n| One contact address for everything: hello@joinsquareup.com | [src: brew/legal-pages.js:5] |\n| No breach runbook, no on call rota, no list of whom to tell | none |\n\nPast incidents, none of them a breach:\n\n| Date | What happened | Source |\n|---|---|---|\n| 16 Oct | One out of stock 15ml variant marked 54 of Arata's Plays out of date | [src: memory brew-move-freshness] |\n| 19 Oct | A deploy from a stale local main dropped fixes from live | [src: memory brew-qa-sweep-2026-10-19] |\n| 25 Oct | A network blip on a Mac rolled back a healthy release; an edit to `promote.sh` mid run caused another rollback | [src: memory brew-prod-backend] |\n| 28 Sep | Tracking keys in the same Google project were pasted in chat; rotation still pending | [src: memory squareup-tracking-structure] |",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| The runbook below | v1 | New, proposed |\n| A security contact, security@ forwarding to Param | v1 | New, proposed |\n| An incident log page in this handbook | v1 | New, proposed |\n| A kill switch per brand that stops posting and scheduling | v2 | New |\n| A kill switch that stops every ad rule at once | v4 | New |\n\n### Proposed severity levels\n\n| Level | Means | Example |\n|---|---|---|\n| 1 | Brand data seen by someone outside its team, or a key leaked | A guest reads another brand's numbers |\n| 2 | Production down, or wrong numbers shown to a brand | Fact sheet shows another store's figures |\n| 3 | A feature broken for some brands | Cards fail to load for one brand |\n\n### Proposed runbook for a level 1\n\n1. **Within 1 hour of noticing:** Param, or whoever saw it, writes in the team channel: what, which brands, since when.\n2. **Stop it:** rotate the leaked key in Secret Manager and promote; end sessions with `/api/admin/sessions`; take the route down by promoting a fix or rolling back.\n3. **Keep evidence:** export the load balancer and Cloud Run logs for the window before they roll off.\n4. **Within 72 hours of confirming:** email every affected brand's admins: what happened, what data, what we did, what they should do.\n5. **Regulators:** the lawyer confirms what Indian law asks. CERT-In's 2022 directions ask for some cyber incidents to be reported within 6 hours, and the Digital Personal Data Protection Act asks for notice to the Data Protection Board and the people affected.\n6. **Platforms:** tell Shopify, Meta or Google when their data or tokens are involved, as their terms ask.\n7. **Within 7 days:** write the incident on the incident log with cause, fix and the test that now guards it.",
  "How we do it": "Outages, wrong numbers and runaway spend follow [incidents](/ops-and-infrastructure/incidents); this page covers data seen outside its team and leaked keys.\n\nUntil the runbook is accepted:\n\n1. Any suspected leak goes to Param at once, by phone if he does not answer in the team channel.\n2. Never disconnect Arata's Meta or Shopify as a response without Param.\n3. Restore from backup only by the steps in `docs/production.md`.",
  "Numbers": "| Metric | Definition | Where measured | Status |\n|---|---|---|---|\n| Time to stop | Minutes from noticing to the leak stopped | Incident log | Proposed |\n| Time to tell | Hours from confirming to the brand's email | Incident log | Proposed |\n| Incidents by level | Count per month | Incident log | Proposed, none at level 1 on record |",
  "Open": "1. **Who is on call?** Proposal: Param in the day, Arvind as backup; one phone number each in the team channel.\n2. **Do we promise brands 72 hours?** Proposal: yes, in the DPA, after the lawyer confirms Indian law needs nothing shorter.\n3. **A security@ address?** Proposal: yes, forwarding to Param and hello@, named on the privacy page.\n4. **Rotate the 28 Sep keys now?** Proposal: yes, before any brand asks about security."
 }
}