---
id: "9.10"
title: "Incident response"
dept: security-privacy-and-legal
status: needs-you
owner: Param
branded: false
last_verified: "2026-10-26"
sources:
  - "brew/docs/production.md:31, 45-57, 112-115"
  - "brew/alerts.js:1-13"
  - "brew/sessions.js:30-32"
  - "brew/connectors/protected.js:1-4"
  - "brew/legal-pages.js:5"
  - "memory brew-prod-backend, brew-move-freshness, brew-qa-sweep-2026-10-19, never-disconnect-arata-meta, squareup-tracking-structure"
---

# Incident response

## Answer

No incident or breach runbook exists; this document proposes one, the steps, who to tell and by when, and Param and a lawyer must confirm the notice times before we promise them to any brand.

## Today

| What exists | Source |
|---|---|
| Cloud Monitoring emails param@ and hello@ on errors, slow requests, a lost worker lease, failed releases, disk, connections, CPU, export failure and site down | [src: brew/docs/production.md:112-115] |
| The app emails hello@ once a day per issue: Vertex billing or quota, the AI pause, treg low, 5xx rate, Sloosh down | [src: brew/alerts.js:1-13] |
| A failed release rolls itself back; a failed rollback prints `ROLLBACK FAILED` with what to restore | [src: brew/docs/production.md:31] |
| Restore steps for the whole database, one brand or files | [src: brew/docs/production.md:45-57] |
| A root admin can end every session of one email | [src: brew/sessions.js:30-32] |
| Arata's connections are protected in code against an accidental disconnect | [src: brew/connectors/protected.js:1-4] [src: memory never-disconnect-arata-meta] |
| One contact address for everything: hello@joinsquareup.com | [src: brew/legal-pages.js:5] |
| No breach runbook, no on call rota, no list of whom to tell | none |

Past incidents, none of them a breach:

| Date | What happened | Source |
|---|---|---|
| 16 Oct | One out of stock 15ml variant marked 54 of Arata's Plays out of date | [src: memory brew-move-freshness] |
| 19 Oct | A deploy from a stale local main dropped fixes from live | [src: memory brew-qa-sweep-2026-10-19] |
| 25 Oct | A network blip on a Mac rolled back a healthy release; an edit to `promote.sh` mid run caused another rollback | [src: memory brew-prod-backend] |
| 28 Sep | Tracking keys in the same Google project were pasted in chat; rotation still pending | [src: memory squareup-tracking-structure] |

## Tomorrow

| Item | Version | Tag |
|---|---|---|
| The runbook below | v1 | New, proposed |
| A security contact, security@ forwarding to Param | v1 | New, proposed |
| An incident log page in this handbook | v1 | New, proposed |
| A kill switch per brand that stops posting and scheduling | v2 | New |
| A kill switch that stops every ad rule at once | v4 | New |

### Proposed severity levels

| Level | Means | Example |
|---|---|---|
| 1 | Brand data seen by someone outside its team, or a key leaked | A guest reads another brand's numbers |
| 2 | Production down, or wrong numbers shown to a brand | Fact sheet shows another store's figures |
| 3 | A feature broken for some brands | Cards fail to load for one brand |

### Proposed runbook for a level 1

1. **Within 1 hour of noticing:** Param, or whoever saw it, writes in the team channel: what, which brands, since when.
2. **Stop it:** rotate the leaked key in Secret Manager and promote; end sessions with `/api/admin/sessions`; take the route down by promoting a fix or rolling back.
3. **Keep evidence:** export the load balancer and Cloud Run logs for the window before they roll off.
4. **Within 72 hours of confirming:** email every affected brand's admins: what happened, what data, what we did, what they should do.
5. **Regulators:** the lawyer confirms what Indian law asks. CERT-In's 2022 directions ask for some cyber incidents to be reported within 6 hours, and the Digital Personal Data Protection Act asks for notice to the Data Protection Board and the people affected.
6. **Platforms:** tell Shopify, Meta or Google when their data or tokens are involved, as their terms ask.
7. **Within 7 days:** write the incident on the incident log with cause, fix and the test that now guards it.

## How we do it

Outages, wrong numbers and runaway spend follow [incidents](/ops-and-infrastructure/incidents); this page covers data seen outside its team and leaked keys.

Until the runbook is accepted:

1. Any suspected leak goes to Param at once, by phone if he does not answer in the team channel.
2. Never disconnect Arata's Meta or Shopify as a response without Param.
3. Restore from backup only by the steps in `docs/production.md`.

## Numbers

| Metric | Definition | Where measured | Status |
|---|---|---|---|
| Time to stop | Minutes from noticing to the leak stopped | Incident log | Proposed |
| Time to tell | Hours from confirming to the brand's email | Incident log | Proposed |
| Incidents by level | Count per month | Incident log | Proposed, none at level 1 on record |

## Open

1. **Who is on call?** Proposal: Param in the day, Arvind as backup; one phone number each in the team channel.
2. **Do we promise brands 72 hours?** Proposal: yes, in the DPA, after the lawyer confirms Indian law needs nothing shorter.
3. **A security@ address?** Proposal: yes, forwarding to Param and hello@, named on the privacy page.
4. **Rotate the 28 Sep keys now?** Proposal: yes, before any brand asks about security.