{
 "id": "5.3",
 "title": "Data model",
 "department": "Engineering",
 "department_slug": "engineering",
 "direction": null,
 "branded": false,
 "status": "draft",
 "owner": "Param",
 "last_verified": "2026-10-26",
 "holds": "Postgres tables, doc parts, bucket mirror, what lives where",
 "urls": {
  "html": "https://handbook.joinsquareup.com/engineering/data-model",
  "markdown": "https://handbook.joinsquareup.com/engineering/data-model.md",
  "json": "https://handbook.joinsquareup.com/engineering/data-model.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "Everything lives in Postgres rows, with only binaries left in the bucket",
  "points": [
   {
    "icon": "box",
    "text": "The objects table holds every structured file; big documents split into doc_parts."
   },
   {
    "icon": "people",
    "text": "A brand pack is shared by the team; a workspace belongs to one viewer."
   },
   {
    "icon": "gear",
    "text": "Real tables exist for jobs, counters, sessions, access log and deletions."
   },
   {
    "icon": "warn",
    "text": "Connections, secrets, Plays and jams are still objects rows, not real tables."
   },
   {
    "icon": "chart",
    "text": "Parity checks that no bucket object is newer than its row."
   }
  ],
  "stat": {
   "value": "85",
   "label": "percent disk alert"
  },
  "decide": "Which domain tables come next? Proposal: connections and secrets, then jams.",
  "mood": "building"
 },
 "sources": [
  "brew origin/main d6154b3: db/migrations/001_objects.sql to 010_data_deletions.sql, db/brand-wipe.js:187-195",
  "brew origin/main: objects.js:1-20, 51-75, 145; shared-doc.js:1-10; store.js:1-6; vault.js:1-12; jams.js:1-8",
  "brew origin/main: SPEC.md (brand pack, ledger, profile, Play shapes)",
  "brew origin/main: deploy.sh:20-23, 39; tools/prod-env.sh:9",
  "brew origin/main: docs/handover/engineering/03-data-model.md (15 Oct, bucket era, corrected here)",
  "brew/.wt-prod/docs/production.md",
  "memory brew-storage, brew-prod-backend, brew-jam-reset, brew-wipe-data"
 ],
 "open_questions": [
  "**Which domain tables come next?** Proposal: connections and secrets first (they carry the protected-connection rule), then jams.",
  "**How long do we keep `_deleted/` archives and `raw/` fetches?** Proposal: 30 days for `_deleted/`, 90 days for `raw/`, by bucket lifecycle rules."
 ],
 "parts": {
  "Answer": "Brew keeps every structured file it used to write to the bucket as a row in one Postgres table (`objects`), large documents split into `doc_parts`, a handful of real tables for work across instances, and only binaries in the bucket; this page says what lives where and who may read it.",
  "Today": "### Two kinds of state\n\n| State | Key | Shared by | Where | Source |\n|---|---|---|---|---|\n| Brand pack: what Brew knows about one brand | `brands/<slug>/…`; slug is the registrable domain with dots as dashes (`arata.in` is `arata-in`) | everyone allowed to see the brand | `objects` rows, binaries in the bucket | [src: brew/SPEC.md] [src: brew/objects.js:1-4] |\n| Workspace (\"tenant\"): one viewer's feed state | `tenants/t_<hex>.json` | one person or browser | `objects`, stored in parts | [src: brew/objects.js:11] |\n| Team jams | `brands/<slug>/private/jams.json` (v1) and `jams-v2.json` (Studio) | the brand's team and jam guests | `objects`, one part per jam, change and version list | [src: brew/jams.js:1-8] [src: brew/objects.js:11-13] |\n\n### Postgres tables\n\n| Table | Holds | Notes | Source |\n|---|---|---|---|\n| `objects` | Every JSON, text, markdown, CSV, JS, HTML or SVG object, keyed by its old bucket name | lz4 compressed body; `version` from one global sequence; `src` marks imported rows; a trigger NOTIFYs `key|version` | [src: brew/db/migrations/001_objects.sql] [src: brew/db/migrations/007_objects_src_notify.sql] [src: brew/db/migrations/008_global_version.sql] |\n| `doc_parts` | Parts of big documents: `moves.json` (public and private) one part per Play, each workspace one part per top-level field, the team jams one part per jam | A save writes only changed parts; the `objects` row stays as a stub (`application/x-parted`) and is the document's lock | [src: brew/db/migrations/004_doc_parts.sql] [src: brew/objects.js:11-16] |\n| `jobs` | Work any instance asks for and one worker does | One open job per key | [src: brew/db/migrations/002_jobs_counters_events.sql] |\n| `counters` | Day counters per UTC day, scope and key (AI calls, treg dollars, build dollars) | | same |\n| `build_events` | A build's progress lines | Purged after 3 days | same, [src: brew/server.js:724] |\n| `sessions`, `session_revocations` | Revocable sign-ins, sign out everywhere | | [src: brew/db/migrations/003_access.sql] |\n| `access_log` | Who opened which brand, with access kind (brew, team, official, blocked) | | same |\n| `jam_presence` | Who has a jam open | Unlogged | [src: brew/db/migrations/005_jam_presence.sql] |\n| `bus` | Messages between instances | Kept a day | [src: brew/db/migrations/006_bus.sql] |\n| `make_claims` | One paid Studio make per board | | [src: brew/db/migrations/009_make_claims.sql] |\n| `data_deletions` | One row per data request deletion by `db/delete-brand.mjs`: slug, request, operator, start and end, planned and removed counts by folder, the grants the brand revokes itself, what backups still hold, rows that came back. Never contents or member names | The only thing kept of a deleted brand; `done_at` empty means it did not finish. On staging; production gets it with the next promote | [src: brew/db/migrations/010_data_deletions.sql:1-18] [src: brew/db/brand-wipe.js:187-195] |\n\nThe approved plan also named real tables for access, connections and secrets, Plays, jams, finals and spend, and the brand feed. Only `access_log`, `sessions` and the deletion audit `data_deletions` exist; the rest are still `objects` rows. [src: memory brew-prod-backend] [src: brew/db/migrations/]\n\n### The bucket\n\n| Bucket | Site | Holds | Source |\n|---|---|---|---|\n| `gs://brew-prod` | production | pictures, video, PDFs, zips; versioning on, 7-day soft delete | [src: brew/.wt-prod/docs/production.md] |\n| `gs://brew-prototype` | staging | binaries, plus a copy of every row while `OBJECTS_MIRROR=1` | [src: brew/deploy.sh:20-23,39] |\n| `gs://brew-prototype-demo` | demo | everything (the demo runs without Postgres) | [src: brew/tools/deploy-demo.sh] |\n| `gs://brew-prod-backups` | production | nightly SQL export, promote lock | [src: brew/.wt-prod/docs/production.md] |\n\nOn staging, `OBJECTS_READTHROUGH=1` reads an object not yet imported from the bucket once and keeps it. Production runs with both switches off. [src: brew/objects.js:145] [src: brew/tools/prod-env.sh:9]\n\n### A brand pack\n\nMain files, from SPEC.md with the corrections the code made since:\n\n| File | What | Who sees it | Source |\n|---|---|---|---|\n| `profile.json` | The read: catalog, reviews, ads, rivals, site, AI answers, socials, company | public pack | [src: brew/SPEC.md] |\n| `ledger.json` | Every fact with source, method, layer (observed or inferred), date | public pack | [src: brew/SPEC.md] |\n| `findings.json` | The onboarding steps and cards shown | team | [src: brew/SPEC.md] |\n| `facts.json`, `facts-daily.json` | The fact sheet cards may quote; the daily price check | public pack | [src: brew/facts-desk.js:1-5] |\n| `brief.md`, `brief.json` | The brand brief | team | [src: brew/brief.js] |\n| `moves.json`, `moves/NN.json` | Every Play, an object keyed by number (not an array) | team | [src: brew/build.js:383-388] |\n| `seed.json`, `config.json`, `labels.json`, `notes.json`, `move-value.json`, `plain.json`, `pages.json` | Cards, config, labels, margin notes, grades, plain words, card pages | via the feed | [src: brew/brand-pack.js:173] |\n| `status.json`, `cost.json`, `updates.json` | Build state, spend, stale flags and the rewrite pot | team | [src: brew/build.js] |\n| `brandbook-read.json` | The Brand Book from public evidence | share view | [src: brew/brand-book-read.js] |\n| `listen/`, `ideas/`, `research.json`, `live-watch.json`, `pulse/`, `video/`, `drafts/`, `memory/`, `learned.json` | Listening, ideas, stories, watchers, pulse, video reads, first drafts, brand memory, learnings | mixed | [src: brew/docs/handover/engineering/03-data-model.md] |\n| `private/…` | Team-only: `members.json`, `connections.json`, `vault/`, `live/<id>.json`, `facts.json`, `moves.json`, `cards.json`, `brief.json`, `profile-live.json`, `jams.json`, `jams-v2.json`, `brandbook.json`, `references.json`, `shop-brand.json`, `meta-winners.json`, `mail-prefs.json`, `finals/`, `pdf/`, `statics/` | team only | [src: git grep \"'private/\" origin/main] |\n\nBucket-wide keys: `tenants/`, `gen/` (made media), `research-media/`, `_cache/` (treg and Apify), `_brew/` (Brew's own day spend, usage, alerts, visitors), `control/live.json`. [src: brew/docs/handover/engineering/03-data-model.md] [src: brew/budget.js:1-5]\n\n### Secrets\n\nConnector tokens and keys are AES-256-GCM encrypted under `brands/<slug>/private/vault/` with `FEED_VAULT_KEY` (production uses `FEED_VAULT_KEY_PROD`). Tokens never leave the server. [src: brew/vault.js:1-12] [src: brew/tools/prod-env.sh:10]\n\n### Play numbering\n\n| Range | What | Source |\n|---|---|---|\n| 0 | The index | [src: brew/data/move-specs.json] |\n| 1 to 50 | Core Plays | same |\n| 51 to 65 | Listening-era Plays, written when their evidence exists | same |\n| 100 to 999 | Dynamic Plays (festival, research gap, community replies) | [src: brew/docs/handover/engineering/03-data-model.md] |\n| 1001 and up | Weekly listening and automation Plays, never reused | [src: brew/moves-cadence.js:1-5] |\n\n### Concurrency rules\n\n- A parted document is written in one transaction; the stub row is taken first as the lock. [src: brew/objects.js:66-70]\n- A jam two instances changed at once keeps this instance's fields and merges the other side's new list items in time order; a removed item stays removed. [src: brew/shared-doc.js:8-12]\n- Workspaces keep Maps and Sets as tagged objects (`__m`, `__s`). [src: brew/store.js:6]",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| `objects` plus `doc_parts` as the store for packs and workspaces | v1 | Stays |\n| Real tables for connections and secrets, jams, finals and spend | v2 | New |\n| The bucket mirror on staging | v2 | Goes once Postgres has run a month clean |\n| Self-serve delete of a brand's data (today an email request) | v2 | New |\n| One version per object with history kept in Postgres | v3 | New |",
  "How we do it": "1. Read and write through `packs` and `objects`; never call the bucket directly for structured data. [src: brew/objects.js:1-5]\n2. A new big document that two instances may change at once goes in `PARTED` and, if shared, `ALWAYS`. [src: brew/objects.js:11-13]\n3. Anything with a brand's account numbers goes under `private/` and is served only to members.\n4. Schema changes are additive migrations in `db/migrations/NNN_name.sql`; production runs them in the `brew-prod-migrate` job before the new code serves. [src: brew/tools/promote.sh:64-68]\n5. After any staging deploy that ran on the bucket, import newer bucket objects:\n\n```bash\ngcloud run jobs execute brew-import --region=asia-south1 --wait\n```\n\n6. Reset jams on Postgres (never the old bucket script):\n\n```bash\ngcloud run jobs execute brew-ops --region=asia-south1 --args=db/reset-jams.mjs,--slug,<slug>,--rev-n,<live n> --wait\n```\n\n[src: memory brew-jam-reset]",
  "Numbers": "| Metric | Definition | Where measured |\n|---|---|---|\n| Rows and parts per brand | Count of `objects` and `doc_parts` rows by brand prefix | `db/status.mjs`; the monthly restore test compares them [src: brew/.wt-prod/docs/production.md] |\n| Parity | Bucket objects newer than their row | `db/parity.mjs` [src: memory brew-prod-backend] |\n| Database disk | Percent used; alert above 85% | Cloud Monitoring [src: brew/.wt-prod/docs/production.md] |",
  "Open": "1. **Which domain tables come next?** Proposal: connections and secrets first (they carry the protected-connection rule), then jams.\n2. **How long do we keep `_deleted/` archives and `raw/` fetches?** Proposal: 30 days for `_deleted/`, 90 days for `raw/`, by bucket lifecycle rules."
 }
}