Answer
Brew keeps every structured file it used to write to the bucket as a row in one Postgres table (objects), large documents split into doc_parts, a handful of real tables for work across instances, and only binaries in the bucket; this page says what lives where and who may read it.
Today
Two kinds of state
| State | Key | Shared by | Where | Source |
|---|---|---|---|---|
| Brand pack: what Brew knows about one brand | brands/<slug>/…; slug is the registrable domain with dots as dashes (arata.in is arata-in) |
everyone allowed to see the brand | objects rows, binaries in the bucket |
brew/SPEC.md brew/objects.js:1-4 |
| Workspace ("tenant"): one viewer's feed state | tenants/t_<hex>.json |
one person or browser | objects, stored in parts |
brew/objects.js:11 |
| Team jams | brands/<slug>/private/jams.json (v1) and jams-v2.json (Studio) |
the brand's team and jam guests | objects, one part per jam, change and version list |
brew/jams.js:1-8 brew/objects.js:11-13 |
Postgres tables
| Table | Holds | Notes | Source |
|---|---|---|---|
objects |
Every JSON, text, markdown, CSV, JS, HTML or SVG object, keyed by its old bucket name | lz4 compressed body; version from one global sequence; src marks imported rows; a trigger NOTIFYs `key |
version` |
doc_parts |
Parts of big documents: moves.json (public and private) one part per Play, each workspace one part per top-level field, the team jams one part per jam |
A save writes only changed parts; the objects row stays as a stub (application/x-parted) and is the document's lock |
brew/db/migrations/004_doc_parts.sql brew/objects.js:11-16 |
jobs |
Work any instance asks for and one worker does | One open job per key | brew/db/migrations/002_jobs_counters_events.sql |
counters |
Day counters per UTC day, scope and key (AI calls, treg dollars, build dollars) | same | |
build_events |
A build's progress lines | Purged after 3 days | same, brew/server.js:724 |
sessions, session_revocations |
Revocable sign-ins, sign out everywhere | brew/db/migrations/003_access.sql | |
access_log |
Who opened which brand, with access kind (brew, team, official, blocked) | same | |
jam_presence |
Who has a jam open | Unlogged | brew/db/migrations/005_jam_presence.sql |
bus |
Messages between instances | Kept a day | brew/db/migrations/006_bus.sql |
make_claims |
One paid Studio make per board | brew/db/migrations/009_make_claims.sql | |
data_deletions |
One row per data request deletion by db/delete-brand.mjs: slug, request, operator, start and end, planned and removed counts by folder, the grants the brand revokes itself, what backups still hold, rows that came back. Never contents or member names |
The only thing kept of a deleted brand; done_at empty means it did not finish. On staging; production gets it with the next promote |
brew/db/migrations/010_data_deletions.sql:1-18 brew/db/brand-wipe.js:187-195 |
The approved plan also named real tables for access, connections and secrets, Plays, jams, finals and spend, and the brand feed. Only access_log, sessions and the deletion audit data_deletions exist; the rest are still objects rows. memory brew-prod-backend brew/db/migrations/
The bucket
| Bucket | Site | Holds | Source |
|---|---|---|---|
gs://brew-prod |
production | pictures, video, PDFs, zips; versioning on, 7-day soft delete | brew/.wt-prod/docs/production.md |
gs://brew-prototype |
staging | binaries, plus a copy of every row while OBJECTS_MIRROR=1 |
brew/deploy.sh:20-23,39 |
gs://brew-prototype-demo |
demo | everything (the demo runs without Postgres) | brew/tools/deploy-demo.sh |
gs://brew-prod-backups |
production | nightly SQL export, promote lock | brew/.wt-prod/docs/production.md |
On staging, OBJECTS_READTHROUGH=1 reads an object not yet imported from the bucket once and keeps it. Production runs with both switches off. brew/objects.js:145 brew/tools/prod-env.sh:9
A brand pack
Main files, from SPEC.md with the corrections the code made since:
| File | What | Who sees it | Source |
|---|---|---|---|
profile.json |
The read: catalog, reviews, ads, rivals, site, AI answers, socials, company | public pack | brew/SPEC.md |
ledger.json |
Every fact with source, method, layer (observed or inferred), date | public pack | brew/SPEC.md |
findings.json |
The onboarding steps and cards shown | team | brew/SPEC.md |
facts.json, facts-daily.json |
The fact sheet cards may quote; the daily price check | public pack | brew/facts-desk.js:1-5 |
brief.md, brief.json |
The brand brief | team | brew/brief.js |
moves.json, moves/NN.json |
Every Play, an object keyed by number (not an array) | team | brew/build.js:383-388 |
seed.json, config.json, labels.json, notes.json, move-value.json, plain.json, pages.json |
Cards, config, labels, margin notes, grades, plain words, card pages | via the feed | brew/brand-pack.js:173 |
status.json, cost.json, updates.json |
Build state, spend, stale flags and the rewrite pot | team | brew/build.js |
brandbook-read.json |
The Brand Book from public evidence | share view | brew/brand-book-read.js |
listen/, ideas/, research.json, live-watch.json, pulse/, video/, drafts/, memory/, learned.json |
Listening, ideas, stories, watchers, pulse, video reads, first drafts, brand memory, learnings | mixed | brew/docs/handover/engineering/03-data-model.md |
private/… |
Team-only: members.json, connections.json, vault/, live/<id>.json, facts.json, moves.json, cards.json, brief.json, profile-live.json, jams.json, jams-v2.json, brandbook.json, references.json, shop-brand.json, meta-winners.json, mail-prefs.json, finals/, pdf/, statics/ |
team only | git grep "'private/" origin/main |
Bucket-wide keys: tenants/, gen/ (made media), research-media/, _cache/ (treg and Apify), _brew/ (Brew's own day spend, usage, alerts, visitors), control/live.json. brew/docs/handover/engineering/03-data-model.md brew/budget.js:1-5
Secrets
Connector tokens and keys are AES-256-GCM encrypted under brands/<slug>/private/vault/ with FEED_VAULT_KEY (production uses FEED_VAULT_KEY_PROD). Tokens never leave the server. brew/vault.js:1-12 brew/tools/prod-env.sh:10
Play numbering
| Range | What | Source |
|---|---|---|
| 0 | The index | brew/data/move-specs.json |
| 1 to 50 | Core Plays | same |
| 51 to 65 | Listening-era Plays, written when their evidence exists | same |
| 100 to 999 | Dynamic Plays (festival, research gap, community replies) | brew/docs/handover/engineering/03-data-model.md |
| 1001 and up | Weekly listening and automation Plays, never reused | brew/moves-cadence.js:1-5 |
Concurrency rules
- A parted document is written in one transaction; the stub row is taken first as the lock. brew/objects.js:66-70
- A jam two instances changed at once keeps this instance's fields and merges the other side's new list items in time order; a removed item stays removed. brew/shared-doc.js:8-12
- Workspaces keep Maps and Sets as tagged objects (
__m,__s). brew/store.js:6
Tomorrow
| Item | Version | Tag |
|---|---|---|
objects plus doc_parts as the store for packs and workspaces |
v1 | Stays |
| Real tables for connections and secrets, jams, finals and spend | v2 | New |
| The bucket mirror on staging | v2 | Goes once Postgres has run a month clean |
| Self-serve delete of a brand's data (today an email request) | v2 | New |
| One version per object with history kept in Postgres | v3 | New |
How we do it
- Read and write through
packsandobjects; never call the bucket directly for structured data. brew/objects.js:1-5 - A new big document that two instances may change at once goes in
PARTEDand, if shared,ALWAYS. brew/objects.js:11-13 - Anything with a brand's account numbers goes under
private/and is served only to members. - Schema changes are additive migrations in
db/migrations/NNN_name.sql; production runs them in thebrew-prod-migratejob before the new code serves. brew/tools/promote.sh:64-68 - After any staging deploy that ran on the bucket, import newer bucket objects:
gcloud run jobs execute brew-import --region=asia-south1 --wait
- Reset jams on Postgres (never the old bucket script):
gcloud run jobs execute brew-ops --region=asia-south1 --args=db/reset-jams.mjs,--slug,<slug>,--rev-n,<live n> --wait
memory brew-jam-reset
Numbers
| Metric | Definition | Where measured |
|---|---|---|
| Rows and parts per brand | Count of objects and doc_parts rows by brand prefix |
db/status.mjs; the monthly restore test compares them brew/.wt-prod/docs/production.md |
| Parity | Bucket objects newer than their row | db/parity.mjs memory brew-prod-backend |
| Database disk | Percent used; alert above 85% | Cloud Monitoring brew/.wt-prod/docs/production.md |
Open
- Which domain tables come next? Proposal: connections and secrets first (they carry the protected-connection rule), then jams.
- How long do we keep
_deleted/archives andraw/fetches? Proposal: 30 days for_deleted/, 90 days forraw/, by bucket lifecycle rules.