Snapshots

One worker, one lease, and every timer runs on the lease holder

Being built
  • Production's brew-worker runs exactly one instance and holds a Postgres lease.
  • Connector reads run every 3 to 12 hours; drafts and listening every 30 minutes.
  • Every paid job has a cap, such as $3 a brand a week for families.
  • Never run a second production worker until per-brand leases exist.
  • A new timer must use leaderOnly, carry a cap, and skip the other site's brands.
$40Cap on first connect rewrite
Your call

Are these building-time cadences right for paying partners?

Ops and infrastructure · 8.11

Background jobs

DraftLast checked Mon Oct 26Owner Param

Answer

All of Brew's background work, the watchers, refreshes, reminders and paid rewrites, runs on timers inside the one worker that holds the background lease, each with its own cadence and spend cap; this page lists them so anyone can tell what should be running, how often and what stops it.

Today

Who runs background work

  • Production: brew-worker, exactly one instance, internal only. It holds the background lease: a Postgres advisory lock on its own connection. brew/.wt-prod/docs/production.md:10 brew/roles.js:1-5
  • Staging: the one shopos-feed instance does both roles. brew/.wt-prod/docs/production.md:10
  • Every timer runs its work through leaderOnly, so only the lease holder acts. If the lease connection drops, the worker exits and Cloud Run restarts it. brew/roles.js:25-32
  • Work a click asks for goes through the Postgres jobs queue: claimed with SKIP LOCKED, leased 120 s, retried, at most one open job per key; delivery is at least once. brew/jobs.js:1-13
  • Bucket-era sites keep one live revision with control/live.json; production sets RETIRE_OFF=1 because the lease does that job. brew/retire.js:1-9 brew/tools/prod-env.sh:9
  • A brand whose home is the other site gets no background spend or email from this one; connector reads keep running on both. brew/.wt-prod/docs/production.md:21

The timers

Job Cadence Cap Source
Connector reads hourly tick; Shopify and Meta every 3 h, Google Ads, GA4 and reviews 6 h, Klaviyo 12 h, marketplaces 24 h free reads; marketplaces are paid treg brew/connectors/index.js:37 brew/connectors/index.js:498
Re-read when a member opens the brand after 1 h for Shopify and Meta, 2 h for Google and reviews, 6 h for Klaviyo same brew/connectors/index.js:40
Connected numbers go stale after 4 of their reads, never under 24 h brew/connectors/index.js:54-55
Weekly families of Plays tick hourly, due every 7 days $3 a brand a week brew/moves-cadence.js:406
Background drafts every 30 min, first after 4 min $4 a brand a day brew/drafts-bg.js:142
Social listening and ideas every 30 min; listening every 8 h, full every 24 h, ideas every 24 h $3 a brand a day brew/social-feed.js:274
Catalog watch every 3 h, free brew/live-watch.js:2-3
AI visibility questions weekly treg brew/live-watch.js:4-6
Market pulse daily, weekly, monthly or quarterly per signal treg, booked per brand brew/market-pulse.js:1-11
Fact sheet check hourly tick, first after 4 min free brew/facts-desk.js:380
Evidence tick every 30 min brew/server.js:520
Stories warm-up hourly brew/server.js:562
Weekly digest email checked hourly, sent Monday 09:00 brand time no AI, no paid call brew/digest.js:1-12
Check and feed reminders every 15 min email only brew/confirm-remind.js:137
Sloosh health hourly; off on production brew/server.js:594 brew/tools/prod-env.sh:9
Ops alerts every 10 min brew/alerts.js:90
Lease and queue watch every 2 min brew/server.js:731-732
Stale sweep of out-of-date Plays hourly and after a stock flag weekly pot, about $10 a brand on Flash memory brew-move-freshness

Paid rewrites the timers can start

Trigger Cap Source
A brand's first connected account $40 brew/build.js:1070
The team adds learnings $30 brew/build.js:1075
A connected number moves 10% from base $30 a week brew/build.js:1076
Everything automatic, in order learnings, events, site check, accounts, low, new one weekly pot per brand brew/rewrite-pot.js:1-6

Scheduled outside the app

Job When Source
Cloud Scheduler brew-prod-nightly-export 21:30 UTC daily brew/.wt-prod/docs/production.md:40
The handbook's morning run 9 am, local scheduled task handbook-morning memory brew-handbook

Tomorrow

Item Version Tag
One worker, one lease, timers on the lease holder v1 Stays
Per-brand build leases so several workers can build v2 New
This list generated from code on every build v2 New
Cadences that post or schedule for a brand v2 New, only after a person says yes

How we do it

  1. Is background work running? Look for the lease line in the worker's logs:
gcloud logging read 'resource.labels.service_name="brew-worker" AND textPayload:"[roles]"' --project=squareup-492617 --freshness=1d --limit=10
  1. Are jobs stuck? Run db/status.mjs through brew-ops; it prints jobs by state. brew/db/status.mjs:19
  2. Never run a second worker on production until per-brand leases exist.
  3. A new timer must use leaderOnly, carry a cap, and skip brands whose home is the other site.
  4. A new flag source only flags; the weekly capped run does the rewriting. memory brew-move-freshness

Numbers

Metric Definition Where measured
Lease holder Which worker revision holds the lease db/prod-clean.mjs and the "worker lost its lease" alert
Oldest waiting job Minutes the oldest interactive job has waited alerts.js queue-stuck at 10 minutes
Background spend per brand per week Paid work booked to a brand by timers Proposed: the spend ledger

Open

  1. Are these cadences right for paying partners? They were set during building. Proposal: keep them for the design partnership and review after a month of spend data.
  2. Should weekly work on production run at a fixed hour? Today each job's first run depends on when the worker started. Proposal: no change until a partner asks for a fixed time.
Sources
  • brew origin/main a559de6: roles.js:1-40, jobs.js:1-13, retire.js:1-9, connectors/index.js:37-55, 498, moves-cadence.js:406, 611, drafts-bg.js:142, 523-524, social-feed.js:274, 391, facts-desk.js:380, digest.js:1-12, 210-211, confirm-remind.js:1-12, 137, live-watch.js:1-6, market-pulse.js:1-11, listen-watch.js:1-10, alerts.js:90, server.js:520, 562, 594, 731-732, rewrite-pot.js:1-6, build.js:1070-1076, tools/prod-env.sh:9
  • brew/.wt-prod/docs/production.md:10, 40
  • memory brew-connected-facts, brew-move-freshness, brew-real-connectors, brew-prod-backend, brew-storage
  • handbook content/engineering/architecture.md

Source file content/ops-and-infrastructure/background-jobs.md · For agents: Markdown · JSON