{
 "id": "5.12",
 "title": "Testing and QA",
 "department": "Engineering",
 "department_slug": "engineering",
 "direction": null,
 "branded": false,
 "status": "draft",
 "owner": "Param",
 "last_verified": "2026-10-26",
 "holds": "The suite, honesty tests, layout audit, QA rounds",
 "urls": {
  "html": "https://handbook.joinsquareup.com/engineering/testing-and-qa",
  "markdown": "https://handbook.joinsquareup.com/engineering/testing-and-qa.md",
  "json": "https://handbook.joinsquareup.com/engineering/testing-and-qa.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "One free test suite guards every deploy, and audits check every brand",
  "points": [
   {
    "icon": "check",
    "text": "215 test files run free with AI_MOCK, and deploy.sh refuses on a failure."
   },
   {
    "icon": "gear",
    "text": "GitHub Actions runs the same gate on every push and pull request."
   },
   {
    "icon": "eye",
    "text": "Audit tools check saved data for every brand, not just one."
   },
   {
    "icon": "flag",
    "text": "A QV round runs after each phase, before the next begins."
   },
   {
    "icon": "money",
    "text": "Paid probes run only on purpose, with Param's yes."
   }
  ],
  "stat": {
   "value": "215",
   "label": "test files"
  },
  "decide": "Should a red CI run block a deploy? Proposal: yes, deploy.sh stops on red.",
  "mood": "solid"
 },
 "sources": [
  "brew origin/main d6154b3: test/ (215 test files, 4 paid probes), deploy.sh:13-16, README.md 'Test', CLAUDE.md",
  "brew origin/main d6154b3: .github/workflows/test.yml:1-37, listen-watch.js:81, reader/util.js:179, test/brief-staff.test.js, test/brand-wipe-pg.test.js, test/product-name.test.js",
  "brew commits fccf3cb, 4799f78; GitHub Actions run https://github.com/wulfor24/brew/actions/runs/40286754218",
  "brew origin/main: tools/audit-layout.mjs, audit-card-fit.mjs, audit-cards.mjs, audit-brand-book.mjs, meta-audit.mjs, shopify-audit.mjs, probe-sources.mjs; db/prod-clean.mjs; db/parity.mjs",
  "brew origin/main: docs/handover/engineering/08-testing-and-qa.md (15 Oct); docs/audit-2026-10-24-design-partners.md; DUMMY.md 'Rules that keep a dummy from passing as real'",
  "memory brew-qa-sweep-2026-10-19, brew-qa-audit-2026-10-20, brew-qa-round2-2026-10-24, brew-qv-process, brew-prod-backend, brew-v1-creative-stage, brew-jam-rca-2026-10-26, fix-as-product, feed-for-all-brands"
 ],
 "open_questions": [
  "**Should a red CI run block a deploy?** Proposal: yes, `deploy.sh` checks the commit's Actions status first and stops on red.",
  "**Who owns QA rounds once Sai and Arvind work in the code?** Proposal: whoever ships a phase runs its QV round; Param signs off the three final rounds."
 ],
 "parts": {
  "Answer": "Brew is guarded by one free test suite that every deploy runs, a set of honesty and brand-matrix tests, audit tools that check saved data for every brand, and QA rounds run after each phase; this page settles what each catches and when it must run.",
  "Today": "### The suite\n\n- Run: `AI_MOCK=1 node --test test/*.test.js`. Free: no AI, no network. [src: brew/README.md]\n- 215 test files and about 1,355 `test()` calls on origin/main d6154b3 (counted from source). The jam audit closed on 25 Oct with 1,173 tests passing. [src: brew/test/] [src: memory brew-v1-creative-stage]\n- `deploy.sh` runs it and refuses to deploy on a failure; a failed run is retried once after 20 s because two sessions may run suites side by side. [src: brew/deploy.sh:13-16]\n- `tools/deploy-demo.sh` runs no tests. [src: brew/docs/handover/engineering/07-deploy-and-ops.md]\n- Postgres tests (`*-pg`, `qv1-*`, `qv2-*`, `phase3-pg`) run against a local Postgres through `test/pg-helper.js`, which starts a throwaway Postgres 17 cluster per file. [src: brew/test/pg-helper.js] [src: brew/.github/workflows/test.yml:22-23]\n\n### CI\n\n- GitHub Actions workflow `test` runs the same gate as `deploy.sh` on every push and every pull request: Ubuntu 24.04, Node 22 as the Dockerfile, `npm ci`, Postgres 17 binaries, then `node --test test/*.test.js` with `AI_MOCK=1`. A newer push cancels the older run; a run stops after 30 minutes. [src: brew/.github/workflows/test.yml:1-37]\n- It deploys nothing. [src: brew/.github/workflows/test.yml:9-37]\n- First green run: https://github.com/wulfor24/brew/actions/runs/40286754218. [src: GitHub Actions run 40286754218]\n- CI found that `timed()` in `listen-watch.js` and `withTimeout()` in `reader/util.js` let a job or a test file exit mid run when a source never settled; the timer is now held and cleared when the work settles. [src: brew commit 4799f78] [src: brew/listen-watch.js:81] [src: brew/reader/util.js:179]\n- Paid probes, run only on purpose with Param's yes: `cadence-probe.mjs`, `ideas-probe.mjs`, `listen-probe.mjs`, `memory-probe.mjs`. [src: brew/test/] [src: memory no-spend-while-building]\n\n### Tests that hold a rule\n\n| Test | Rule it holds | Source |\n|---|---|---|\n| `honesty.test.js` | A dummy never passes as real: simulated rows never evidence, no canned sign-offs, healed old wording | [src: brew/DUMMY.md] |\n| `non-shopify.test.js` | The brand matrix: platforms, enquiry stores, markets, names | [src: brew/CLAUDE.md] |\n| `brand-book-guards.test.js` | The 23 Oct Brand Book issue classes never return | [src: brew/CLAUDE.md] |\n| `no-undef-consts.test.js` | Fails on any undeclared ALL_CAPS name (a missing import once shipped) | [src: memory brew-one-version] |\n| `qv1-*`, `qv2-*`, `qv3-guards` | Storage, roles, queue, cache, HTTP and promote findings from the 25 Oct QV rounds | [src: memory brew-prod-backend] |\n| `one-stage.test.js`, `setup-tab.test.js` | The jam stage contract and Set-up | [src: memory brew-one-stage] [src: memory brew-setup-v2] |\n| `private-leaks.test.js`, `access.test.js`, `qa-final-access.test.js` | Team-only data never reaches visitors | [src: brew/test/] |\n| `make-claim.test.js` | One paid make per board across instances | [src: git b91e364] |\n| `product-name.test.js` | Customer templates never say ShopOS or moves, name the product only from `product-name.js`, and every served page has a language and a description | [src: brew/test/product-name.test.js:51-84] |\n| `brief-staff.test.js` | Only our team moves a brief on; the client's steps match the server's; the controls never show to a brand member, guest or the demo | [src: brew/test/brief-staff.test.js:31-94] |\n| `brand-wipe-pg.test.js` | Reset and delete: dry run removes nothing, partners refused, connections unchanged, audit row written, bucket scripts refuse on Postgres | [src: brew/test/brand-wipe-pg.test.js:46-142] |\n\n### Audit tools (saved data, every brand)\n\n| Tool | Checks | Source |\n|---|---|---|\n| `tools/audit-card-fit.mjs DIR [--before]` | Wrong medium, empty asset lines, button and asset fit, old vs new code | [src: memory brew-card-fit-and-counts] |\n| `tools/audit-brand-book.mjs DIR` | Brand Book issue classes in every scope | [src: brew/CLAUDE.md] |\n| `tools/audit-layout.mjs <links>` | The jam rendered at 1000×600, 1280×800 and 390×844, every tab | [src: memory brew-jam-rca-2026-10-26] |\n| `tools/meta-audit.mjs`, `tools/shopify-audit.mjs` | What Brew shows against the platform, read only | [src: memory brew-meta-connector-qa] |\n| `tools/probe-sources.mjs` | treg and Apify sources on real brands | [src: memory brew-apify-watch] |\n| `.claude/skills/brew-external-data/audit.py` | Errors shown as none, outdated stage versions | [src: brew/CLAUDE.md] |\n| `db/prod-clean.mjs` | Production settings and data are clean | [src: brew/db/prod-clean.mjs:1-6] |\n| `db/parity.mjs` | Bucket and database agree | [src: memory brew-prod-backend] |\n\n### QA rounds so far\n\n| Date | Round | Found and fixed | Source |\n|---|---|---|---|\n| 15 Oct | Handover audits A to F | 96 feature rows; P0 honesty leaks fixed the same day | [src: brew/docs/handover/engineering/02-feature-audit.md] |\n| 19 to 20 Oct | Sai's QA sheet | 49 bugs, then about 35 half-done fixes | [src: memory brew-qa-sweep-2026-10-19] |\n| 20 Oct | Four auditors, 11 brands | about 86 issues | [src: memory brew-qa-audit-2026-10-20] |\n| 22 to 25 Oct | Jam audits | about 60, then about 50, then the close-out | [src: memory brew-v1-creative-stage] |\n| 24 Oct | Round 2 report and design-partner audit | sections A, B, C approved and fixed | [src: memory brew-qa-round2-2026-10-24] [src: brew/docs/audit-2026-10-24-design-partners.md] |\n| 25 Oct | Production backend QV, phases 1 to 6 plus 3 final rounds | about 45, about 18, then about 35 | [src: memory brew-prod-backend] |\n| 26 Oct | Jam RCA from Arata screenshots | five root causes | [src: memory brew-jam-rca-2026-10-26] |\n\n### What still has no test\n\nRoute gating has no single route table to test against; ranking (`score`, `rank`, `feed`) has no fixtures from several brands; few client files have a test, `brief-staff.test.js` among them; there is no lint or type check, and CI runs only the suite. [src: brew/docs/handover/engineering/08-testing-and-qa.md] [src: brew/.github/workflows/test.yml:35-37]",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| The free suite as the deploy gate | v1 | Stays |\n| A GitHub Action running `npm ci` and the suite on every push and pull request, live since 26 Oct | v1 | Stays |\n| `node --check` and a lint step in the same Action | v1 | New, proposed |\n| A route table plus one test of every route's auth level | v2 | New |\n| Ranking fixtures from the five partners | v2 | New |\n| Tests run before the demo deploy | v1 | Changes |\n| Headless smoke of the feed and jam on staging after each deploy | v2 | New |",
  "How we do it": "1. Every bug: fix the shared cause, add a test or audit that would have caught it, heal saved data, deploy. [src: memory fix-as-product]\n2. Before saying something works, check it on several brands and card types, and the feed view (renders, scrolls, opens, no console errors). [src: memory feed-for-all-brands]\n3. After each phase of phased work: an independent review of the diff, live checks on staging, fixes with tests, redeploy, verify. Three more full rounds before closing. [src: memory brew-qv-process]\n4. A local `.brands/` left in a worktree breaks `feed-engagement.test.js`; remove it or symlink the real one before deploying. [src: memory brew-v1-creative-stage]\n5. After a deploy, reload the page before judging; hash navigation keeps the old app.js. [src: memory brew-one-stage]",
  "Numbers": "| Metric | Definition | Where measured |\n|---|---|---|\n| Suite size | Test files and passing tests | `node --test` output at deploy |\n| Suite time | Wall time of the suite | deploy log, and each GitHub Actions run |\n| CI pass rate | Share of Actions runs on main that pass | GitHub Actions, workflow `test`, Proposed |\n| Audit findings | Findings per audit tool run on the five partners | tool output (Proposed to log weekly) |\n| Escaped bugs | Bugs found on production that a test should have caught | Proposed |",
  "Open": "1. **Should a red CI run block a deploy?** Proposal: yes, `deploy.sh` checks the commit's Actions status first and stops on red.\n2. **Who owns QA rounds once Sai and Arvind work in the code?** Proposal: whoever ships a phase runs its QV round; Param signs off the three final rounds."
 }
}