---
id: "5.12"
title: "Testing and QA"
dept: engineering
status: draft
owner: Param
branded: false
last_verified: "2026-10-26"
sources:
  - "brew origin/main d6154b3: test/ (215 test files, 4 paid probes), deploy.sh:13-16, README.md 'Test', CLAUDE.md"
  - "brew origin/main d6154b3: .github/workflows/test.yml:1-37, listen-watch.js:81, reader/util.js:179, test/brief-staff.test.js, test/brand-wipe-pg.test.js, test/product-name.test.js"
  - "brew commits fccf3cb, 4799f78; GitHub Actions run https://github.com/wulfor24/brew/actions/runs/40286754218"
  - "brew origin/main: tools/audit-layout.mjs, audit-card-fit.mjs, audit-cards.mjs, audit-brand-book.mjs, meta-audit.mjs, shopify-audit.mjs, probe-sources.mjs; db/prod-clean.mjs; db/parity.mjs"
  - "brew origin/main: docs/handover/engineering/08-testing-and-qa.md (15 Oct); docs/audit-2026-10-24-design-partners.md; DUMMY.md 'Rules that keep a dummy from passing as real'"
  - "memory brew-qa-sweep-2026-10-19, brew-qa-audit-2026-10-20, brew-qa-round2-2026-10-24, brew-qv-process, brew-prod-backend, brew-v1-creative-stage, brew-jam-rca-2026-10-26, fix-as-product, feed-for-all-brands"
---

# Testing and QA

## Answer

Brew is guarded by one free test suite that every deploy runs, a set of honesty and brand-matrix tests, audit tools that check saved data for every brand, and QA rounds run after each phase; this page settles what each catches and when it must run.

## Today

### The suite

- Run: `AI_MOCK=1 node --test test/*.test.js`. Free: no AI, no network. [src: brew/README.md]
- 215 test files and about 1,355 `test()` calls on origin/main d6154b3 (counted from source). The jam audit closed on 25 Oct with 1,173 tests passing. [src: brew/test/] [src: memory brew-v1-creative-stage]
- `deploy.sh` runs it and refuses to deploy on a failure; a failed run is retried once after 20 s because two sessions may run suites side by side. [src: brew/deploy.sh:13-16]
- `tools/deploy-demo.sh` runs no tests. [src: brew/docs/handover/engineering/07-deploy-and-ops.md]
- Postgres tests (`*-pg`, `qv1-*`, `qv2-*`, `phase3-pg`) run against a local Postgres through `test/pg-helper.js`, which starts a throwaway Postgres 17 cluster per file. [src: brew/test/pg-helper.js] [src: brew/.github/workflows/test.yml:22-23]

### CI

- GitHub Actions workflow `test` runs the same gate as `deploy.sh` on every push and every pull request: Ubuntu 24.04, Node 22 as the Dockerfile, `npm ci`, Postgres 17 binaries, then `node --test test/*.test.js` with `AI_MOCK=1`. A newer push cancels the older run; a run stops after 30 minutes. [src: brew/.github/workflows/test.yml:1-37]
- It deploys nothing. [src: brew/.github/workflows/test.yml:9-37]
- First green run: https://github.com/wulfor24/brew/actions/runs/40286754218. [src: GitHub Actions run 40286754218]
- CI found that `timed()` in `listen-watch.js` and `withTimeout()` in `reader/util.js` let a job or a test file exit mid run when a source never settled; the timer is now held and cleared when the work settles. [src: brew commit 4799f78] [src: brew/listen-watch.js:81] [src: brew/reader/util.js:179]
- Paid probes, run only on purpose with Param's yes: `cadence-probe.mjs`, `ideas-probe.mjs`, `listen-probe.mjs`, `memory-probe.mjs`. [src: brew/test/] [src: memory no-spend-while-building]

### Tests that hold a rule

| Test | Rule it holds | Source |
|---|---|---|
| `honesty.test.js` | A dummy never passes as real: simulated rows never evidence, no canned sign-offs, healed old wording | [src: brew/DUMMY.md] |
| `non-shopify.test.js` | The brand matrix: platforms, enquiry stores, markets, names | [src: brew/CLAUDE.md] |
| `brand-book-guards.test.js` | The 23 Oct Brand Book issue classes never return | [src: brew/CLAUDE.md] |
| `no-undef-consts.test.js` | Fails on any undeclared ALL_CAPS name (a missing import once shipped) | [src: memory brew-one-version] |
| `qv1-*`, `qv2-*`, `qv3-guards` | Storage, roles, queue, cache, HTTP and promote findings from the 25 Oct QV rounds | [src: memory brew-prod-backend] |
| `one-stage.test.js`, `setup-tab.test.js` | The jam stage contract and Set-up | [src: memory brew-one-stage] [src: memory brew-setup-v2] |
| `private-leaks.test.js`, `access.test.js`, `qa-final-access.test.js` | Team-only data never reaches visitors | [src: brew/test/] |
| `make-claim.test.js` | One paid make per board across instances | [src: git b91e364] |
| `product-name.test.js` | Customer templates never say ShopOS or moves, name the product only from `product-name.js`, and every served page has a language and a description | [src: brew/test/product-name.test.js:51-84] |
| `brief-staff.test.js` | Only our team moves a brief on; the client's steps match the server's; the controls never show to a brand member, guest or the demo | [src: brew/test/brief-staff.test.js:31-94] |
| `brand-wipe-pg.test.js` | Reset and delete: dry run removes nothing, partners refused, connections unchanged, audit row written, bucket scripts refuse on Postgres | [src: brew/test/brand-wipe-pg.test.js:46-142] |

### Audit tools (saved data, every brand)

| Tool | Checks | Source |
|---|---|---|
| `tools/audit-card-fit.mjs DIR [--before]` | Wrong medium, empty asset lines, button and asset fit, old vs new code | [src: memory brew-card-fit-and-counts] |
| `tools/audit-brand-book.mjs DIR` | Brand Book issue classes in every scope | [src: brew/CLAUDE.md] |
| `tools/audit-layout.mjs <links>` | The jam rendered at 1000×600, 1280×800 and 390×844, every tab | [src: memory brew-jam-rca-2026-10-26] |
| `tools/meta-audit.mjs`, `tools/shopify-audit.mjs` | What Brew shows against the platform, read only | [src: memory brew-meta-connector-qa] |
| `tools/probe-sources.mjs` | treg and Apify sources on real brands | [src: memory brew-apify-watch] |
| `.claude/skills/brew-external-data/audit.py` | Errors shown as none, outdated stage versions | [src: brew/CLAUDE.md] |
| `db/prod-clean.mjs` | Production settings and data are clean | [src: brew/db/prod-clean.mjs:1-6] |
| `db/parity.mjs` | Bucket and database agree | [src: memory brew-prod-backend] |

### QA rounds so far

| Date | Round | Found and fixed | Source |
|---|---|---|---|
| 15 Oct | Handover audits A to F | 96 feature rows; P0 honesty leaks fixed the same day | [src: brew/docs/handover/engineering/02-feature-audit.md] |
| 19 to 20 Oct | Sai's QA sheet | 49 bugs, then about 35 half-done fixes | [src: memory brew-qa-sweep-2026-10-19] |
| 20 Oct | Four auditors, 11 brands | about 86 issues | [src: memory brew-qa-audit-2026-10-20] |
| 22 to 25 Oct | Jam audits | about 60, then about 50, then the close-out | [src: memory brew-v1-creative-stage] |
| 24 Oct | Round 2 report and design-partner audit | sections A, B, C approved and fixed | [src: memory brew-qa-round2-2026-10-24] [src: brew/docs/audit-2026-10-24-design-partners.md] |
| 25 Oct | Production backend QV, phases 1 to 6 plus 3 final rounds | about 45, about 18, then about 35 | [src: memory brew-prod-backend] |
| 26 Oct | Jam RCA from Arata screenshots | five root causes | [src: memory brew-jam-rca-2026-10-26] |

### What still has no test

Route gating has no single route table to test against; ranking (`score`, `rank`, `feed`) has no fixtures from several brands; few client files have a test, `brief-staff.test.js` among them; there is no lint or type check, and CI runs only the suite. [src: brew/docs/handover/engineering/08-testing-and-qa.md] [src: brew/.github/workflows/test.yml:35-37]

## Tomorrow

| Item | Version | Tag |
|---|---|---|
| The free suite as the deploy gate | v1 | Stays |
| A GitHub Action running `npm ci` and the suite on every push and pull request, live since 26 Oct | v1 | Stays |
| `node --check` and a lint step in the same Action | v1 | New, proposed |
| A route table plus one test of every route's auth level | v2 | New |
| Ranking fixtures from the five partners | v2 | New |
| Tests run before the demo deploy | v1 | Changes |
| Headless smoke of the feed and jam on staging after each deploy | v2 | New |

## How we do it

1. Every bug: fix the shared cause, add a test or audit that would have caught it, heal saved data, deploy. [src: memory fix-as-product]
2. Before saying something works, check it on several brands and card types, and the feed view (renders, scrolls, opens, no console errors). [src: memory feed-for-all-brands]
3. After each phase of phased work: an independent review of the diff, live checks on staging, fixes with tests, redeploy, verify. Three more full rounds before closing. [src: memory brew-qv-process]
4. A local `.brands/` left in a worktree breaks `feed-engagement.test.js`; remove it or symlink the real one before deploying. [src: memory brew-v1-creative-stage]
5. After a deploy, reload the page before judging; hash navigation keeps the old app.js. [src: memory brew-one-stage]

## Numbers

| Metric | Definition | Where measured |
|---|---|---|
| Suite size | Test files and passing tests | `node --test` output at deploy |
| Suite time | Wall time of the suite | deploy log, and each GitHub Actions run |
| CI pass rate | Share of Actions runs on main that pass | GitHub Actions, workflow `test`, Proposed |
| Audit findings | Findings per audit tool run on the five partners | tool output (Proposed to log weekly) |
| Escaped bugs | Bugs found on production that a test should have caught | Proposed |

## Open

1. **Should a red CI run block a deploy?** Proposal: yes, `deploy.sh` checks the commit's Actions status first and stops on red.
2. **Who owns QA rounds once Sai and Arvind work in the code?** Proposal: whoever ships a phase runs its QV round; Param signs off the three final rounds.