Answer
There is no eval suite for model output today; this document proposes one fixed set of test brands, the measures, the cadence and the pass bar that decide every model switch and every prompt change, and waits for Param to accept it.
Today
- No eval suite exists. Nothing scores Play, card, draft or picture quality on a fixed set over time.
- What exists instead:
| What | What it covers | Source |
|---|---|---|
Unit and guard tests, 211 test files, run on every deploy with AI_MOCK=1 |
Guards, parsing, routing, budgets; no model output | brew/deploy.sh:12-16 brew/test/ |
| Audit tools | Cards, card fit, Brand Book, connectors, sources, run by hand over bucket copies | brew/tools/audit-cards.mjs brew/tools/audit-brand-book.mjs brew/tools/probe-sources.mjs |
| One model comparison, 20 Oct | 54 Arata Play pairs, Pro against Opus, read by hand | memory brew-model-compare-2026-10-20 |
| QA rounds by fresh reviewers | 86 findings on 20 Oct, report A, B, C on 24 Oct | memory brew-qa-audit-2026-10-20 memory brew-qa-round2-2026-10-24 |
| Picture check history | Of 54 pictures thrown away on 16 Oct, 47 showed the right product | brew/server.js:218 |
What the 20 Oct comparison measured, the closest thing to an eval so far memory brew-model-compare-2026-10-20:
| Per Play, median | Gemini 3.1 Pro | Claude Opus 5.5 |
|---|---|---|
| Cost | $0.18 | $0.83 |
| Time | 2.4 minutes | 5.7 minutes |
| Words | about 2,250 | about 5,450 |
| Facts | 10 | 54 |
| Sources | 20 | 62 |
| Total for 54 Plays | $9.9 | $46.7 |
- It was confounded: Opus had the team's 19 Oct audit upload and Pro did not. A fair rerun of about 5 Plays for about $1 was offered and not run memory brew-model-compare-2026-10-20.
- A bake-off of the same 5 Plays on Opus, Pro, Flash and 2 open models, blind-scored, about $5 to $10, was proposed and not run memory brew-model-compare-2026-10-20.
Tomorrow
The proposed eval set. Every number in it is Proposed until Param accepts it.
Test brands, frozen
Five brand packs copied from the bucket once and frozen, so runs compare. Evals read copies only and never write to a live brand memory brew-live-stores.
| Brand | Why it is in the set |
|---|---|
| Arata, arata-in | Hair care, India, claim-heavy category, the most history and the Opus reference Plays |
| Nutrova, nutrova-com | Supplements, India, health claims, a different category |
| Puma India, in-puma-com | A large catalog and a famous brand, apparel |
| Scrubroom, scrubroom-in | Medical apparel, a small catalog, enquiry-like buyers |
| Cozy Snugglet, cozysnugglet-shop | Our own test brand, non-India market, safe for paid make tests |
What we measure
| Area | Measure | How | Pass bar, Proposed |
|---|---|---|---|
| Plays | Blind pairwise win rate against the current model | 10 fixed Plays per brand, two readers, Param and one of Sai or Arvind, score which is more useful to the founder | New model wins or ties 60% of pairs |
| Plays | Brand-specific facts per Play | Count of facts tied to the brand's ledger | Not below the current model's median |
| Plays | Repeated stat rate | One stat on more than 2 Plays of a brand | At most 1 in 10 Plays |
| Plays | Unbacked numbers after the Play check | Fact gate on the saved Play | 0 |
| Plays | "To confirm" leaking mid-sentence | String check | 0 |
| Cards and drafts | Claim flags removed per 100 drafts | Clean and Check copy return values | Falls or holds |
| Cards and drafts | Writer slips on first draft | writerSlips before mend |
Under 10% of packages |
| Guards | Claim guard recall and precision | A labelled corpus of 200 sentences from real incidents and real store copy, half backed and half not | Recall 95%, precision 90% |
| Pictures | First-try look pass | Look judge GOOD on first make | 80% |
| Pictures | Judge agreement with a person | 40 pictures a run labelled by hand | 90% |
| Pictures | Wrong product shown | Hand check of the same 40 | 0 |
| Video finals | On-screen spelling pass | text-check | 90% |
| Reader | Category and one-line description right | Hand label once per test brand | 5 of 5 |
| Reader | Rivals right | At least 3 of the brand's real rivals, labelled once | 4 of 5 brands |
| Cost and time | Dollars and minutes per Play and per build | meta.usd, build timings |
Within the caps in Cost per brand |
Cadence
| Run | When | Cost, Proposed |
|---|---|---|
| Guard corpus | Every deploy, inside the test suite, no model calls | free |
| Play bake-off | Before any change of the Plays model or the Play prompt, and once a month | about $5 to $10 per run, from the 20 Oct estimate memory brew-model-compare-2026-10-20 |
| Card and draft run | Weekly, on the frozen packs | Proposed, to measure on the first run |
| Picture run | Weekly, 40 pictures across the five brands | about $1.60 at $0.04 a draft picture brew/data/engines.json |
| Reader run | When a reader stage changes | Proposed, to measure on the first run |
Versions
| Item | Version | Tag |
|---|---|---|
| Frozen test brands and the guard corpus | v1 | New |
| Play bake-off as the gate for any model switch | v1 | New |
| Weekly picture and card runs with a one-page result | v1 | New |
| Evals for posting and scheduling: right channel, right time, guards passed | v2 | New |
| Evals that compare predicted impact with measured results | v3 | New |
| Evals for ad changes inside the brand's rules | v4 | New |
How we do it
Proposed runbook, once accepted:
- Freeze the packs once, read only:
mkdir -p $TMPDIR/evals && for s in arata-in nutrova-com in-puma-com scrubroom-in cozysnugglet-shop; do gsutil -m cp -r gs://brew-prototype/brands/$s $TMPDIR/evals/; done - Run the candidate locally against the copies with
BRANDS_DIRset to the frozen folder, never against the live bucket. - Strip model names, shuffle pairs, score blind, then unblind.
- Write the result into Numbers below, with the date and the commit.
- Ask Param before any run that spends; follow the no-spend rule while building memory no-spend-while-building.
Numbers
| Metric | Definition | Where measured | Status |
|---|---|---|---|
| Play win rate | Share of blind pairs won or tied by a candidate | eval sheet | Proposed |
| Guard recall and precision | On the labelled claim corpus | test suite | Proposed |
| First-try look pass | GOOD on first make | picture run | Proposed |
| Judge agreement | Judge verdict equals a person's label | picture run | Proposed |
| Cost per Play | Median dollars | Play meta | Measured once: $0.18 Pro, $0.83 Opus on Arata memory brew-model-compare-2026-10-20 |
Open
- Accept this eval set? Proposal: accept as written, run the first Play bake-off on 5 Arata Plays across Opus, Pro and Flash before deciding the Plays model.
- Who scores blind pairs? Proposal: Param and one of Sai or Arvind each score every pair; a tie between them counts as a tie.
- Open models in the bake-off? Proposal: leave them out of the first run; add one when the gates need a cheaper classifier.
- Use real partner packs? Proposal: yes, as frozen read-only copies that never leave our machines and never write back.
- Eval budget. Proposal: $20 a month while building, from the Google credits for Gemini runs, with your yes before any Claude run.