Answer
In v1 the product runs no experiments, on brands or on itself; the maths for fair tests already exists, and v3 brings tests and holdouts that run through the brand's own tools, so a result says what the Play did and not what the season did.
Today
- Measurement and results are cut, so no test is run or read. brew/SCOPE.md:39
- Sending to a brand's customers is cut, holdouts included. brew/SCOPE.md:38
- The methods exist, strongest first: holdout against a control group, launched against untouched, tagged sales only, before and after. brew/measure-stats.js:1-23
- Every estimate is a difference against something with a bootstrap range, seeded so it never wobbles on reload. brew/measure-stats.js:1-9
- A method needs at least 3 points a side, or 5 tagged outcomes. brew/measure-stats.js:48-50
- Verdicts: Win, Helped short of target, Likely helped, No effect, Inconclusive, Too early to call, Hurt, Not shipped; Inconclusive never teaches "this does not work". brew/measure-stats.js:93-94
- Each family names which comparisons are possible; paid ads, site and messages can reach a holdout. brew/data/measure-families.json
- Site and checkout tests decide once their sample of 1000 is reached, in whole weeks, never before day 14, at most day 56. brew/measure-families.js:11-40
- A brief's How we measure it can describe a holdout split the team runs; the product only writes the plan. brew/measure-spec.js:16-23, 86
- The product shows one version, then Redo it, never three options at once. memory brew-one-version handbook content/company-and-strategy/decisions-log.md:37
Tomorrow
| Item | Version | Tag |
|---|---|---|
| No experiments; the brief carries the team's own test plan | v1 | Stays |
| Posting with a person's yes makes a launch date exact, the first input a test needs | v2 | New |
| Holdouts through the brand's own Klaviyo or WhatsApp tool, never our sending | v3 | New, proposed |
| Launched against untouched for site, price and listing Plays | v3 | New |
| A test decides only on the family's decide day | v3 | New |
| Only A and B results teach ranking | v3 | New, proposed |
| Ad rules hold back a share of spend as a control when the brand allows it | v4 | New, proposed |
| Product tests on our own pages | v3 | New, proposed, only with enough active brands |
Proposed rules for v3
- One test per number per brand at a time, so two launches never share a result.
- A holdout is declared at launch and locked; only a declared holdout earns grade A. memory feed-measurement-engine
- The control group gets nothing extra; the default share is 10%, and the brand may change it.
- An early stop rule pauses a test only when the family names one, and a person says yes.
- A result shows its method, sample, range and grade beside the number.
How we do it
Nothing to run in v1. When v3 starts:
- Pick the family; read its comparisons and decide day.
- Write the test into the brief before launch: the metric, the groups, the decide day.
- Read results only from fresh, real account rows.
Numbers
| Metric | Definition | Where measured | Status |
|---|---|---|---|
| Tests running | Declared tests not yet decided | Launch records | v3, Proposed |
| Holdout share | Share of results at grade A | Launch records | v3, Proposed |
| Inconclusive share | Share of decided tests with no clear answer | Launch records | v3, Proposed |
Open
- Holdouts through the brand's own tools? Proposal: yes, Klaviyo and the brand's WhatsApp tool first, never our own sending. handbook content/company-and-strategy/open-questions.md:39
- Test our own product before v3? Proposal: no; with five partners, compare weeks before and after a change and ask them.
- Default holdout share? Proposal: 10%, shown to the brand before launch.