---
id: "7.7"
title: "Experiments"
dept: data-and-analytics
status: draft
owner: Param
branded: false
last_verified: 2026-10-26
sources:
  - "brew/SCOPE.md:38-39"
  - "brew/measure-stats.js:1-23, 48-64, 93-94"
  - "brew/measure-families.js:11-40"
  - "brew/data/measure-families.json"
  - "brew/measure-spec.js:16-23, 86"
  - "memory feed-measurement-engine, brew-one-version"
  - "handbook content/company-and-strategy/open-questions.md:37-39, content/company-and-strategy/decisions-log.md:36-37"
---

# Experiments

## Answer

In v1 the product runs no experiments, on brands or on itself; the maths for fair tests already exists, and v3 brings tests and holdouts that run through the brand's own tools, so a result says what the Play did and not what the season did.

## Today

- Measurement and results are cut, so no test is run or read. [src: brew/SCOPE.md:39]
- Sending to a brand's customers is cut, holdouts included. [src: brew/SCOPE.md:38]
- The methods exist, strongest first: holdout against a control group, launched against untouched, tagged sales only, before and after. [src: brew/measure-stats.js:1-23]
- Every estimate is a difference against something with a bootstrap range, seeded so it never wobbles on reload. [src: brew/measure-stats.js:1-9]
- A method needs at least 3 points a side, or 5 tagged outcomes. [src: brew/measure-stats.js:48-50]
- Verdicts: Win, Helped short of target, Likely helped, No effect, Inconclusive, Too early to call, Hurt, Not shipped; Inconclusive never teaches "this does not work". [src: brew/measure-stats.js:93-94]
- Each family names which comparisons are possible; paid ads, site and messages can reach a holdout. [src: brew/data/measure-families.json]
- Site and checkout tests decide once their sample of 1000 is reached, in whole weeks, never before day 14, at most day 56. [src: brew/measure-families.js:11-40]
- A brief's How we measure it can describe a holdout split the team runs; the product only writes the plan. [src: brew/measure-spec.js:16-23, 86]
- The product shows one version, then Redo it, never three options at once. [src: memory brew-one-version] [src: handbook content/company-and-strategy/decisions-log.md:37]

## Tomorrow

| Item | Version | Tag |
|---|---|---|
| No experiments; the brief carries the team's own test plan | v1 | Stays |
| Posting with a person's yes makes a launch date exact, the first input a test needs | v2 | New |
| Holdouts through the brand's own Klaviyo or WhatsApp tool, never our sending | v3 | New, proposed |
| Launched against untouched for site, price and listing Plays | v3 | New |
| A test decides only on the family's decide day | v3 | New |
| Only A and B results teach ranking | v3 | New, proposed |
| Ad rules hold back a share of spend as a control when the brand allows it | v4 | New, proposed |
| Product tests on our own pages | v3 | New, proposed, only with enough active brands |

### Proposed rules for v3

1. One test per number per brand at a time, so two launches never share a result.
2. A holdout is declared at launch and locked; only a declared holdout earns grade A. [src: memory feed-measurement-engine]
3. The control group gets nothing extra; the default share is 10%, and the brand may change it.
4. An early stop rule pauses a test only when the family names one, and a person says yes.
5. A result shows its method, sample, range and grade beside the number.

## How we do it

Nothing to run in v1. When v3 starts:

1. Pick the family; read its comparisons and decide day.
2. Write the test into the brief before launch: the metric, the groups, the decide day.
3. Read results only from fresh, real account rows.

## Numbers

| Metric | Definition | Where measured | Status |
|---|---|---|---|
| Tests running | Declared tests not yet decided | Launch records | v3, Proposed |
| Holdout share | Share of results at grade A | Launch records | v3, Proposed |
| Inconclusive share | Share of decided tests with no clear answer | Launch records | v3, Proposed |

## Open

1. **Holdouts through the brand's own tools?** Proposal: yes, Klaviyo and the brand's WhatsApp tool first, never our own sending. [src: handbook content/company-and-strategy/open-questions.md:39]
2. **Test our own product before v3?** Proposal: no; with five partners, compare weeks before and after a change and ask them.
3. **Default holdout share?** Proposal: 10%, shown to the brand before launch.