{
 "id": "7.7",
 "title": "Experiments",
 "department": "Data and analytics",
 "department_slug": "data-and-analytics",
 "direction": null,
 "branded": false,
 "status": "draft",
 "owner": "Param",
 "last_verified": "Mon Oct 26",
 "holds": "Tests and holdouts in v3",
 "urls": {
  "html": "https://handbook.joinsquareup.com/data-and-analytics/experiments",
  "markdown": "https://handbook.joinsquareup.com/data-and-analytics/experiments.md",
  "json": "https://handbook.joinsquareup.com/data-and-analytics/experiments.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "No experiments in v1, but the fair-test maths is already built",
  "points": [
   {
    "icon": "lock",
    "text": "Measurement and sending to a brand's customers are both cut from v1."
   },
   {
    "icon": "chart",
    "text": "Every estimate is a difference with a seeded range, so it never wobbles on reload."
   },
   {
    "icon": "flag",
    "text": "Site and checkout tests wait for a sample of 1000, never before day 14."
   },
   {
    "icon": "rocket",
    "text": "v3 adds holdouts through the brand's own Klaviyo or WhatsApp tool."
   }
  ],
  "stat": {
   "value": "1000",
   "label": "Sample before site tests decide"
  },
  "decide": "Run holdouts through the brand's own tools, and at what default share?",
  "mood": "waiting"
 },
 "sources": [
  "brew/SCOPE.md:38-39",
  "brew/measure-stats.js:1-23, 48-64, 93-94",
  "brew/measure-families.js:11-40",
  "brew/data/measure-families.json",
  "brew/measure-spec.js:16-23, 86",
  "memory feed-measurement-engine, brew-one-version",
  "handbook content/company-and-strategy/open-questions.md:37-39, content/company-and-strategy/decisions-log.md:36-37"
 ],
 "open_questions": [
  "**Holdouts through the brand's own tools?** Proposal: yes, Klaviyo and the brand's WhatsApp tool first, never our own sending. [src: handbook content/company-and-strategy/open-questions.md:39]",
  "**Test our own product before v3?** Proposal: no; with five partners, compare weeks before and after a change and ask them.",
  "**Default holdout share?** Proposal: 10%, shown to the brand before launch."
 ],
 "parts": {
  "Answer": "In v1 the product runs no experiments, on brands or on itself; the maths for fair tests already exists, and v3 brings tests and holdouts that run through the brand's own tools, so a result says what the Play did and not what the season did.",
  "Today": "- Measurement and results are cut, so no test is run or read. [src: brew/SCOPE.md:39]\n- Sending to a brand's customers is cut, holdouts included. [src: brew/SCOPE.md:38]\n- The methods exist, strongest first: holdout against a control group, launched against untouched, tagged sales only, before and after. [src: brew/measure-stats.js:1-23]\n- Every estimate is a difference against something with a bootstrap range, seeded so it never wobbles on reload. [src: brew/measure-stats.js:1-9]\n- A method needs at least 3 points a side, or 5 tagged outcomes. [src: brew/measure-stats.js:48-50]\n- Verdicts: Win, Helped short of target, Likely helped, No effect, Inconclusive, Too early to call, Hurt, Not shipped; Inconclusive never teaches \"this does not work\". [src: brew/measure-stats.js:93-94]\n- Each family names which comparisons are possible; paid ads, site and messages can reach a holdout. [src: brew/data/measure-families.json]\n- Site and checkout tests decide once their sample of 1000 is reached, in whole weeks, never before day 14, at most day 56. [src: brew/measure-families.js:11-40]\n- A brief's How we measure it can describe a holdout split the team runs; the product only writes the plan. [src: brew/measure-spec.js:16-23, 86]\n- The product shows one version, then Redo it, never three options at once. [src: memory brew-one-version] [src: handbook content/company-and-strategy/decisions-log.md:37]",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| No experiments; the brief carries the team's own test plan | v1 | Stays |\n| Posting with a person's yes makes a launch date exact, the first input a test needs | v2 | New |\n| Holdouts through the brand's own Klaviyo or WhatsApp tool, never our sending | v3 | New, proposed |\n| Launched against untouched for site, price and listing Plays | v3 | New |\n| A test decides only on the family's decide day | v3 | New |\n| Only A and B results teach ranking | v3 | New, proposed |\n| Ad rules hold back a share of spend as a control when the brand allows it | v4 | New, proposed |\n| Product tests on our own pages | v3 | New, proposed, only with enough active brands |\n\n### Proposed rules for v3\n\n1. One test per number per brand at a time, so two launches never share a result.\n2. A holdout is declared at launch and locked; only a declared holdout earns grade A. [src: memory feed-measurement-engine]\n3. The control group gets nothing extra; the default share is 10%, and the brand may change it.\n4. An early stop rule pauses a test only when the family names one, and a person says yes.\n5. A result shows its method, sample, range and grade beside the number.",
  "How we do it": "Nothing to run in v1. When v3 starts:\n\n1. Pick the family; read its comparisons and decide day.\n2. Write the test into the brief before launch: the metric, the groups, the decide day.\n3. Read results only from fresh, real account rows.",
  "Numbers": "| Metric | Definition | Where measured | Status |\n|---|---|---|---|\n| Tests running | Declared tests not yet decided | Launch records | v3, Proposed |\n| Holdout share | Share of results at grade A | Launch records | v3, Proposed |\n| Inconclusive share | Share of decided tests with no clear answer | Launch records | v3, Proposed |",
  "Open": "1. **Holdouts through the brand's own tools?** Proposal: yes, Klaviyo and the brand's WhatsApp tool first, never our own sending. [src: handbook content/company-and-strategy/open-questions.md:39]\n2. **Test our own product before v3?** Proposal: no; with five partners, compare weeks before and after a change and ask them.\n3. **Default holdout share?** Proposal: 10%, shown to the brand before launch."
 }
}