{
 "id": "6.6",
 "title": "Outside data rules",
 "department": "AI and models",
 "department_slug": "ai-and-models",
 "direction": null,
 "branded": false,
 "status": "draft",
 "owner": "Param",
 "last_verified": "Mon Oct 26",
 "holds": "treg, Apify, Ad Library, fallbacks, caching, never repeat a paid call",
 "urls": {
  "html": "https://handbook.joinsquareup.com/ai-and-models/outside-data-rules",
  "markdown": "https://handbook.joinsquareup.com/ai-and-models/outside-data-rules.md",
  "json": "https://handbook.joinsquareup.com/ai-and-models/outside-data-rules.json"
 },
 "other_directions": [],
 "tldr": {
  "headline": "Pay for outside data once, cap every source, never hide errors",
  "points": [
   {
    "icon": "money",
    "text": "treg first, Apify only for gaps, and a free fallback wherever one exists."
   },
   {
    "icon": "box",
    "text": "Every paid answer is cached and shared, so the same call is paid once per window."
   },
   {
    "icon": "lock",
    "text": "Daily, monthly and per-run caps sit on every source."
   },
   {
    "icon": "warn",
    "text": "A failure is never shown as a fact: found, truly none, or could not check."
   },
   {
    "icon": "link",
    "text": "The brew-external-data skill lives outside the repo, so not every session sees it."
   }
  ],
  "stat": {
   "value": "40",
   "label": "dollar daily treg cap"
  },
  "decide": "Rotate the Apify token pasted in chat on 21 Oct, and put the skill in the repo?",
  "mood": "waiting"
 },
 "sources": [
  "brew origin/main b91e364: treg.js:1-139, apify.js:1-54, reader/adlib.js:1-86, shelf.js:1-8 and 69, listen-watch.js:10 and 684, market-pulse.js:318, live-watch.js:21-24, deploy.sh:38-41, tools/prod-env.sh:9-11",
  "skill /Users/param/Documents/ChatGPT/.claude/skills/brew-external-data/SKILL.md, with audit.py and heal-test.mjs; not on brew origin/main",
  "memory treg-usage-rules, treg-catalog-for-brew, treg-tool-catalog, brew-external-data-skill, brew-apify-watch, brew-lean-pipeline, brew-viral-remakes, brew-community-seat, no-spend-while-building"
 ],
 "open_questions": [
  "**The skill is not in the repo.** It lives in the parent folder, not on brew origin/main. Proposal: move it into brew/.claude/skills so every Brew session loads it.",
  "**One cap per brand.** treg is capped per day for all brands, Apify per brand per month. Proposal: add a per-brand monthly treg cap of $15 beside Apify's $10, both shown on the watch page.",
  "**The Apify token was pasted in chat on 21 Oct.** Proposal: rotate it [src: memory brew-viral-remakes]."
 ],
 "parts": {
  "Answer": "How Brew reads data it does not own, through treg, Apify, the public Meta Ad Library and model web search: treg first, a free fallback wherever one exists, every paid answer cached and shared so it is paid for once, a cap on every source, and a failure never shown as a fact.",
  "Today": "### The sources\n\n| Source | Used for | Client | Key | Source |\n|---|---|---|---|---|\n| treg.to | Listening, SERP, keywords, reviews, AI answers, rival ads, shopping | `treg.js`, shared, one per server | `TREG_TOKEN`, agent key brew-live, team squareup, 20,000 calls a day | [src: brew/treg.js:1-5] [src: memory brew-apify-watch] |\n| Apify | Indian marketplaces, quick commerce, Similarweb, Instagram mentions, YouTube clip downloads, Trustpilot backup | `apify.js` | `APIFY_TOKEN`, Starter plan bought 23 Oct | [src: brew/apify.js:1-4] [src: memory brew-apify-watch] |\n| Meta Ad Library page | Free fallback for every Ad Library call | `reader/adlib.js`, headless Chromium, 2 at once | none | [src: brew/reader/adlib.js:1-9] |\n| Model web search | Rivals, company facts, the Claude AI answer | ai.js `search` | the model's | [src: brew/ai.js:62] |\n| Connected accounts | GA4, Search Console, Instagram, Google Ads: free | connectors | the brand's grant | [src: memory treg-catalog-for-brew] |\n\n### The rules in code\n\n1. **treg first, Apify only where treg has nothing or is weak** [src: memory brew-apify-watch].\n2. **Data only through treg; assets never.** Images, video and voice run on our own Vertex and OpenAI keys [src: memory treg-usage-rules].\n3. **Every call goes through the shared client.** Never `fetch('https://treg.to/...')` directly; every Ad Library call goes through `adCall(treg, id, o)` [src: brew/reader/adlib.js:76] [src: memory brew-external-data-skill].\n4. **Paid once per window.** The same endpoint and parameters are paid for once per signal window, by any caller on any instance; identical calls in flight share one answer [src: brew/treg.js:11-15] [src: brew/treg.js:57].\n\n| Signal | Cache window | Source |\n|---|---|---|\n| Company, tech stack, whois | 720 hours | [src: brew/treg.js:20-27] |\n| Reviews, SERP, keywords, trends, AI answers, profiles | 168 hours | [src: brew/treg.js:20-27] |\n| Ad libraries, news, web search, shopping, prices | 24 hours | [src: brew/treg.js:20-27] |\n| Anything else | `TREG_CACHE_HOURS`, 24 | [src: brew/treg.js:17] |\n| Apify reviews actors, other actors | 168 hours, 24 hours | [src: brew/apify.js:17] |\n| Generation, job polling, downloads, failures | never cached | [src: brew/treg.js:16] |\n\n5. **No `cache: false` for read-only data.** A caller that needs fresher data passes `maxAgeH` instead [src: brew/treg.js:64-65] [src: memory treg-usage-rules].\n6. **Re-reads reuse what we have:** the read keeps the category and its search terms, and reuses the last rivals, terms and questions [src: brew/reader/stages/answers.js:39-47] [src: memory brew-lean-pipeline].\n7. **An error is never an empty result.** Three branches every time: found, truly none, could not check and retries [src: memory brew-external-data-skill]. Apify rows say a platform could not be read, never \"not sold there\" [src: brew/shelf.js:5].\n8. **A fallback must match the paid source's numbers, not just its shape.** The Ad Library fallback reads the page itself for its full count; keyword results once gave Snitch 23 ads instead of 516 [src: memory brew-external-data-skill] [src: brew/reader/adlib.js:62-75].\n9. **Every live call is measured** for source health: ok, empty or error, time and cost; the router sends work to the healthiest source first [src: brew/treg.js:76-79] [src: memory brew-apify-watch].\n10. **Saved reads heal** on open and on the check page, once a day or hourly while errors remain; a stage that reads differently bumps its version, such as `ADS_V` [src: memory brew-external-data-skill].\n11. **Brand-agnostic.** Never hardcode a brand; a bucket patch is a stopgap [src: memory brew-external-data-skill].\n12. **No captcha bypass.** The Ad Library reader stops when the page asks for a captcha [src: brew/reader/adlib.js:37] [src: brew/reader/adlib.js:54].\n\n### Caps\n\n| Cap | Value | Scope | Source |\n|---|---|---|---|\n| `TREG_DAY_CAP_USD` | $40 | all brands, every instance, a day | [src: brew/deploy.sh:41] [src: brew/tools/prod-env.sh:9] [src: brew/treg.js:82] |\n| `APIFY_BRAND_MONTH_CAP_USD` | $10 | a brand, a month | [src: brew/shelf.js:69] [src: brew/tools/prod-env.sh:9] |\n| Apify run | $0.25 by default through `maxTotalChargeUsd` | a run | [src: brew/apify.js:40] |\n| `LISTEN_RUN_CAP_USD` | $0.60 | a listening run | [src: brew/listen-watch.js:684] |\n| `PULSE_CAP_USD` | $3 | a market pulse run | [src: brew/market-pulse.js:318] |\n| Brand Book enrichment | $0.50 | a brand, weekly | [src: memory brew-brand-book] |\n\n### Preferred endpoints\n\nThe full map is `treg-map.js` on main; memory keeps the prices of 14 Oct [src: memory treg-catalog-for-brew].\n\n| Signal | Use | Avoid |\n|---|---|---|\n| TikTok search, comments | `tikhub.tiktok.search.videos`, `tikhub.tiktok.video.comments`, $0.001 | `anyapi.tiktok.ad_library_search`, 25% success |\n| Instagram reels, comments | `anyapi.instagram.search.reels`, `anyapi.instagram.post.comments` | |\n| Reddit | `tikhub.x.reddit-app-fetch-dynamic-search`, $0.001 | Reddit's own .json, 403 to our servers [src: memory brew-community-seat] |\n| Facebook comments | `scrapecreators.x.v1-facebook-post-comments` | routed `treg.facebook.post.comments`, 26% |\n| Keyword volume | `dataforseo.google.keywords.volume`, $0.09 per 1,000; free through Google Ads when connected | |\n| Trends | `dataforseo.x.keywords-data-dataforseo-trends-explore-live`, $0.0012 | serpapi, 12 times dearer |\n| AI answers | `dataforseo.x.ai-optimization-*`, $0.004 | `crawl4ai.ai-search.chatgpt.scrape`, 3%; llm-mentions at $0.10 |\n| Google ads | `adyntel.google.ads.transparency`, $0.011 | |\n| Amazon reviews | `justoneapi.x.amazon-get-product-top-reviews-v1`, country-aware | crawl4ai, which always reads amazon.com [src: memory brew-apify-watch] |\n| Nykaa, Flipkart, Myntra, Blinkit, Zepto, Instamart | Apify actors that passed the 23 Oct probe, each with a backup | treg has none, request #1171 |\n| Video understanding | download through treg or Apify, read on our Vertex | |",
  "Tomorrow": "| Item | Version | Tag |\n|---|---|---|\n| treg first, Apify for gaps, a free fallback for every paid source | v1 | Stays |\n| The brew-external-data skill moved into brew origin/main, so every session sees it | v1 | Changes |\n| One monthly outside-data cap per brand across treg and Apify, shown to the team | v2 | New |\n| Source health shown per brand on the watch page with the last good read | v1 | Changes |\n| Connected first-party data preferred over paid scraping when both exist | v2 | Changes |\n| Indian marketplaces through treg once request #1171 lands | v2 | Changes |",
  "How we do it": "1. Load the brew-external-data skill before any change that reads or shows outside data [src: memory brew-external-data-skill].\n2. Check the live balance before saying treg is empty; never trust old logs [src: memory treg-usage-rules]:\n   ```bash\n   treg balance\n   ```\n3. Find stragglers that skip the shared client:\n   ```bash\n   git -C /Users/param/Documents/ChatGPT/brew grep -n \"treg.to/call\\|adlibrary\" origin/main -- '*.js'\n   ```\n   Only treg.js, treg-map.js, adlib.js and guarded local tools should show [src: memory brew-external-data-skill].\n4. Audit every brand, then test with treg forced to fail:\n   ```bash\n   python3 /Users/param/Documents/ChatGPT/.claude/skills/brew-external-data/audit.py brew-prototype\n   node /Users/param/Documents/ChatGPT/.claude/skills/brew-external-data/heal-test.mjs <slug>\n   ```\n5. Probe a new Apify actor on real brands before using it: `node tools/probe-sources.mjs` [src: memory brew-apify-watch].\n6. Tag every call `<what>-<slug>`, so spend is booked per brand in `spend/<YYYY-MM>.json` [src: brew/treg.js:3-5] [src: brew/treg.js:121].",
  "Numbers": "| Metric | Definition | Value or status | Source |\n|---|---|---|---|\n| treg per brand | Estimated monthly treg spend | about $10 a brand a month, estimate of 14 Oct | [src: memory treg-tool-catalog] |\n| Viral finder | treg per brand per weekly run | $0.015 to $0.04 | [src: memory brew-viral-remakes] |\n| AI answers weekly check | Claude and treg per run | about $0.50 | [src: brew/live-watch.js:23] |\n| Cache hits | `meter.hits` over calls | Live, not reported | [src: brew/treg.js:67] |\n| Source health | ok, empty, error by source and month | `_brew/ops/sources-YYYY-MM.json`, `/api/admin/sources` | [src: memory brew-apify-watch] |\n| Full reads, 23 Oct | Quotes read per brand | Arata 2,888 from 21 sources; Nutrova 1,127; Cozy 797 | [src: memory brew-apify-watch] |",
  "Open": "1. **The skill is not in the repo.** It lives in the parent folder, not on brew origin/main. Proposal: move it into brew/.claude/skills so every Brew session loads it.\n2. **One cap per brand.** treg is capped per day for all brands, Apify per brand per month. Proposal: add a per-brand monthly treg cap of $15 beside Apify's $10, both shown on the watch page.\n3. **The Apify token was pasted in chat on 21 Oct.** Proposal: rotate it [src: memory brew-viral-remakes]."
 }
}