---
id: "6.6"
title: "Outside data rules"
dept: ai-and-models
status: draft
owner: Param
branded: false
last_verified: 2026-10-26
sources:
  - "brew origin/main b91e364: treg.js:1-139, apify.js:1-54, reader/adlib.js:1-86, shelf.js:1-8 and 69, listen-watch.js:10 and 684, market-pulse.js:318, live-watch.js:21-24, deploy.sh:38-41, tools/prod-env.sh:9-11"
  - "skill /Users/param/Documents/ChatGPT/.claude/skills/brew-external-data/SKILL.md, with audit.py and heal-test.mjs; not on brew origin/main"
  - "memory treg-usage-rules, treg-catalog-for-brew, treg-tool-catalog, brew-external-data-skill, brew-apify-watch, brew-lean-pipeline, brew-viral-remakes, brew-community-seat, no-spend-while-building"
---

# Outside data rules

## Answer

How Brew reads data it does not own, through treg, Apify, the public Meta Ad Library and model web search: treg first, a free fallback wherever one exists, every paid answer cached and shared so it is paid for once, a cap on every source, and a failure never shown as a fact.

## Today

### The sources

| Source | Used for | Client | Key | Source |
|---|---|---|---|---|
| treg.to | Listening, SERP, keywords, reviews, AI answers, rival ads, shopping | `treg.js`, shared, one per server | `TREG_TOKEN`, agent key brew-live, team squareup, 20,000 calls a day | [src: brew/treg.js:1-5] [src: memory brew-apify-watch] |
| Apify | Indian marketplaces, quick commerce, Similarweb, Instagram mentions, YouTube clip downloads, Trustpilot backup | `apify.js` | `APIFY_TOKEN`, Starter plan bought 23 Oct | [src: brew/apify.js:1-4] [src: memory brew-apify-watch] |
| Meta Ad Library page | Free fallback for every Ad Library call | `reader/adlib.js`, headless Chromium, 2 at once | none | [src: brew/reader/adlib.js:1-9] |
| Model web search | Rivals, company facts, the Claude AI answer | ai.js `search` | the model's | [src: brew/ai.js:62] |
| Connected accounts | GA4, Search Console, Instagram, Google Ads: free | connectors | the brand's grant | [src: memory treg-catalog-for-brew] |

### The rules in code

1. **treg first, Apify only where treg has nothing or is weak** [src: memory brew-apify-watch].
2. **Data only through treg; assets never.** Images, video and voice run on our own Vertex and OpenAI keys [src: memory treg-usage-rules].
3. **Every call goes through the shared client.** Never `fetch('https://treg.to/...')` directly; every Ad Library call goes through `adCall(treg, id, o)` [src: brew/reader/adlib.js:76] [src: memory brew-external-data-skill].
4. **Paid once per window.** The same endpoint and parameters are paid for once per signal window, by any caller on any instance; identical calls in flight share one answer [src: brew/treg.js:11-15] [src: brew/treg.js:57].

| Signal | Cache window | Source |
|---|---|---|
| Company, tech stack, whois | 720 hours | [src: brew/treg.js:20-27] |
| Reviews, SERP, keywords, trends, AI answers, profiles | 168 hours | [src: brew/treg.js:20-27] |
| Ad libraries, news, web search, shopping, prices | 24 hours | [src: brew/treg.js:20-27] |
| Anything else | `TREG_CACHE_HOURS`, 24 | [src: brew/treg.js:17] |
| Apify reviews actors, other actors | 168 hours, 24 hours | [src: brew/apify.js:17] |
| Generation, job polling, downloads, failures | never cached | [src: brew/treg.js:16] |

5. **No `cache: false` for read-only data.** A caller that needs fresher data passes `maxAgeH` instead [src: brew/treg.js:64-65] [src: memory treg-usage-rules].
6. **Re-reads reuse what we have:** the read keeps the category and its search terms, and reuses the last rivals, terms and questions [src: brew/reader/stages/answers.js:39-47] [src: memory brew-lean-pipeline].
7. **An error is never an empty result.** Three branches every time: found, truly none, could not check and retries [src: memory brew-external-data-skill]. Apify rows say a platform could not be read, never "not sold there" [src: brew/shelf.js:5].
8. **A fallback must match the paid source's numbers, not just its shape.** The Ad Library fallback reads the page itself for its full count; keyword results once gave Snitch 23 ads instead of 516 [src: memory brew-external-data-skill] [src: brew/reader/adlib.js:62-75].
9. **Every live call is measured** for source health: ok, empty or error, time and cost; the router sends work to the healthiest source first [src: brew/treg.js:76-79] [src: memory brew-apify-watch].
10. **Saved reads heal** on open and on the check page, once a day or hourly while errors remain; a stage that reads differently bumps its version, such as `ADS_V` [src: memory brew-external-data-skill].
11. **Brand-agnostic.** Never hardcode a brand; a bucket patch is a stopgap [src: memory brew-external-data-skill].
12. **No captcha bypass.** The Ad Library reader stops when the page asks for a captcha [src: brew/reader/adlib.js:37] [src: brew/reader/adlib.js:54].

### Caps

| Cap | Value | Scope | Source |
|---|---|---|---|
| `TREG_DAY_CAP_USD` | $40 | all brands, every instance, a day | [src: brew/deploy.sh:41] [src: brew/tools/prod-env.sh:9] [src: brew/treg.js:82] |
| `APIFY_BRAND_MONTH_CAP_USD` | $10 | a brand, a month | [src: brew/shelf.js:69] [src: brew/tools/prod-env.sh:9] |
| Apify run | $0.25 by default through `maxTotalChargeUsd` | a run | [src: brew/apify.js:40] |
| `LISTEN_RUN_CAP_USD` | $0.60 | a listening run | [src: brew/listen-watch.js:684] |
| `PULSE_CAP_USD` | $3 | a market pulse run | [src: brew/market-pulse.js:318] |
| Brand Book enrichment | $0.50 | a brand, weekly | [src: memory brew-brand-book] |

### Preferred endpoints

The full map is `treg-map.js` on main; memory keeps the prices of 14 Oct [src: memory treg-catalog-for-brew].

| Signal | Use | Avoid |
|---|---|---|
| TikTok search, comments | `tikhub.tiktok.search.videos`, `tikhub.tiktok.video.comments`, $0.001 | `anyapi.tiktok.ad_library_search`, 25% success |
| Instagram reels, comments | `anyapi.instagram.search.reels`, `anyapi.instagram.post.comments` | |
| Reddit | `tikhub.x.reddit-app-fetch-dynamic-search`, $0.001 | Reddit's own .json, 403 to our servers [src: memory brew-community-seat] |
| Facebook comments | `scrapecreators.x.v1-facebook-post-comments` | routed `treg.facebook.post.comments`, 26% |
| Keyword volume | `dataforseo.google.keywords.volume`, $0.09 per 1,000; free through Google Ads when connected | |
| Trends | `dataforseo.x.keywords-data-dataforseo-trends-explore-live`, $0.0012 | serpapi, 12 times dearer |
| AI answers | `dataforseo.x.ai-optimization-*`, $0.004 | `crawl4ai.ai-search.chatgpt.scrape`, 3%; llm-mentions at $0.10 |
| Google ads | `adyntel.google.ads.transparency`, $0.011 | |
| Amazon reviews | `justoneapi.x.amazon-get-product-top-reviews-v1`, country-aware | crawl4ai, which always reads amazon.com [src: memory brew-apify-watch] |
| Nykaa, Flipkart, Myntra, Blinkit, Zepto, Instamart | Apify actors that passed the 23 Oct probe, each with a backup | treg has none, request #1171 |
| Video understanding | download through treg or Apify, read on our Vertex | |

## Tomorrow

| Item | Version | Tag |
|---|---|---|
| treg first, Apify for gaps, a free fallback for every paid source | v1 | Stays |
| The brew-external-data skill moved into brew origin/main, so every session sees it | v1 | Changes |
| One monthly outside-data cap per brand across treg and Apify, shown to the team | v2 | New |
| Source health shown per brand on the watch page with the last good read | v1 | Changes |
| Connected first-party data preferred over paid scraping when both exist | v2 | Changes |
| Indian marketplaces through treg once request #1171 lands | v2 | Changes |

## How we do it

1. Load the brew-external-data skill before any change that reads or shows outside data [src: memory brew-external-data-skill].
2. Check the live balance before saying treg is empty; never trust old logs [src: memory treg-usage-rules]:
   ```bash
   treg balance
   ```
3. Find stragglers that skip the shared client:
   ```bash
   git -C /Users/param/Documents/ChatGPT/brew grep -n "treg.to/call\|adlibrary" origin/main -- '*.js'
   ```
   Only treg.js, treg-map.js, adlib.js and guarded local tools should show [src: memory brew-external-data-skill].
4. Audit every brand, then test with treg forced to fail:
   ```bash
   python3 /Users/param/Documents/ChatGPT/.claude/skills/brew-external-data/audit.py brew-prototype
   node /Users/param/Documents/ChatGPT/.claude/skills/brew-external-data/heal-test.mjs <slug>
   ```
5. Probe a new Apify actor on real brands before using it: `node tools/probe-sources.mjs` [src: memory brew-apify-watch].
6. Tag every call `<what>-<slug>`, so spend is booked per brand in `spend/<YYYY-MM>.json` [src: brew/treg.js:3-5] [src: brew/treg.js:121].

## Numbers

| Metric | Definition | Value or status | Source |
|---|---|---|---|
| treg per brand | Estimated monthly treg spend | about $10 a brand a month, estimate of 14 Oct | [src: memory treg-tool-catalog] |
| Viral finder | treg per brand per weekly run | $0.015 to $0.04 | [src: memory brew-viral-remakes] |
| AI answers weekly check | Claude and treg per run | about $0.50 | [src: brew/live-watch.js:23] |
| Cache hits | `meter.hits` over calls | Live, not reported | [src: brew/treg.js:67] |
| Source health | ok, empty, error by source and month | `_brew/ops/sources-YYYY-MM.json`, `/api/admin/sources` | [src: memory brew-apify-watch] |
| Full reads, 23 Oct | Quotes read per brand | Arata 2,888 from 21 sources; Nutrova 1,127; Cozy 797 | [src: memory brew-apify-watch] |

## Open

1. **The skill is not in the repo.** It lives in the parent folder, not on brew origin/main. Proposal: move it into brew/.claude/skills so every Brew session loads it.
2. **One cap per brand.** treg is capped per day for all brands, Apify per brand per month. Proposal: add a per-brand monthly treg cap of $15 beside Apify's $10, both shown on the watch page.
3. **The Apify token was pasted in chat on 21 Oct.** Proposal: rotate it [src: memory brew-viral-remakes].