.claude/skills/video-script-production/references/s1-rival-brief.md

# S1: the rival-video research brief (from "given X" to a cited brief)

**Goal.** Given only a topic ("the fall of Cafe Coffee Day"), find what YouTube already says about it, pull the transcripts, extract every dated event, figure, cause and surprising claim with a verified quote and a timestamp, merge them across videos, and produce a cited brief. The brief is a list of **leads and rival claims**, not facts: it tells S3 what to verify and S4 what the field already says.

**Tool.** `src/topic-research/cli.js` (cached, resumable). **Output.** `research/<slug>/report.md` and `synthesis.json`, plus `search.json`, `transcripts/`, `extractions/`.

**What it is not.** It does not check anything against a source outside YouTube. "Several sources" can be one original claim repeated: YouTube explainers copy each other. Auto-caption names and numbers are unreliable ("Rahul Puri" for Ratul Puri; "Moserbaer"). Nothing from this brief goes into a script until S3 has verified it.

## 0. Before you start

| Need | Check |
|---|---|
| Node 22+, `npm ci` run once in the repo root | `node --version`; `ls node_modules` |
| An LLM backend: `GEMINI_API_KEY` in `.env` (Gemma 4 on the Gemini API), or Ollama (`OLLAMA_HOST`, `OLLAMA_MODEL`) | `src/llm.js` header; `.env.example`. The key is read from the repo's `.env` wherever you run from |
| Network access for YouTube search and captions | the first run prints `[youtube-search] Searching:` lines |
| The topic string and the entity | topic: `"the fall of cafe coffee day"`; entity: the name a title must contain, `"Cafe Coffee Day"` (if you omit `--entity` it is the topic minus framing words such as the, fall, rise, why, how) |

**Choosing the topic (upstream of S1).** For the "fall of X" series, `reports/series-analysis/fall-of-x-topics.md` (made by `src/series-analysis/topic-study.js --series=fall-of-x`) ranks subjects by demand, recency, engagement and headroom (a high-demand subject with few strong videos is a gap). It is gitignored with the rest of `reports/`; regenerate it when you need it. The user normally names the topic; this report is the evidence for the choice, not a step you repeat.

## 1. Run it

```
node src/topic-research/cli.js "the fall of cafe coffee day" --entity="Cafe Coffee Day" --max=8
```

| Flag | Meaning |
|---|---|
| `--entity="..."` | the name a video's title or channel must contain to count as on-topic. Use the shortest unambiguous form |
| `--max=N` | how many videos to take (default 8). 6 to 8 is the working range; more only adds repeats |
| `--also="query"` (repeatable) | extra searches on top of the fixed eight; use for the local-language title ("cafe coffee day kaise doob gaya"), the founder's name, or an angle |
| `--refresh` | redo this stage's files (`search.json`, `transcripts/`, `extractions/`, `synthesis.json`, `report.md`, `.llm-cache/`). It no longer deletes the folder; earlier versions removed all of `research/<slug>/`, including `script/`, so do not run an old copy |

Everything is cached under `research/<slug>/` (the slug is the topic, lower-cased with dashes). Re-running with the same arguments reuses the search, the transcripts and every model answer (cached by prompt hash), so it costs nothing and gives the same result. A change to `--also` or `--max` only takes effect with `--refresh` (the search is cached).

## 2. What it does, stage by stage

1. **Search.** Eight fixed query templates (`{topic}`, `{entity}`, `{entity} documentary`, `explained`, `what happened`, `case study`, `rise and fall`, `collapse scandal`) plus any `--also`, 20 results each, through `ytsearch.js`. Every video seen is kept in `search.json` → `candidates`.
2. **Filter.** A video is dropped if the entity is in neither its title nor its channel name, if its title looks like noise (ads, trailers, songs, unboxing, price pages), or if it is under three minutes. The reason is stored in `candidates[].reject_reason`.
3. **Rank.** `score = log10(views + 10) + 0.5 × (queries that found it, max 4) + 1 if 5 to 60 minutes + 1.5 if the title carries a story word (fall, collapse, why, scandal, explained...) + 1 if the entity is in the title`. The top `--max` are selected. **The score does not know whether a video is about the collapse**: "The Rebirth of Nokia" was selected on a test run. Read the list (section 4).
4. **Transcripts.** `fetchOriginalTranscript` picks the caption track in the video's original spoken language (the auto-generated track), not the first track listed, and caches `transcripts/<id>.json` and `.txt`. A video with no captions is skipped with a warning.
5. **Extract.** One model pass per ~6,000-word chunk of each transcript, at temperature 0.1: relevance, summary, events, figures, surprises, causes, entities, claims to verify. Rules in the prompt: use only what the transcript says, keep numbers and units exactly as spoken, quote verbatim (≤25 words), mark `stated | hedged | attributed`. **In code**, every quote is then checked against the transcript text; `verified` is true only if it occurs, and that pins the timestamp.
6. **Synthesise.** Facts are merged across videos into a timeline, a figures table, surprising points and causes. Each merged item must cite fact ids; items with no valid id are dropped; the source count and the verified-quote count are computed in code, not by the model. A final pass writes the one-line story, the overview, entities, open questions, contested claims and 6 to 12 `verification_queries`.
7. **Report.** `report.md`: the story in brief, timeline, hard numbers, surprising points, why it happened, people, contested claims, what is still open, the queued verification queries, a sources table (with facts verified/total) and a method note.

## 3. Optional exports (needed for S2)

```
node src/topic-research/export-csv.js research/<slug>            # all-videos.csv: every video seen, with channel subscriber counts (visits each channel; slow)
node src/topic-research/export-cohort.js research/<slug> research/<slug>/teardown/cohort-<slug> <videoId> [<videoId> ...]
```

`export-csv` needs `search.json` to have `candidates`. `export-cohort` needs `all-videos.csv` (for subscribers) and writes one transcript `.txt` per video in the layout the teardown reads (`=== VIDEO ===`, `=== PERFORMANCE ===`, `=== CHANNEL ===`, `=== TRANSCRIPT ===`). Views per day uses YouTube's rounded age text ("10 mo ago"), so it is approximate; likes and the description are not captured.

## 4. Read the output (this is the work of S1)

Open `report.md`, `synthesis.json` and `search.json`. Do these in order and write the result into `FACTS.md` (the tables are in the scaffold).

1. **Were the right videos selected?** In `search.json` → `selected`, check each title is about the collapse or story you will tell. A rebirth, a stock-tips or a product video is the wrong source. To change the set: edit `selected` in `search.json` by copying entries from `candidates` (the cached search is returned as written) and re-run (transcripts and extraction are fetched for the new set); or add `--also` queries with `--refresh`. Record why you swapped.
2. **How many sources were usable?** `sourcesUsed` versus `sourcesExcluded` (no captions, off-topic). Fewer than four usable: widen with `--also` and `--max`.
3. **Mine the report into FACTS.md:**

| Report section | Goes into | How to treat it |
|---|---|---|
| Timeline | the facts table as `[CHECK]` rows, in order | each row's "Support" tells you how many rival videos say it; 1 source with an unverified quote (`?`) is the weakest |
| Hard numbers | the magnitude cross-check table | numbers are as spoken in the captions; conflicts (⚠) are the first things to verify |
| Why it happened | the causal-claims table | every cause is a claim to find a source for; note which ones every video repeats (table stakes) |
| Surprising points | candidate teases and landings | verify before using; they are what rivals already lead with |
| Key people | the people table | start the person research in S3 from these names |
| Contested or shaky claims | the do-not-use table (until verified) | |
| Still open | S4's list of what a viewer will ask | the "open questions" are your destination-hint material |
| Verification queries | the S3 search list | run them; they were written for exactly this |
| Sources table | the rival-claims inventory | which channels, how many views, which language |

4. **Inventory the rival claims you may need to refute.** Any figure that appears in more than one video and looks round, dramatic or unsourced goes in "Claims you must NOT use" with the reason, once S3 fails to find it.

## 5. Gate to leave S1

- [ ] `report.md` and `synthesis.json` exist; 4 or more usable sources (or the shortfall is stated).
- [ ] The selected videos are about the story you will tell; swaps are recorded.
- [ ] `FACTS.md` has the timeline rows, the figures to cross-check, the causes, the people, the contested claims and the verification queries copied in as `[CHECK]` rows.
- [ ] Nothing from the brief is marked `[V]`.

## 6. Failure modes and fixes

| Symptom | Cause | Fix |
|---|---|---|
| `GEMINI_API_KEY is not set` | no key; no fallback | put it in `.env`, or run Ollama (`LLM_PROVIDER=ollama`) |
| Rate limit or daily limit messages, then slow progress | Gemini quota (per project, resets at midnight Pacific) | wait or switch provider; cached answers are reused on re-run |
| `[transcript] <id> skipped` | captions disabled | the video drops out; take the next candidate by editing `selected` |
| Selected videos off-topic (rebirth, stock tips, a different company with the same name) | the score is blind to angle | edit `search.json` `selected`, or tighten `--entity` |
| Names or numbers garbled | auto-captions | trust the model's spelling fixes only for obvious cases; verify every name in S3 |
| A model answer is not valid JSON | the model | it retries three times; if it still fails, re-run (the good answers are cached) |
| The run is slow | one model call per ~6,000 words per video, then four synthesis calls and one overview | normal: a test run with `--max=3` took 8 min 21 s (see section 8); with `--max=8` expect roughly 20 to 25 minutes (an estimate from the 3-video run). Run it in the background |
| `--refresh` and your scripts | older copies of the CLI deleted the whole `research/<slug>/` folder | this repo's CLI now removes only its own files; confirm with `git diff src/topic-research/cli.js` |
| The AI Mode source | not wired in (`sources/google-ai-mode.js` throws) | verification queries are searched by hand in S3 |

## 7. What to record

In `learnings.md`: the topic, how many videos were seen/kept/selected, how many swapped and why, how long it took, and anything the report got wrong that S3 later caught (so the next run can tell how far to trust it).

## 8. Observed on a test run (2026-10-25, topic "the fall of nokia", entity "Nokia", `--max=3`, run from a scratch folder)

- It ran from outside the repo (the key is read from the repo's `.env`) and wrote `research/the-fall-of-nokia/` in the scratch folder.
- Search: 8 queries, 116 videos seen, 99 passed the filter, 3 selected. All 3 had captions (one Hindi, two English).
- **The selection included two "rebirth of Nokia" videos** (the angle is the recovery, not the fall), because the score does not read the angle. The third, a 13-million-view Hindi case study, was on topic. This is why section 4 step 1 says to read the selected list.
- Extraction: 94 facts from 3 transcripts, 83 with a verified quote (88%); the unverified ones are the weakest rows.
- Synthesis: 33 timeline items, 38 figures, 8 causes, 4 surprising points, 12 verification queries.
- Time: 8 min 21 s for 3 videos, almost all of it model calls (8 cached answers: 3 extractions, 4 synthesis sections, 1 overview). Cost scales with video count and length.
- The CLI fix (`--refresh` no longer deletes the folder) was made after reading the code; it was not exercised by this run.