Mining and verifying sources

Stages S1 to S3: rival videos as leads, a teardown of what they do, and independent research that builds the fact base.

The three layers

S1: the rival-video brief

Tool: src/topic-research/cli.js. Cached and resumable. Full procedure: references/s1-rival-brief.md.

Before you start

• Node 22 or later; npm ci run once.
• A language-model backend: GEMINI_API_KEY in .env (Gemma 4 on the Gemini API), or Ollama (OLLAMA_HOST, OLLAMA_MODEL, LLM_PROVIDER=ollama).
• Network access for YouTube search and captions.
• The topic string and the entity, the name a title must contain. If you omit --entity it is the topic minus framing words. Use the shortest unambiguous form (Cafe Coffee Day used Coffee Day because some videos say CCD).

Run

node src/topic-research/cli.js "the fall of cafe coffee day" --entity="Coffee Day" --max=8 \
  --also="cafe coffee day siddhartha" --also="cafe coffee day kaise doob gaya"

The cache means a re-run with the same arguments costs nothing. A change to --also or --max takes effect only with --refresh, because the search is cached.

What it does

1. Search. Eight fixed query templates (topic; entity; entity documentary; explained; what happened; case study; rise and fall; collapse scandal) plus any --also, 20 results each. Every video seen is kept in search.json under candidates.
2. Filter. Dropped if the entity is in neither title nor channel name, if the title looks like noise (ads, trailers, songs, unboxing, prices), or if shorter than three minutes. The reason is stored.
3. Rank. score = log10(views + 10) + 0.5 x queries that found it (max 4) + 1 if 5 to 60 minutes + 1.5 if the title carries a story word + 1 if the entity is in the title. The top --max are selected. The score does not read the angle: "The Rebirth of Nokia" was selected on a test run.
4. Transcripts. Picks the caption track in the video's original spoken language, not the first listed. A video with no captions is skipped.
5. Extract. One model pass per about 6,000 words of each transcript at temperature 0.1: relevance, summary, events, figures, surprises, causes, entities, claims to verify. The prompt says use only the transcript, keep numbers and units as spoken, quote verbatim up to 25 words, mark each stated, hedged or attributed. In code, each quote is checked against the transcript; verified is true only if it occurs, and that pins the timestamp.
6. Synthesise. Facts merged across videos into a timeline, a figures table, surprising points and causes. Each merged item must cite fact ids; items without a valid id are dropped; source counts and verified-quote counts are computed in code. A final pass writes the one-line story, entities, open questions, contested claims and 6 to 12 verification queries.
7. Report. report.md: story in brief, timeline, hard numbers, surprising points, why it happened, people, contested claims, what is open, the queued verification queries, a sources table and a method note.

Read the output (this is the work of S1)

1. Were the right videos selected? In search.json check each selected title is about the collapse. A rebirth, stock-tip or product video is the wrong source. To change the set, copy entries from candidates into selected, re-run without --refresh (new transcripts are fetched), and record the swap and why. The Cafe Coffee Day run swapped three: two "Malavika Hegde success story" videos and "The Inspiring Comeback Story of Cafe Coffee Day"; one added video said "CCD" in its title and had been rejected on the entity rule.
2. How many sources were usable? Fewer than four: widen with --also and --max.
3. Mine the report into FACTS.md as [CHECK] rows:

Gate to leave S1

• report.md and synthesis.json exist; 4 or more usable sources, or the shortfall is stated.
• The selected videos are about the story you will tell; swaps are recorded.
• FACTS.md holds the timeline, the figures, the causes, the people, the contested claims and the verification queries as [CHECK]. Nothing from the brief is marked [V].

Failure modes

Observed test (topic "the fall of nokia", --max=3): 8 queries, 116 videos seen, 99 passed the filter, 3 selected (two were "rebirth" videos). 94 facts, 83 with a verified quote (88%); 33 timeline items, 38 figures, 8 causes, 12 verification queries; 8 min 21 s.

Optional exports (needed for S2)

node src/topic-research/export-csv.js research/<slug>            # all-videos.csv with channel subscriber counts (visits every channel; slow)
node src/topic-research/export-cohort.js research/<slug> research/<slug>/teardown/cohort-<slug> <videoId> ...

Shortcut that worked: copy search.json to search.json.full, trim candidates to the videos you care about, run the export, restore the file. Views per day use YouTube's rounded age text, so they are approximate.

S2: the teardown of rival videos

Skill: winning-video-teardown. Full procedure: references/s2-teardown.md. Goal: write down, with quotes, what the rival videos all do (table stakes), what the big ones do that the small ones do not, what their packaging promises, where they are weak, and which of their claims you must not repeat.

Do you need it?

Who may run the analysts

Steps (A and B)

node .claude/skills/winning-video-teardown/scripts/prepare.js --cohort research/<slug>/teardown/cohort-<slug> --out research/<slug>/teardown/run --perf-unknown
node .claude/skills/winning-video-teardown/scripts/validate.js --file run/analyses/V01.json --case run/cases/V01 --strict 1
node .claude/skills/winning-video-teardown/scripts/aggregate.js --run research/<slug>/teardown/run
node .claude/skills/winning-video-teardown/scripts/report.js    --run research/<slug>/teardown/run

• --perf-unknown tells the analyst the video was picked for its topic, not because it outperformed. Without it, the task files claim it "performed far above the norm", which is false for a topic set.
• Choose 5 to 8 videos (3 for B or C): the most-viewed that are about the story; a mix of big and small channels; no more than 2 per channel; no rebirth, stock-tip or product videos. Hindi, English and Malayalam captions all worked.
• The analyst task: mechanism, not praise; every claim in doing_well and script.mechanisms has a verbatim quote (4 to 25 words, original language, English gloss) and timestamp; find every open loop; type every junction (THEREFORE, BUT, AND_THEN, NEW_TOPIC); record how each hero figure was framed; name real weaknesses; do not speculate about the audience or the algorithm.
• The validator checks structure, that every mechanism id is in taxonomy.md, minimum counts (3 to 6 doing-well; at least 5 script mechanisms) and that at least 85% of the quotes literally occur in the transcript. A failing analysis is rewritten, not patched.
• Read at least five analyst claims against the transcript yourself and drop or downgrade any the text does not support.

The synthesis (the only prose)

teardown/SYNTHESIS.md: read-this-first limits; table stakes (in N of N, so they do not explain why one video beat another); common but not universal; where big videos differ from small (a lead only); moves worth copying with quotes; where these videos are weak (the opening a new video can take); claims they make that you could not verify (this feeds the do-not-use table); new mechanisms found by one analyst; spot-checks; packaging notes. The Moser Baer synthesis is the model.

Pitfalls

• Treating a topic set as winners. Use --perf-unknown and say how many have reach.
• Counting a table-stakes mechanism as a differentiator.
• Copying a rival's dramatised or unsourced claim into the script. They go in the do-not-use table.
• Hindi or Malayalam captions garble names (an analyst wrote "Rahul Puri" for Ratul Puri). Work from meaning and correct names in S3.
• Analysts flatter. The prompt requires real weak points; a teardown with none is re-run.

S3: independent research and the fact base

Procedure: references/research-protocol.md. This stage decides the thesis. It changed the Moser Baer thesis once (v4) and its whole shape once (v5).

The eight rules, each earned by a mistake

1. The person before the company. The first Moser Baer fact base was built from rival videos and results; two Business Today profiles of the founder (2003 and 2011) were never opened. Opening them turned a story about a bond into a story about a man who always left what was running out.
2. Verify the link, not just the numbers. v3 said the solar bond turned decline into debt; sources showed the debt was already there in 2005 and none says where the bond's money went.
3. Cross-check the magnitude of every key figure in a second source. One article's "In 2004, 3,500 crore" was a garble: another magazine had total debt at 1,620 crore in 2005 and 3,500 crore in 2011.
4. Prefer primary or near-primary sources: rating-agency notes, liquidator and tribunal documents, results as reported by a wire, press conferences as reported by the trade press, court reports, long-form business magazines. Wikipedia is a map to the cited page, not a source.
5. Mark every row with a status (below).
6. Rival-video claims you cannot verify go in the do-not-use table with the reason. Moser Baer: "solar unit sold for 72 crore", "787 crore", "8,000 crore", market shares of 17, 70 and 80 percent, workers burning effigies.
7. Sources disagree; say so, and use the safer one. The gate closing was 3 November in one report and 4 November in another, so the script says "November 2017".
8. A person named in the script needs a record: on-record quotes in a national outlet, not a rumour. A private person is the user's decision.

Status tags

[V]

Checked against a non-YouTube page you opened and read.

[V-search]

Seen only in a search-result summary; the page could not be opened. Confirm before filming.

[CHECK]

Unverified.

derived

Computed; the arithmetic is shown.

analysis

Your reasoning; the voice-over must flag it as such.

Order of work

1. Read the S1 brief: timeline with support counts, contested claims, verification queries. They are the first search list.
2. The person and the early company: founder profile, family, first jobs, first company, why they left, partners, the first big bet, who doubted it.
3. The rise: who financed it, the cost advantage, capacity, the peak numbers, the rival that was ahead, the company's own explanation then.
4. The turn: the first bad result, what the company blamed, what changed in the market. Write the prediction your thesis makes and look for the figure that tests it.
5. The responses: each diversification or rescue with dates and amounts, and what the company said in public at the time.
6. The debt and the date: rating notes, bond terms, due dates, restructuring agreements, missed payments.
7. The end: the insolvency petition, buyers who walked away, the liquidation order, the claims and the liquidation value.
8. The people: workers' accounts (named, on record), when the signs began, what happened afterwards.
9. The blame question: cases, agencies, amounts, court decisions, appeals; compute the scale of the allegations against the claims.
10. The rival claims you will refute: search for each number a rival used and you did not; write "not found in any court record or news report" only after you looked.

Asking the fetch tool so it does not paraphrase

Fetch tools summarise. Ask: "Give the exact sentences, quoted verbatim, about X"; "What is the publication date or issue date of this article, and the year?"; "Return the article text verbatim as far as available; say if it is paywalled or truncated". If a page cannot be opened, search for the title and key names, accept the summary and mark the row [V-search].

Query templates (replace the entity)

<founder> founder profile early life how he started <first product> <city>
<company> history founded <year> <partner> <first plant>
<company> <year> results net loss revenue <business> Business Standard OR Storage Newsletter OR pv-magazine
<company> bond OR FCCB OR debenture <amount> due <year> recast OR restructuring
<company> CARE OR ICRA OR CRISIL rating <year> continuing losses negative net worth
<company> workers plant closed <year> salary unpaid <city> union president
<company> insolvency NCLT petition <lender> liquidation order claims <year>
<company> CBI OR ED case <bank> crore discharged OR chargesheet
<industry> price fall <years> <industry body>
"<company>" <exact figure a rival video used>

For each fact, note the stake

Whose loss it is, which belief of the viewer's it breaks, or which contrast it makes. A figure with no stake goes in the ledger and not in the voice-over (Lesson A). The same note becomes the stake line and the landing in S5.

FACTS.md: five tables

When to stop

Stop when each planned beat has at least one sourced fact for its claim, every causal link has a source or has been softened, and every key figure appears in two places. Do not stop at the first plausible account. Do not keep researching past the point where the open items are things only a primary document can answer (the bond's use of proceeds); list them and say aloud in the script that you could not find them.

Resolving conflicting figures (Cafe Coffee Day)

Resolve each conflicting figure by its date and basis before choosing. Three of the S1 brief's conflicts were different dates or bases, not errors: Rs 7,200 crore was the debt at the start of 2019, Rs 3,200 crore the debt in July 2020; Rs 6,550 crore (March 2019) and Rs 4,970 crore (group, August 2019) are different entities or dates. A real error (the 2015 IPO at Rs 50 crore; it raised about Rs 1,150 crore) was visible only against a second source.

The research record

Every search and fetch of a session is written to the repository, so the research behind a version is not only in the chat.

node .claude/skills/video-script-production/scripts/session-log.js --since <start of the run> --out research/<slug>/script/RESEARCH-LOG-vN.md

Existing records: references/research-log.md (every search and fetch behind Moser Baer v4 to v6, by phase), research-log-round7.md (v7), and research/the-fall-of-cafe-coffee-day/script/RESEARCH-LOG-v1.md (20 searches, 15 fetches, 0 subagents). For Cafe Coffee Day a PROGRESS.md logged each stage, its output, the gate result and the next action, so that a session that lost context could resume.

Special care: people, deaths and allegations

• Name a living person only if they are on the record in a national outlet. Single-source anecdotes about private individuals stay out. Dramatised claims (a death, self-immolation) stay out on purpose.
• State exactly what is alleged and what a court decided. Never call anyone a villain, crook, scam or fraud without a conviction or final finding. Moser Baer beat 12 states the CBI figures as allegations, the Ratul Puri discharges in 2025 and the CBI's challenge.
• A death is public record but also a family's grief. State only what national outlets and official findings state, in one or two plain sentences, with no detail of how, no speculation about why, and put the decision under "Decisions for the user".
• Claim sizes are not stolen money: comparing alleged fraud with creditor claims shows scale, not innocence.

Written 2026-10-27 from the repository files and the project memory notes. If a page and a file disagree, the file is right: fix the page.