.claude/skills/script-cohort-analysis/references/learnings.md

# Learnings log

Read this before every analysis. Add new points after every analysis: one or two sentences each, with the run they came from.
Tags: **[method]** how to run the analysis · **[measure]** what our instruments can and can't do · **[content]** what seems to make scripts work (hypotheses until confirmed blind on a new cohort).

## Method
- **[method]** Every rater must be blind to titles, view counts and filenames. Unblinded readers fit stories to outcomes, so blind scores are evidence and post-hoc readings are only hypotheses. (run1)
- **[method]** Pre-register features and directions before unsealing, and treat any explanation found afterwards as a ledger hypothesis for the *next* cohort, never a finding. (run1)
- **[method]** With 6 videos the smallest possible p is 0.0014 and about 3 of ~30 tests pass by chance, so one cohort only generates hypotheses. Evidence comes from repeats across cohorts. (run1–3)
- **[method]** Check confounds first: format, publish period and age explained the worst tier better than any script feature. A whole-tier difference is a confound until shown otherwise. (run1–3)
- **[method]** Use single-format cohorts. The pair judges flagged 11 of 15 HinExp pairs as different formats, which makes craft comparisons unfair. (run3)
- **[method]** Don't test hypotheses on the cohort they came from. That's circular. (run1)
- **[method]** Ideas about the *promise* (topic, framing) can be tested cheaply on titles across hundreds of videos before spending transcript cohorts on them. (post-hoc discussion)
- **[method]** Haiku is fine for raters at 3 per video, but it bunches scores toward the middle more than larger models. Watch the reliability and saturation flags. (run2, run3)
- **[method]** Claude Code runs at most 20 subagents at once, so launch in batches. (run3)

## Measurement
- **[measure]** Absolute 1–5 scores bunched at 3–4 (93% of scores) and couldn't separate scripts. Use yes/no checks plus pairwise comparison. (run1–2 audit)
- **[measure]** The v2 checklist hits its ceiling: hook checks pass 94% of the time, and the composite spreads only 0.12. Checks must be harder or graded for professional scripts. (run3)
- **[measure]** Raters reliably catch local, visible faults (a generic preamble dropped the hook score by 0.8–0.9 every time) but are mostly blind to structure, padding and a missing ending. (perturb1, run3 calibration)
- **[measure]** Calibrate every rubric version with the damage test. The cut-ending test has to actually remove the answer, which may come before the 70% mark. (run3)
- **[measure]** Guard scores drift by up to 0.25–0.44 with 3 raters. That's the noise floor, so smaller movements mean nothing. (run3)
- **[measure]** Voice, credibility, clarity and value density move together as a "presenter documentary vs animated narrative" detector, so they partly measure format. (run1 audit)
- **[measure]** Curiosity and loop-timing measures are unreliable: raters disagree on when questions open and close. (run1–3)
- **[measure]** Pair judges pick the second-shown script 67% of the time on the overall verdict, so positions must be balanced. (run3)
- **[measure]** An LLM topic judge is consistent (0.87 across orders) but ranked topics in reverse order of performance. It measures LLM taste (drama, biography), not this audience's demand. Measure topic demand with data. (run3)
- **[measure]** The pairwise craft judges share the topic judge's taste (rho 0.89). LLM preferences aren't a stand-in for audience preferences. (run3)
- **[measure]** The Hindi counters need care: "you" often lives in the verb ending (करो/कीजिए), and "लेकिन" ("but") counts plot twists, not argument. (run1)

## Content (hypotheses, not yet confirmed blind)
- **[content]** In 3 of 3 runs, nothing blind about script craft predicted performance. The idea and promise of the video seem to matter more than how well it's written. (run1–3)
- **[content]** Both winners correct a belief the viewer holds about their *own* life ("the bus owner must be rich", "rent is wasted money"). This was the cleanest dividing line across the six. (post-hoc)
- **[content]** The winning reveal is the hidden *mechanism* (the money) behind something the viewer uses every day, not the hidden *drama* of famous people. The IPL video promised insider information about something familiar and still failed. (post-hoc)
- **[content]** Loss and disaster appeared in all six videos and didn't separate them. What differed was whose loss: the winners put it on the viewer, the losers on others. (post-hoc)
- **[content]** The best videos open with the viewer inside a familiar scene and state the promise later (~47s vs ~15s). "State the promise as early as possible" was contradicted 3 times. (run1–3)
- **[content]** Escalation and emotion/story scores went the wrong way, probably because the animated narratives had the most of both. (run1–3)
- **[content]** Revealing all the answers in the first minute (Lenskart) and list-in-a-story structures (the legal video) seem to cap a video at median. (post-hoc, weak)
- **[content]** Earned loss with real numbers plus a balanced ending beats manufactured fear. Both winners avoided alarmism. (post-hoc)
- **[content]** A transcript can't explain the worst videos: a well-written IPL script got 2.8k views. Format, packaging and distribution dominate there. (run1–3)

## Added by the series-analysis "why-x-died" runs (cohort-new, cohort-established; 2 x n=10, 2026-10-20)
- **[method]** A cohort made only of top performers (selected on the outcome) is range-restricted: the checklist ceilings (hook 85-95%, emotion/story 95%, escalation 92-97%, payoff 88-90%) mean the instrument cannot separate winners from bigger winners. To learn what works, include weak performers with the same format in the same cohort. (cohort-new, cohort-established)
- **[method]** Two cohorts of 10 (new vs established, same selection rule) is a cheap replication design: a rule counts only if it holds in both. Only 2 of 80 directional tests reached "explains" against about 8 expected by chance. (cohort-new, cohort-established)
- **[method]** A cross-channel performance metric must be chosen carefully: a blend of views vs the channel's own norm, views/day and like rate penalises big channels (a 10M-view Byju's video scored only 2.6x its channel norm) and inflates tiny ones (317x on a 1.2k-subscriber channel). (cohort-established)
- **[method]** Transcript capture needs its own gate: reject captions under about 95 words/min, with subscribe loops or one token above 8% of the text; a 707-word Hindi transcript of an 8-minute video was subscribe spam. (cohort capture)
- **[measure]** The Haiku pair judges flagged only 1-2 of 45 pairs per cohort as cross-format here, so single-format cohorts work. The topic judge was consistent (1.00 and 0.93 across orders) but again did not track performance. (both)
- **[measure]** Curiosity, forward references, interest mean, loops/10min and the holistic percentile had rater agreement of 0.08-0.39 and are BROKEN as measures again. Voice (0.79), emotion/story and escalation are reliable. (both)
- **[measure]** Calibration (damage test): 6/8 detected; hook, padding and shuffle damage are caught; a cut ending was missed on one of two scripts. Payoff/ending stays a weak instrument. (cohort-new)
- **[measure]** Header parsing: `Published: ... (age 1,461 days)` has a thousands separator that the age regex `age (\d+) days` misses, so age was null for 9 of 10 established videos. (cohort-established)
- **[content]** Cold-opening on the collapse before telling the origin was CONTRADICTED in both cohorts (rho -0.47 and -0.45) and is rejected for this audience; the top new video begins in 1865 at a paper mill. (both; ledger H-005)
- **[content]** Numbers in the first 60 seconds went the wrong way in both cohorts (-0.32, -0.51). (both)
- **[content]** Named people carrying the story (H-004) was weak support in both (0.34, 0.33) with high rater agreement; a single stated thesis (H-006) 0.30/0.53; named blame (H-007) 0.16/0.33. Names and blame appear in nearly every script (mean 4.2-4.6 of 5), so they are table stakes, not differentiators. (both)
- **[content]** Presenter voice WORKED in the established cohort (rho 0.80, reliable, calibrated, agrees with pair judges) but not in the new one (-0.16). Treat as a lead (ledger H-010). (cohort-established)
- **[content]** Storytelling itself is not the differentiator in this genre: every top video already passes the emotion/story, escalation and hook checks. (both)

## Added by the Moser Baer script review (rubric v3 and video-script-production; 2026-10-24)
- **[method]** Scripts we write ourselves can now declare their structure (loops, junctions, stake lines, promise ladder) in a sidecar, which the `video-script-production` linter checks without rater judgment. Use that for authored scripts; use blind raters (this skill) for scripts written by others. The two cover each other's blind spots: the linter cannot tell whether a care line lands, and the raters cannot count figures per 30 seconds reliably. (moser-baer v4)
- **[method]** Rubric v3 changes the hash. Recalibrate with `perturb.js` before trusting any v3 result, and extend the damage tests to cover the new checks (a "flat joins" damage that replaces joins with "and then", a "naked numbers" damage that strips stake lines, a "no escalation" hook damage). (moser-baer v4, not yet done)
- **[measure]** The new checks H5, H6 and C5 depend on judging openings and handoffs, which raters have been unreliable at. Screen them on agreement and discrimination before reading their predictions. G1 and G2 are the most countable and should be the first to show a usable signal. K4 overlaps the deterministic `det_additive_rate`; the lexicon now misses signpost joins ("here is the second thing I promised"), which the `video-script-production` linter catches. (moser-baer v4)
- **[content]** Six untested hypotheses (ledger H-012 to H-017) now stand for the four lessons: stake before figure, confirm-then-escalate openings, destination without route, consequence/complication joins, forced handoffs, and low figure density. They run on the next cohort and must include weak performers (range restriction, see above). Nothing here is a finding yet. (moser-baer v4)
- **[content]** Tension to resolve: this skill's own data contradicted cold-opening on the collapse and figures in the first 60 s, which constrains the v4 hook (one short real scene, no figures). If H-014 is contradicted too, drop the destination hint from the hook rule. (moser-baer v4)
- **[measure]** The additive lexicon in `scripts/lexicon.json` was extended on 2026-10-24 (meanwhile, after that, afterwards, moving on, at the same time, signpost joins, and Hindi signposts such as अब बात करते हैं). `det_additive_rate` from runs before that date used the shorter list, so do not compare it across the change. (moser-baer v4)