.claude/skills/script-cohort-analysis/SKILL.md

---
name: script-cohort-analysis
description: Blind, pre-registered analysis of a cohort of YouTube transcripts (e.g. 2 best / 2 median / 2 worst from one channel) to test which script rules actually predict performance, which rules failed or could not be measured, and where the analysis is stumped. Produces a per-run rule scorecard with evidence. Use when the user wants to compare high- vs low-performing video scripts, validate a script rubric, or explain why some videos outperformed. Input is a folder of transcript .txt files with VIDEO/PERFORMANCE/TRANSCRIPT sections.
---

# Script cohort analysis (rubric v3)

The goal is to find which script properties *predict* performance, without the analyst (you)
fitting a story to known outcomes, and to record **for every rule whether it worked as
expected and what the evidence is**. Core rules:
- Nobody who scores a script knows how it performed.
- Every claim is registered before the answers are revealed.
- Explanations invented after unblinding are hypotheses for the *next* cohort.

Scripts live in `scripts/` (Node, no dependencies). Prompts live in `references/`. The
checklist lives in `scripts/checklist.js`. `S` below means `node "<skill dir>/scripts/...`.

**All subagents run on Haiku.** Spawn them with `subagent_type: "script-rater"` and
`model: "haiku"`. If that agent type isn't loaded yet, use `general-purpose` with
`model: "haiku"`. Every subagent gets exactly this prompt:

> Read the task file `<task path>` and follow it exactly. Read only the files it lists. Finish with one line: output path + validator result.

## Step 0 and final step: the learnings log (mandatory)

- **Before Phase 1,** read `references/learnings.md` and apply its [method] and [measure] points to this run (cohort design, confounds, known-broken measures). Its [content] points are hypotheses to test blind. Never use them as findings.
- **After Phase 5,** append what this run taught as one- or two-sentence bullets tagged [method], [measure] or [content], with the run name. Update or strike any existing point this run confirmed or contradicted.

## What gets measured

| Method | What | Why |
|---|---|---|
| Checklist (3 raters × video) | 45 yes/no checks over 11 dimensions, each with quotes | Observable and literal. Replaces vague 1–5 impressions. |
| Structure extraction | Sections with `depends_on` and `references_unintroduced` | Makes broken order and missing setups visible |
| Pairwise craft (all n·(n−1)/2 pairs) | A vs B on each dimension and overall, plus a `same_format` flag | Comparison is more reliable than absolute scores. Flags unfair cross-format pairs. |
| Blind topic judge | Neutral topic cards only, every pair in both orders | Scores the **idea** without seeing the execution, and measures its own consistency |
| Title-only packaging | Titles under a separate shuffle | Separates packaging from script |
| Deterministic counters | Question, number, connective and second-person rates | Zero-judgment anchors |
| Calibration (damage test) | Hook, structure, padding and ending damaged one at a time | Shows whether each measure detects what it claims to |

## Phase 1: Blind and pre-register

```
node S/prepare.js --cohort <cohortDir> [--raters 3] [--seed N] [--metric views_per_day|views] [--ledger <path>] [--audience "<who watches>"]
```

This creates `<cohortDir>/_runs/<timestamp>/`, containing:
- Blind transcripts and blind titles.
- `features.json`.
- `checklist.md`: the frozen rubric for this run.
- `prereg.json` + `prereg.sha256`, including the rubric hash.
- `sealed/key.json`: **don't open this, the cohort's original files, or `_index.json` before Phase 4.**
- Task files for wave 1.

## Phase 2: Blind rating

1. **Wave 1, in parallel:** every `tasks/rate_*.md`, every `tasks/pair_*.md`, and `tasks/packaging.md`.
   Claude Code runs at most 20 subagents at once by default (`CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS`),
   so launch in batches of 20 or fewer and start the next task as each one finishes.
2. **Wave 2:** run `node S/topics.js --run <runDir>`, which builds topic cards from the
   raters' neutral `topic_card` field and warns if one mentions execution or format. Then
   run `tasks/topic.md`.
3. Run `node S/check.js --run <runDir>`. Re-run any MISSING or INVALID task with a fresh subagent.

## Phase 3: Calibrate (once per rubric version per language; recommended every run)

```
node S/perturb.js make --run <runDir>          # auto-picks 2 base scripts by blind checklist score
# spawn every <runDir>/calibration/tasks/rate_*.md
node S/perturb.js analyze --dir <runDir>/calibration
```

To reuse a calibration from an earlier run with the same rubric hash, pass
`--calibration <thatRun>/calibration` to analyze.js.

## Phase 4: Unseal, test and build the scorecard

```
node S/analyze.js --run <runDir>
```

This writes the following into the run directory:
- `analysis.json` and `report-data.md`: feature verdicts, blind predictions vs actual, stumped samples, confounds.
- **`rule-scorecard.md` / `.json`**: for every rule (each dimension, the composite, pairwise
  craft, topic, each deterministic and extraction feature, and every checklist item), five
  lines of evidence:
  1. **Reliability**: do 3 blind raters agree?
  2. **Discrimination**: does it separate the videos, or is it constant or a dead item?
  3. **Calibration**: does it move when its own property is damaged, and hold still when others are?
  4. **Convergence**: does the checklist agree with the independent pairwise judges?
  5. **Prediction**: rho and p in the pre-registered direction, the drop-one check, and confounds.

  Status values:
  - `WORKED`
  - `CONFOUNDED`
  - `FRAGILE`
  - `WEAK_SUPPORT`
  - `MEASURES_OK_NO_EFFECT`
  - `CONTRADICTED`
  - `SATURATED`: checks pass for almost every script, so they cannot separate videos.
  - `BROKEN_MEASUREMENT`: if evidence 1–3 fail, the prediction result is meaningless.
- It also appends to `<transcripts>/_rule_history.json`, so each rule builds a track record across cohorts.

## Phase 5: Interpret (the only phase where you write prose)

Write `<runDir>/report.md`:
1. **Verdict.** Do checklist, pairwise craft, topic or packaging predict performance? Give rho
   and p for each. Is idea separable from execution (topic↔craft rho)?
2. **Rules that worked**, and rules that worked but are confounded or fragile. Quote the
   evidence from `checks.<ID>` in the rater files.
3. **Rules that contradicted** expectations. Don't explain these away.
4. **Broken measurements**: which rules couldn't be measured, and why (reliability,
   discrimination, calibration). These need rubric work before they mean anything.
5. **Stumped samples.** At most 3 post-hoc candidates each, labelled
   *script / topic / packaging / format-production / external*.
6. **Confounds.** Post-hoc tags go in `covariates.json` (`{ "<video_id>": { "name": 1 } }`); re-run analyze.js afterwards.
7. **New hypotheses → ledger:** `node S/ledger.js add --ledger <ledger> --statement "..." --question "<1-5 question>" --direction 1 --origin <runDir>`.
8. **Limits.** Report n, the smallest p, the chance-"explains" count, the rater model (Haiku), and what's unobservable.

Rules:
- Never upgrade a post-hoc idea to a finding.
- Never drop a failed pre-registered rule.
- A BROKEN_MEASUREMENT rule says nothing about scripts. It says something about the rubric.

## Power and honesty

With n=6, the smallest one-sided p is 1/720, and about 3 false "explains" verdicts are
expected across about 30 tests. One cohort generates hypotheses. Evidence comes from
`_rule_history.json` and the ledger across cohorts: a rule or hypothesis needs `explains`
in at least two cohorts with no `contradicts`.

## Rubric v3 (2026-10-24)

Added after the Moser Baer script review, to test blind the lessons that `video-script-production` enforces mechanically: **H5** (confirm the promise by ~30 s and signal more by 60 s), **H6** (destination hint without the route), **C5** (section endings are forced handoffs), **K4** (at most one flat "and then" jump or signpost join), **K5** (swap test) and a new dimension **stakes_load** with **G1** (no figure without a stake before it), **G2** (at most four figures or new entities per 30 s), **G3** (terms glossed at first use, one new concept per section) and **G4** (a human-scale translation of a big figure). The rubric hash changed, so recalibrate (`perturb.js`) before trusting any v3 result, and do not compare v3 scores with v2 runs except dimension by dimension. Raters were unreliable on loops and timing in earlier cohorts (agreement 0.08 to 0.39), so expect C5, H5 and H6 to need the usual reliability screen before they mean anything. The matching hypotheses are in the ledger (H-012 to H-017); a v3 run tests them on the next cohort.

## Extending

- **New language:** add markers to `scripts/lexicon.json` and a preamble to `perturb.js`.
- **New or changed check:** edit `scripts/checklist.js` between cohorts. The rubric hash
  changes, so recalibrate.
- **Comparing runs** (models, rubric versions): `node S/compare.js --a <runA> --b <runB> --out <file.md>`.