SCIENCE-ECOSYSTEM-RUNBOOK.md

# Science YouTube Ecosystem — Setup, Execution Order, Checks, and Debugging

This runbook is written against the supplied `ecosystem-generic` project dump (2026-10-18). It deliberately keeps the first pass lean: collect and validate raw evidence, qualify channels, then score the five queues. Vocabulary building, embeddings, topic ontology, channel profiling, and within-channel performance analysis are optional downstream research stages—not prerequisites for queue ratings.

## 0. Before changing anything

1. Work from the project root—the directory containing `package.json` and `config.js`.
2. Back up the existing `config.js` and, if present, the current `data/` directory.
3. Replace only the root `config.js` with the one in this kit.
4. Confirm the config points at `data/science-ecosystem.db`. Do **not** reuse an art, cricket, geopolitics, Indian-business, or other ecosystem database; a mixed corpus invalidates qualification and rankings.
5. The supplied project dump has `seeds/topics.json` and `seeds/discovery-v2.json` as empty arrays. The new config defines its own science `topicSeeds` and `adjacentSeeds`, which Discovery V2 reads from config. Do not populate old ecosystem terms into the seed files accidentally.
6. `referenceChannels` is intentionally empty to avoid biasing the first discovery pass toward manually selected creators.

## 1. Install and run static/test checks

```bash
npm ci
node --check config.js
npm run test:all
```

**Check before continuing**
- `npm ci` finishes without native-module/build errors for `better-sqlite3`.
- `node --check config.js` exits successfully.
- `npm run test:all` completes. If a test fails, keep the complete failing test name, assertion, stack trace, Node version, and command; fix or diagnose that before interpreting production results.
- Verify the terminal is in the intended project root. Because `dbPath` is resolved from `process.cwd()`, launching commands from another directory can create/use a different database path.

## 2. Initial discovery (Stage 1)

```bash
npm run discover
```

**Check**
- Record the number of search queries, results/videos accepted, channels added, and any rejected/failed results.
- Check that channel URLs are valid YouTube channel URLs/handles and video URLs are valid watch URLs. A URL containing `undefined`, an empty channel URL, or an empty video ID is a data-quality defect.
- Confirm the new database is `data/science-ecosystem.db`, not a previous ecosystem DB.
- A search result is a candidate, not proof that a channel is core. Qualification comes later from recurring evidence.

## 3. Enrich the initial catalogue

Run these in order:

```bash
npm run enrich
npm run channel-metadata
npm run video-metadata
```

`npm run enrich` gathers channel video samples (Popular and Latest) and related records. `channel-metadata` fills static channel facts. `video-metadata` backfills video fields such as views, dates, and duration where available.

**Check after each command**
- Logs should report processed/skipped/failed channels and returned Popular/Latest counts. Do not assume that a tab label proves the returned list is correctly sorted; inspect `channel_videos.source` and sample positions/titles.
- Check channel metadata coverage for `channel_id`, `description`, `subscriber_count`, `video_count`, `total_view_count`, `joined_date`, and `canonical_url`/`handle` where exposed by YouTube.
- Check video coverage for `view_count`, publication date (raw and normalized), and `duration_seconds`.
- Distinguish source-unavailable data from fetch failures. A null value is not evidence of zero; a fetch error should be visible and retryable.
- Keep an eye out for `InnertubeError: Request contains an invalid argument`, `ParsingError: Type mismatch`, unavailable channel feeds, and repeated zero-video returns. Log the channel URL, feed mode, page/cursor, error message, and whether the same channel succeeds on retry.

## 4. Normalize and repair temporal fields

```bash
npm run normalize:temporal
npm run dates:repair
```

`dates:repair` is a dry run by default. Review the candidate sample first. If the proposed corrections are plausible, apply them:

```bash
npm run dates:repair -- --apply
```

**Check**
- A human-readable relative date such as `2 weeks ago` must not be silently treated as an exact calendar date without a documented normalization rule.
- Compare raw display text (`published_at_text`) with normalized date fields and precision markers. Keep raw text for auditability.
- If `dates:repair` finds candidates, check the table counts and examples before applying. It updates only rows whose display text parses as an absolute date and whose normalized value differs.
- Rerun `npm run normalize:temporal` after later ingestion/backfills; new videos do not inherit an earlier normalization pass.

## 5. First enrichment coverage checkpoint

```bash
npm run enrichment:report
```

Open the newest files in `reports/enrichment/` (especially `coverage.md` and `field-gaps.csv`). This is the main checkpoint before qualification and scoring.

**Check**
- Channel count and video count are plausible.
- Channels with no linked videos are identified.
- Missing channel metadata, video views, publication dates, durations, and qualifications are quantified.
- Read the recommended actions in the report rather than trying to enrich every field indiscriminately.
- A channel profile is marked research-only in this report and does not affect queue scores.

## 6. Expand discovery with Discovery V2

Use a named phase ID so the results are easy to find and reproduce:

```bash
npm run discovery:v2 -- --phase=science-round-001
npm run discovery:v2:report -- --phase=science-round-001
```

LLM-generated search expansion is disabled by default. Keep it that way for the baseline. If you intentionally want it, opt in explicitly:

```bash
npm run discovery:v2 -- --phase=science-round-001-llm --llm
```

**Check**
- Inspect `reports/discovery-v2/` for the phase report and new-channel CSV.
- Check the counts in `discovery_v2_phases`, `discovery_v2_queries`, `discovery_v2_channel_events`, and `discovery_v2_scores`.
- A phase with `newly-added=0` is not automatically a crash: it can mean the queries found no new channels, or results were already in the database. Inspect query statuses, result counts, errors, and `was_new_channel` before deciding.
- Discovery V2 score is a **discovery triage score**, not one of the five queue scores.
- Routes A/B can derive queries from the existing corpus; route C uses configured science topic seeds; route D explores adjacent fields. If the starting corpus is tiny, expect the corpus-derived routes to be weaker until the first pass is enriched.

## 7. Enrich channels found by Discovery V2

```bash
npm run discovery:v2:enrich -- --phase=science-round-001 --limit=200 --catalogue=40
```

The current stage selects channels marked newly added for the specified phase. It attempts channel metadata and, by default, a catalogue sample (up to 40 per feed mode). This is not a guarantee that 40 Popular plus 40 Latest unique videos will be saved.

**Check**
- Review the final `fetched`, `skipped`, `failed`, `catalogueVideos`, and `catalogueFailures` counters.
- If a channel is skipped, check whether required metadata fields were already present. Use `--force` only when a deliberate refresh is wanted.
- If a channel is missing from enrichment, verify that its phase event has `was_new_channel=1`; this stage does not automatically re-enrich every previously known channel found again.
- For important channels that need a larger sample, selectively run catalogue mining rather than expanding every channel indiscriminately:

```bash
npm run catalogue:mine -- --channel-url="https://www.youtube.com/@HANDLE" --limit-per-channel=300
```

Replace the example URL with the exact channel URL from the database. Check the command's own help/output if flags differ in your checkout. A cap of 300 is a collection limit, not proof that YouTube exposed 300 accessible videos.

Then repeat the relevant backfills for newly collected data:

```bash
npm run channel-metadata
npm run video-metadata
npm run normalize:temporal
npm run dates:repair
npm run enrichment:report
```

If date repair proposes changes, review first and use `--apply` only after that review.

## 8. Qualify channels for the science ecosystem

```bash
npm run qualification:llm
```

Configure `OLLAMA_HOST` and `OLLAMA_MODEL` in the environment if using the local Ollama/Gemma setup. The exact model must be available at that endpoint. Run the stage after discovery/enrichment, not before, or the newly added channels will remain unqualified until a later run.

**Check**
- Review the qualification report(s) in `reports/` and the `channel_qualification_llm_results` processing statuses.
- Confirm classifications use the configured labels: `core`, `adjacent`, `unknown`, `irrelevant`.
- `core` requires recurring science communication/history/explanation evidence; a keyword match or one science video is not enough.
- `adjacent` is recurring overlap, not the main subject. `unknown` is the correct result for insufficient evidence. Missing evidence must not be converted to zero or treated as irrelevant.
- If many channels remain missing/stale, inspect failures and input fingerprints before retrying. Do not hide a failed qualification with a broad catch-and-continue.

## 9. Preview and persist the five queue ratings

First produce a preview:

```bash
npm run queues:rate -- --notapply
```

Review the generated Markdown/JSON reports under `reports/` and the qualification diagnostics. If the output and coverage look sound, persist the ratings:

```bash
npm run queues:rate
```

In the supplied project version, the normal command persists rows; `--notapply` is the preview-only switch. Do not rely on `--apply` as a required flag in this version.

**Check**
- Confirm the run reports the expected channel count, five queues, and the number of ratings written.
- Confirm the dedicated tables exist and receive rows: `channel_queue_learn_from`, `channel_queue_breaking_out`, `channel_queue_durable_exemplars`, `channel_queue_unexplored_niches`, and `channel_queue_reference_inspiration`. The compatibility tables `channel_queue_runs` and `channel_queue_scores` may also be populated.
- Check `score`, `rank`, `confidence`, `coverage`, `sample_size`, `status`, `reason_codes_json`, `evidence_refs_json`, `components_json`, and `rationale` for each rating. Interpret score only alongside eligibility/status, confidence, and evidence coverage.
- Re-run qualification before queue scoring when the ecosystem config, channel evidence, or qualification inputs changed. The scorer checks qualification freshness; stale/missing LLM assessments should remain low-confidence/pending rather than silently becoming confident ratings.
- These five queues answer different questions. A channel can appear in multiple queues. Breaking-out is a static potential proxy unless repeated snapshots establish momentum.

## 10. Discovery score and final coverage check

```bash
npm run discovery:v2:score -- --phase=science-round-001
npm run enrichment:report
```

Use discovery scores only to prioritize further investigation of newly discovered channels. Do not compare them directly with the five queue scores.

## 11. Tables to inspect and what each tells you

| Table | Why it matters |
|---|---|
| `channels` | Canonical channel rows and static fields copied onto them |
| `channel_metadata` | Channel metadata fetch results, precision/provenance, and `fetch_error` |
| `videos` | Current captured video facts: title, URL, channel link, views, publication date, duration |
| `channel_videos` | Relationship between channel and videos; `source` and `position` help inspect Popular/Latest feed capture |
| `video_metric_snapshots` | Historical video measurements; useful for trends, but current queue scoring in this version uses static captured video fields rather than snapshot fallback |
| `channel_metric_snapshots` | Historical subscriber/total-view/video-count observations; helpful for time-series growth, not a substitute for missing static fields |
| `channel_qualifications` | Canonical stored ecosystem classification/reason/method |
| `channel_qualification_llm_results` | LLM result history, status, timestamp, and input freshness diagnostics |
| `channel_queue_runs`, `channel_queue_scores` | Run-level and compatibility rating records |
| `channel_queue_learn_from`, `channel_queue_breaking_out`, `channel_queue_durable_exemplars`, `channel_queue_unexplored_niches`, `channel_queue_reference_inspiration` | The five dedicated queue-specific rating tables |
| `discovery_v2_phases`, `discovery_v2_queries`, `discovery_v2_channel_events`, `discovery_v2_scores` | Discovery phases, query execution, channel discovery events, and triage scores |

For metrics like subscribers per upload, views per upload, channel size, and age: first verify the underlying static fields exist (`subscriber_count`, `video_count`, `total_view_count`, `joined_date`) and that the video sample is representative. Ratios are undefined when the denominator is missing/zero; leave them null. A single captured subscriber count or view count is not growth. Use snapshots only when measuring change over time, and record observation dates/source.

## 12. Debugging playbook

### A. `InnertubeError: Request contains an invalid argument`
- Capture the exact channel URL, request/feed mode, page/cursor, package versions, and full error.
- Retry without `--force` first so successful cached/stored work is not needlessly repeated.
- Separate channel metadata failure from Popular/Latest catalogue failure. One may succeed while the other fails.
- Do not mark a channel irrelevant because the API failed.

### B. `ParsingError: Type mismatch` (for example, a title/header view expected another view)
- Treat it as a YouTube page-shape/library compatibility issue, not a bad science classification.
- Record the channel, feed mode, youtubei.js version, and stack. Retry later or with a fallback fetch path if the project implements one; do not fabricate missing fields.

### C. Channel metadata or channel table looks empty
- Verify the process prints the intended database path.
- Confirm `npm run discover` completed and `channels` contains rows.
- Check `channel_metadata.fetch_error`, timestamps, and required fields. The channel row and metadata row are separate layers.
- Do not switch database files or delete the DB to make a report look clean.

### D. Many channels have no videos or only 1–2 videos
- Inspect `channel_videos` grouped by `source`, then compare against `videos` and the channel's declared `video_count`.
- Run `catalogue:mine` selectively for channels important to analysis, with a bounded limit (e.g. 300).
- Check `channel_url`, `video_id`, and canonical URL formation. Never accept `https://www.youtube.comundefined` as valid.
- Distinguish “not fetched”, “fetch failed”, “feed returned zero”, and “feed returned fewer than requested”; log each separately.

### E. Dates look wrong or videos disappear from recency-based calculations
- Run `npm run video-metadata`, `npm run normalize:temporal`, then `npm run dates:repair` in dry-run mode.
- Compare `published_at_text`, `published_at`, `published_at_normalized`, and precision columns. Do not infer exact dates from month/year-only or relative strings.
- Apply date repair only after reviewing examples. Repeat normalization after any later metadata ingestion.

### F. Queue qualification coverage is low / many scores are unsupported
- Run `npm run enrichment:report` and inspect its coverage tables before changing thresholds.
- Verify qualification was run after the latest discovery phase.
- Inspect LLM status, errors, prompt version/input fingerprint freshness, missing descriptions, and sparse video samples.
- Do not lower confidence thresholds just to make more channels rank. Improve evidence or retain `unknown`/pending.

### G. `discovery:v2:score` reports `newly-added=0` or “No discovery phase found”
- Ensure `npm run discovery:v2` ran successfully in the same project root/database.
- Use the exact phase ID shown in the discovery log/report; list `discovery_v2_phases` and inspect phase status.
- Check `discovery_v2_channel_events` for that phase, especially `was_new_channel`, and query statuses/result counts/errors in `discovery_v2_queries`.
- A zero-new-channel phase can be legitimate if results were already known. The score is triage, not queue rating.

### H. LLM/Ollama failures
- Confirm `OLLAMA_HOST` is reachable and `OLLAMA_MODEL` names a model actually installed there.
- Test one small batch before scaling. Keep the first pass deterministic and LLM query expansion disabled unless intentionally testing it.
- Preserve the error, model name, prompt version, batch size, and channel IDs. Do not report failed calls as successful classifications.

### I. Avoid wasting time on optional stages
- Do **not** run these as prerequisites for the five queues: `vocabulary:build`, `vocabulary:review`, `features:build`, `videos:embed`, `priorities:build`, or `profile:channels`.
- `profile:channels`, `profile:videos`, `analyze:corpus`, embeddings, vocabulary, and within-channel performance are downstream research tools. Use them after the raw evidence and queue baseline are healthy.
- `video-signals` may be useful for richer signals but is not part of the lean default pass; run it only when a concrete analysis needs its fields.
- After any new catalogue ingestion, re-run the relevant metadata/date repair, qualification, queue rating, and coverage report stages.

## 13. Minimal repeatable first-pass sequence

```bash
npm ci
node --check config.js
npm run test:all
npm run discover
npm run enrich
npm run channel-metadata
npm run video-metadata
npm run normalize:temporal
npm run dates:repair
npm run enrichment:report
npm run discovery:v2 -- --phase=science-round-001
npm run discovery:v2:report -- --phase=science-round-001
npm run discovery:v2:enrich -- --phase=science-round-001 --limit=200 --catalogue=40
npm run channel-metadata
npm run video-metadata
npm run normalize:temporal
npm run dates:repair
npm run enrichment:report
npm run qualification:llm
npm run queues:rate -- --notapply
# Inspect the preview, then persist:
npm run queues:rate
npm run discovery:v2:score -- --phase=science-round-001
npm run enrichment:report
```

For date repair, the command without `--apply` is a dry run. If it reports candidates, inspect them and run `npm run dates:repair -- --apply` only when the corrections are valid.