Docs/Skills and code/Project documents

catalogue-crawl-investigation-v1.md

Markdown19 lines1.4 KBProject documents

Mentioned on: page 02

YouTube catalogue acquisition notes

Goals

Keep channels as the primary research objects and videos as evidence. Preserve stable IDs, raw observations, timestamps, pagination status, partial failures, and provenance.

Catalogue enumeration and enrichment

These are separate tasks:

1. Enumerate accessible video IDs and basic metadata, following continuation pages until the feed genuinely ends or a configured limit is reached.
2. Enrich records with views, duration, publication timestamps, descriptions, and other available fields.
3. Preserve snapshots for time-dependent metrics; one snapshot cannot establish velocity.
4. Track completeness explicitly (not_started, in_progress, complete, partial, blocked).
5. Distinguish failed requests, parser failures, missing fields, and genuinely empty feeds.

Acquisition tradeoffs

The project uses youtubei.js/InnerTube and therefore depends on an internal, changeable response format. A production-scale crawl should be bounded, throttled, checkpointed, resumable, deduplicated, and transparent about partial results. An official API integration may be preferable where API credentials and quota are available.

Validation

Before relying on a sort order, verify that returned records actually satisfy the requested ordering. A locally sorted sample is not a globally complete popular list unless catalogue coverage supports that claim.

# YouTube catalogue acquisition notes

## Goals
Keep channels as the primary research objects and videos as evidence. Preserve stable IDs, raw observations, timestamps, pagination status, partial failures, and provenance.

## Catalogue enumeration and enrichment
These are separate tasks:
1. Enumerate accessible video IDs and basic metadata, following continuation pages until the feed genuinely ends or a configured limit is reached.
2. Enrich records with views, duration, publication timestamps, descriptions, and other available fields.
3. Preserve snapshots for time-dependent metrics; one snapshot cannot establish velocity.
4. Track completeness explicitly (`not_started`, `in_progress`, `complete`, `partial`, `blocked`).
5. Distinguish failed requests, parser failures, missing fields, and genuinely empty feeds.

## Acquisition tradeoffs
The project uses `youtubei.js`/InnerTube and therefore depends on an internal, changeable response format. A production-scale crawl should be bounded, throttled, checkpointed, resumable, deduplicated, and transparent about partial results. An official API integration may be preferable where API credentials and quota are available.

## Validation
Before relying on a sort order, verify that returned records actually satisfy the requested ordering. A locally sorted sample is not a globally complete popular list unless catalogue coverage supports that claim.

Shown as it is in the repository at build time (2026-11-03); the home directory appears as <home>. If this page and the file differ, the file is right. Back to the catalogue.