What problem does it solve?
Manually saving every post from a blog or newsletter archive into a knowledge base is slow, error-prone, and produces duplicates on re-runs. This Skill automates whole-publication ingestion: it discovers the feed, walks the full archive, normalizes each post, deduplicates by canonical URL, and writes clean source pages that are safe to re-run.
Core Features & Use Cases
- Feed Discovery & Archive Walking: Finds RSS/Atom feeds via link tags, conventional paths, or sitemaps, then paginates through the full archive before fetching any post bodies.
- Canonical-URL Deduplication: Strips tracking parameters and resolves redirects so re-runs skip existing pages instead of duplicating them, with slug-collision handling for same-titled posts.
- Pacing, Paywall & Husk Handling: Paces requests with 429 backoff, skips gated Substack posts without workarounds, and detects and repairs or deletes near-empty "husk" pages.
- Use Case: Point it at a Substack publication and it enumerates every public post, writes one clean Markdown page per article under
sources/articles/<publication-slug>/, then hands the batch to the brain-ingest-gate skill for entity and backlink enrichment.
Quick Start
Ask the agent to ingest this whole blog into the brain, walking its feed and writing one deduplicated page per public post.