blog-ingest

Ingest entire blogs, newsletters, and RSS/Atom feeds into deduplicated brain source pages.

29.4k|4.4k|Updated Apr 5, 2026
One-click install
npx skills add https://github.com/garrytan/gbrain --skill blog-ingest
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: blog-ingest
Source: https://github.com/garrytan/gbrain/tree/main/plugin/skills/blog-ingest
Command: npx skills add https://github.com/garrytan/gbrain --skill blog-ingest

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manually saving every post from a blog or newsletter archive into a knowledge base is slow, error-prone, and produces duplicates on re-runs. This Skill automates whole-publication ingestion: it discovers the feed, walks the full archive, normalizes each post, deduplicates by canonical URL, and writes clean source pages that are safe to re-run.

Core Features & Use Cases

  • Feed Discovery & Archive Walking: Finds RSS/Atom feeds via link tags, conventional paths, or sitemaps, then paginates through the full archive before fetching any post bodies.
  • Canonical-URL Deduplication: Strips tracking parameters and resolves redirects so re-runs skip existing pages instead of duplicating them, with slug-collision handling for same-titled posts.
  • Pacing, Paywall & Husk Handling: Paces requests with 429 backoff, skips gated Substack posts without workarounds, and detects and repairs or deletes near-empty "husk" pages.
  • Use Case: Point it at a Substack publication and it enumerates every public post, writes one clean Markdown page per article under sources/articles/<publication-slug>/, then hands the batch to the brain-ingest-gate skill for entity and backlink enrichment.

Quick Start

Ask the agent to ingest this whole blog into the brain, walking its feed and writing one deduplicated page per public post.

Frequently Asked Questions about blog-ingest

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I ingest an entire blog or newsletter archive at once?

Provide the publication URL and the skill discovers its RSS/Atom feed or sitemap, enumerates the full archive, then writes one normalized Markdown page per public post. It tests on 3-5 posts first, then ramps to the full archive with pacing between fetches.

How does RSS feed ingestion avoid duplicates on re-runs?

Each post is keyed by its canonical URL after stripping tracking parameters and resolving redirects. Before writing, the brain is searched for that URL; existing pages are skipped, making re-runs idempotent and cheap.

Can I ingest paywalled or gated Substack posts?

No. Gated posts are detected via paywall markers or truncated preview bodies and skipped with a logged reason. The skill never uses alternate endpoints, session cookies, or credentials to reach gated content.

What is the difference between blog-ingest and idea-ingest?

blog-ingest handles publication scope: whole blogs, feeds, and archives requiring enumeration. idea-ingest handles single-item scope: one article or essay URL in, one page out. Single URLs are routed away from blog-ingest automatically.

What happens when a fetch returns an empty or broken page?

Pages under roughly 50 words or matching subscribe/paywall boilerplate are flagged as husks after the run. Each husk is re-fetched slowly; real content rewrites the page, while confirmed gated posts are deleted and logged as skipped.

How does the skill handle HTTP 429 rate limits?

Requests are paced at 1.5 seconds by default. On a 429 it backs off exponentially from 5 seconds up to a 30-second cap, honoring Retry-After headers, and pauses the run after repeated 429s from the same host.