content-ingestion

Name: content-ingestion
Availability: InStock
Author: boringdata

Map sitemaps, fetch selected URLs, and extract content as markdown with metadata.

2|Updated Oct 23, 2025

Search Tags:#automation #data collection #web scraping #sitemap #trafilatura #content ingestion #web content

Ready to run

npx skills add https://github.com/boringdata/kurt-demo --skill content-ingestion

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: content-ingestion
Source: https://github.com/boringdata/kurt-demo/tree/main/.claude/skills/ingest-content-skill
Command: npx skills add https://github.com/boringdata/kurt-demo --skill content-ingestion

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trafilatura, and includes scripts (resource) components.

What problem does it solve?

Gathering web content for AI analysis can be slow and inefficient, especially with large websites. This skill streamlines the process with a "map-then-fetch" workflow, allowing selective, fast, and parallel content acquisition.

Core Features & Use Cases

Sitemap Mapping: Quickly discover all URLs from a website's sitemap without downloading content.
Selective Fetching: Review discovered URLs and fetch only the content you need, individually or in parallel batches.
Date Discovery: Automatically extract publish dates from blogrolls and changelogs for freshness tracking.
Use Case: You need to analyze all blog posts from a competitor's website. This skill lets you map their sitemap, filter for blog URLs, and then fetch hundreds of posts in minutes, complete with publish dates, ready for analysis.

Quick Start

Map all URLs from https://www.example.com and then fetch all documents that contain "/blog/".

Frequently Asked Questions about content-ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract web content from a sitemap for AI analysis?▼

Web content extraction from sitemaps uses automated mapping to discover all URLs without downloading, then selective fetching of only the content you need. This skill maps sitemaps first, lets you preview and filter URLs, then fetches them in parallel batches with metadata like publish dates and titles automatically extracted and stored as markdown.

Can I fetch multiple web pages in parallel and extract content as markdown?▼

Yes, this skill supports fast parallel fetches of multiple URLs with automatic markdown extraction. After mapping your sitemap and selecting which URLs to download, it retrieves content in batches, extracts structured data including title, author, and publish date, and stores results with lifecycle tracking (NOT_FETCHED/FETCHED status) in a sources/ directory.

What's the best way to collect blog posts or documentation from a website at scale?▼

Map the target website's sitemap to discover all URLs, filter for content matching your pattern (like /blog/ paths), then fetch selectively in parallel. This approach handles large websites efficiently, automatically extracts publish dates for freshness tracking, and stores all content with metadata for downstream AI analysis.

Do I need to manually list URLs or can I automate sitemap discovery?▼

Sitemap discovery is automated—the skill rapidly maps all URLs from a website's sitemap without requiring manual lists. You review the discovered URLs, optionally filter them, and then proceed to selective fetching, eliminating the need to know every URL in advance.

What metadata does content ingestion extract from web pages?▼

Web content ingestion automatically extracts title, author, publish date, and categories from fetched pages. Metadata is stored alongside content in a sources/ directory with content_path references and status tracking, enabling structured analysis of content freshness and attribution.

How does trafilatura handle content extraction from different website structures?▼

Trafilatura is the underlying extraction library that converts fetched web pages into clean markdown regardless of website structure or layout variations. It handles diverse HTML patterns, removes boilerplate, and preserves structured content, making it suitable for blogs, documentation, product pages, and tutorials.