content-ingestion

Web content extraction from sitemaps uses automated mapping to discover all URLs without downloading, then selective fetching of only the content you need. This skill maps sitemaps first, lets you preview and filter URLs, then fetches them in parallel batches with metadata like publish dates and titles automatically extracted and stored as markdown.

Yes, this skill supports fast parallel fetches of multiple URLs with automatic markdown extraction. After mapping your sitemap and selecting which URLs to download, it retrieves content in batches, extracts structured data including title, author, and publish date, and stores results with lifecycle tracking (NOT_FETCHED/FETCHED status) in a sources/ directory.

Map the target website's sitemap to discover all URLs, filter for content matching your pattern (like /blog/ paths), then fetch selectively in parallel. This approach handles large websites efficiently, automatically extracts publish dates for freshness tracking, and stores all content with metadata for downstream AI analysis.

Sitemap discovery is automated—the skill rapidly maps all URLs from a website's sitemap without requiring manual lists. You review the discovered URLs, optionally filter them, and then proceed to selective fetching, eliminating the need to know every URL in advance.

Web content ingestion automatically extracts title, author, publish date, and categories from fetched pages. Metadata is stored alongside content in a sources/ directory with content_path references and status tracking, enabling structured analysis of content freshness and attribution.

Trafilatura is the underlying extraction library that converts fetched web pages into clean markdown regardless of website structure or layout variations. It handles diverse HTML patterns, removes boilerplate, and preserves structured content, making it suitable for blogs, documentation, product pages, and tutorials.