site-crawler

Web content extraction for RAG ingestion automates harvesting documentation sites and blogs into clean, structured formats. This skill crawls websites, respects robots.txt, extracts content with markdown conversion, and generates metadata for downstream AI systems.

Respectful web crawling checks robots.txt compliance and applies rate limiting to avoid overloading servers. This skill handles both automatically, enabling compliant harvesting of documentation, blogs, and knowledge bases for data ingestion.

Sitemap-based discovery uses XML sitemaps to identify all pages on a site efficiently. This skill leverages sitemaps for initial crawling and incremental updates, reducing redundant fetches and keeping crawled content synchronized with source changes.

Sitemaps provide structured page discovery and enable efficient incremental updates; robots.txt defines crawl permissions and rate-limiting boundaries. Both are complementary: sitemaps guide *what* to crawl, robots.txt governs *how* to crawl respectfully.

Yes. The skill accepts a base URL and harvests content into named collections. Multiple documentation sites can be crawled separately or combined into a single collection for unified RAG ingestion and downstream indexing.

The skill generates metadata alongside extracted content to support downstream ingestion and indexing. Metadata includes document structure, source URLs, and extraction context, enabling better RAG retrieval and semantic indexing.