webcrawler

What problem does it solve?

Documentation sites are often dispersed and hard to index for offline use. This skill crawls structured documentation portals and converts pages into clean Markdown with metadata to enable offline access, search, and integration with RAG workflows.

Core Features & Use Cases

• Recursive, depth-bounded crawling of documentation sites (ReadTheDocs, GitBook, Docusaurus, MkDocs) to surface relevant content.
• HTML-to-Markdown extraction with optional code blocks preservation and source attribution.
• Output an organized corpus with per-page Markdown files, a master index, and machine-readable metadata suitable for embedding.

Quick Start

Crawl the docs you care about and generate a local knowledge base with Markdown files and a navigable index.