webcrawler

Crawl documentation sites and convert pages into structured Markdown and JSON.

Updated Aug 23, 2026

Search Tags:#documentation #markdown #RAG #offline #web-scraping #knowledge-base #crawl

Ready to run

npx skills add https://github.com/narduk-enterprises/myboat --skill webcrawler

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: webcrawler
Source: https://github.com/narduk-enterprises/myboat/tree/main/.github/skills/webcrawler
Command: npx skills add https://github.com/narduk-enterprises/myboat --skill webcrawler

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, html2text, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Documentation sites are often dispersed and hard to index for offline use. This skill crawls structured documentation portals and converts pages into clean Markdown with metadata to enable offline access, search, and integration with RAG workflows.

Core Features & Use Cases

Recursive, depth-bounded crawling of documentation sites (ReadTheDocs, GitBook, Docusaurus, MkDocs) to surface relevant content.
HTML-to-Markdown extraction with optional code blocks preservation and source attribution.
Output an organized corpus with per-page Markdown files, a master index, and machine-readable metadata suitable for embedding.

Quick Start

Crawl the docs you care about and generate a local knowledge base with Markdown files and a navigable index.

Frequently Asked Questions about webcrawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I convert documentation sites to Markdown for offline access?▼

You can convert documentation sites to Markdown for offline access by using a web crawler to extract HTML and transform it into structured Markdown files with metadata. This generates a local knowledge base suitable for offline search.

Can I crawl ReadTheDocs or Docusaurus sites for a RAG knowledge base?▼

Yes, you can crawl ReadTheDocs or Docusaurus sites for a RAG knowledge base. The crawler recursively harvests documentation portals and outputs machine-readable metadata alongside Markdown, making the corpus ready for embedding workflows.

Does the web scraper respect robots.txt when extracting documentation?▼

Yes, the web scraper respects robots.txt by default when extracting documentation. It operates within a configurable domain, depth, and filtering scope to ensure structured documentation is harvested compliantly.

What Python dependencies do I need to scrape docs into Markdown?▼

To scrape docs into Markdown, you need Python with the requests, beautifulsoup4, and html2text packages. These libraries handle fetching web pages, parsing HTML structure, and converting content into clean Markdown.

How do I build an offline knowledge base from API references and tutorials?▼

To build an offline knowledge base from API references and tutorials, configure a depth-bounded crawl across your target documentation domain. The process outputs an organized corpus with per-page Markdown files and a master navigable index.

What is the best way to preserve code blocks when converting HTML documentation to Markdown?▼

The best way to preserve code blocks when converting HTML documentation to Markdown is to use a crawler with html2text extraction that offers optional code block preservation. This ensures technical formatting remains intact for offline reference.