Crawl documentation sites and convert pages into structured Markdown and JSON.
npx skills add https://github.com/narduk-enterprises/myboat --skill webcrawler
Please help me install this Agent Skill. Skill: webcrawler Source: https://github.com/narduk-enterprises/myboat/tree/main/.github/skills/webcrawler Command: npx skills add https://github.com/narduk-enterprises/myboat --skill webcrawler
Documentation sites are often dispersed and hard to index for offline use. This skill crawls structured documentation portals and converts pages into clean Markdown with metadata to enable offline access, search, and integration with RAG workflows.
Crawl the docs you care about and generate a local knowledge base with Markdown files and a navigable index.
High-intent search queries and answers about installing and using this skill.
You can convert documentation sites to Markdown for offline access by using a web crawler to extract HTML and transform it into structured Markdown files with metadata. This generates a local knowledge base suitable for offline search.
Yes, you can crawl ReadTheDocs or Docusaurus sites for a RAG knowledge base. The crawler recursively harvests documentation portals and outputs machine-readable metadata alongside Markdown, making the corpus ready for embedding workflows.
Yes, the web scraper respects robots.txt by default when extracting documentation. It operates within a configurable domain, depth, and filtering scope to ensure structured documentation is harvested compliantly.
To scrape docs into Markdown, you need Python with the requests, beautifulsoup4, and html2text packages. These libraries handle fetching web pages, parsing HTML structure, and converting content into clean Markdown.
To build an offline knowledge base from API references and tutorials, configure a depth-bounded crawl across your target documentation domain. The process outputs an organized corpus with per-page Markdown files and a master navigable index.
The best way to preserve code blocks when converting HTML documentation to Markdown is to use a crawler with html2text extraction that offers optional code block preservation. This ensures technical formatting remains intact for offline reference.
lpjhelderInstall and configure kronon-forge with Node.js and npm.
jhl-labsRetrieve relevant documents and augment LLM prompts with vector search.
AbilityaiExtract EPUB chapters into separate markdown documents.
JongsixGenerate technical documentation from source code and API definitions.
