skills.rest
robit-man
robit-manCOMMUNITY
Explore all skills by this author →
GitHub

doc-scraper

Scrape documentation websites into categorized Markdown reference files.

Updated Mar 11, 2026
Search Tags:#automation#documentation#knowledge base#web scraping#content extraction#reference generation
One-click install
npx skills add https://github.com/robit-man/transcribe-cli --skill doc-scraper-robit-man
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: doc-scraper
Source: https://github.com/robit-man/transcribe-cli/tree/main/.claude/skills/doc-scraper
Command: npx skills add https://github.com/robit-man/transcribe-cli --skill doc-scraper-robit-man

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates the process of converting entire documentation websites into structured, categorized reference files, making them easily searchable and usable for building AI skills or for offline archives.

Core Features & Use Cases

  • Automated Web Scraping: Extracts content from specified URLs based on configurable patterns.
  • Content Categorization: Organizes scraped content into predefined categories.
  • Configuration-driven: Uses a JSON configuration file for detailed control over selectors, URL patterns, and rate limiting.
  • Use Case: You need to create a searchable knowledge base from a complex API documentation website. This Skill can scrape all relevant pages, categorize them by API endpoint, concept, or guide, and save them as Markdown files.

Quick Start

Use the doc-scraper skill to scrape the documentation at https://docs.example.com/ into the output directory.

Frequently Asked Questions about doc-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape documentation websites into structured reference files?▼

Scrape documentation websites into structured reference files by configuring a JSON file with target URLs, content selectors, and output structure. The scraper parses content based on your defined selectors, categorizes pages, and saves them as organized Markdown files for offline archives.

Can I automate web scraping to build a searchable knowledge base from API documentation?▼

Automate web scraping to build a searchable knowledge base by extracting content from API documentation URLs. The process categorizes scraped pages by endpoint, concept, or guide based on predefined patterns, creating structured Markdown reference files for AI skill development.

Does this documentation scraper require configuration before extracting content?▼

Documentation scraping requires JSON configuration to specify target URLs, content selectors, and output structure. You must define URL patterns and rate limiting parameters in the configuration file before initiating the content extraction process.

How does rate limiting and checkpointing work when scraping large documentation sites?▼

Rate limiting and checkpointing manage scraping processes by controlling request frequency and saving progress. This prevents server overload and allows the scraper to resume interrupted content extraction tasks without losing previously categorized reference files.

What is the best way to organize scraped web content into predefined categories?▼

Organize scraped web content into predefined categories by defining URL patterns and content selectors in a JSON configuration file. The scraper parses documentation pages and automatically categorizes them into structured Markdown reference files based on your specified output structure.

Can I use scraped documentation references for offline archives and AI skill development?▼

Scraped documentation references serve as structured Markdown files for AI skill development and offline archives. The scraper converts entire documentation websites into categorized, searchable knowledge bases that can be used independently of the original web source.

Related Skills

Explore More Software Engineering Skills→
winsorllcwinsorllc

link-scraper

Fetch, extract, and summarize web content with Node.js and cheerio.

Community
Intermediate
Gimpy666Gimpy666
1

scraped_docs

Convert Tribot JavaDoc HTML pages into per-class Markdown files and a consolidated reference.

Community
Intermediate
XJTLUmediaXJTLUmedia
28

web-reader

Extract structured content and metadata from public web URLs.

Community
Intermediate
aaronjmarsaaronjmars
17

Update Gallery

Extract and format articles, logs, and site data for GitHub Pages publishing.

Community
Intermediate