doc-scraper

Scrape documentation websites into organized reference files with configurable selectors.

175|26|Updated Aug 14, 2025

Search Tags:#automation #documentation #reference #website #scraping

Ready to run

npx skills add https://github.com/jmagly/ai-writing-guide --skill doc-scraper-jmagly

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: doc-scraper
Source: https://github.com/jmagly/ai-writing-guide/tree/main/.factory/skills/doc-scraper
Command: npx skills add https://github.com/jmagly/ai-writing-guide --skill doc-scraper-jmagly

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Converts documentation websites into organized, categorized reference files suitable for Claude skills or offline archives.

Core Features & Use Cases

Grounding checks: Verify target URL accessibility and structure.
Config templates: Generate scraper config with selectors and categories.
Output validation: Ensure references and metadata are properly structured.

Quick Start

Scrape a docs site into output/<skill-name>/references with a basic config.

Frequently Asked Questions about doc-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scrape documentation websites into structured reference files?▼

Scraping documentation websites converts them into organized reference files by extracting main content, titles, and code blocks using configurable selectors. The skill handles URL patterns, category mappings, and rate limits to produce deterministic, resumable outputs suitable for Claude skills or offline archives.

What documentation site structures does the scraper support?▼

The scraper supports sites with identifiable main content, titles, and code blocks. It applies to API references, guides, and tutorials with configurable CSS selectors and URL include/exclude patterns to adapt to different site layouts.

Can I resume a documentation scrape if it fails partway through?▼

Yes, the scraper enforces configurable workflow steps and recoverable checkpoints, enabling deterministic, resumable scrapes. You can pick up from where the process stopped without re-scraping already-collected content.

How do I verify a documentation site before scraping it?▼

The skill includes grounding checks that verify target URL accessibility and structure before scraping begins, ensuring the site is reachable and contains the expected content layout.

What configuration do I need to scrape a documentation site?▼

The skill generates config templates with CSS selectors, URL patterns, category mappings, and rate-limit settings. Output structure, reference organization, and scrape scope are all configurable to match your docs site and archive needs.

How does the scraper validate the final reference output?▼

Output validation ensures references and metadata are properly structured before saving. The skill produces organized, categorized reference files with validated data ready for use in Claude skills or offline archives.