Verified:GitHub

scrape-firecrawl

Crawl documentation sites via Firecrawl and store pages as markdown with manifests.

8|1|Updated Apr 20, 2018

Search Tags:#firecrawl #web scraping #data provenance #markdown extraction #documentation crawling #knowledge corpus #link metadata

Ready to run

npx skills add https://github.com/Executioner1939/awesome-engineering-research --skill scrape-firecrawl

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: scrape-firecrawl
Source: https://github.com/Executioner1939/awesome-engineering-research/tree/main/.claude/skills/scrape-firecrawl
Command: npx skills add https://github.com/Executioner1939/awesome-engineering-research --skill scrape-firecrawl

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manually collecting and maintaining a complete set of documentation pages is slow, error-prone, and hard to keep as a source-of-truth for downstream knowledge extraction.

Core Features & Use Cases

Full docs crawling: Captures a site using Firecrawl endpoints to recursively discover and retrieve every relevant page.
Clean, persisted corpus: Stores each page as cleaned markdown plus link metadata, organized under a topic’s 20-sources/ directory.
Downstream-ready manifests: Appends per-page rows to _manifest.md and the global INDEX/sources.md for consistent reuse by later pipelines.
Content integrity & retries: Computes a SHA-256 content hash for verification and records failures for later retry without re-fetching.

Quick Start

Run the scrape-firecrawl pipeline on a documentation base URL with your include/exclude path regexes so it generates topics/<topic_id>/20-sources/ with content.md and _meta.md for every captured page.

Frequently Asked Questions about scrape-firecrawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I crawl a documentation site and save it as a markdown corpus?▼

You can crawl a documentation site and save it as a markdown corpus by using a recursive web scraping tool that discovers pages, extracts markdown, and stores content under a dedicated sources directory for downstream analysis.

Can I use include and exclude path filters to control web scraping scope?▼

Yes, web scraping scope is controlled using base URL selection alongside include and exclude path regex filters, ensuring only relevant documentation pages are discovered and retrieved during the recursive crawl.

How does a documentation crawler avoid re-fetching pages during pipeline runs?▼

A documentation crawler avoids re-fetching pages by computing a SHA-256 content hash for integrity verification and updating manifest files, allowing subsequent pipelines to use the stored markdown corpus as the source-of-truth without duplicate requests.

What is the best way to maintain a documentation knowledge corpus for downstream analysis?▼

The best way to maintain a documentation knowledge corpus is to persist cleaned markdown and link metadata in organized directories while appending per-page rows to a manifest, ensuring consistent reuse by downstream data extraction pipelines.

Does Firecrawl web scraping handle content integrity and failed page retries?▼

Firecrawl web scraping handles content integrity by recording SHA-256 content hashes and logging failures for later retry, ensuring the persisted markdown corpus remains complete and verifiable without re-fetching successful pages.