scrape-firecrawl

What problem does it solve?

Manually collecting and maintaining a complete set of documentation pages is slow, error-prone, and hard to keep as a source-of-truth for downstream knowledge extraction.

Core Features & Use Cases

• Full docs crawling: Captures a site using Firecrawl endpoints to recursively discover and retrieve every relevant page.
• Clean, persisted corpus: Stores each page as cleaned markdown plus link metadata, organized under a topic’s 20-sources/ directory.
• Downstream-ready manifests: Appends per-page rows to _manifest.md and the global INDEX/sources.md for consistent reuse by later pipelines.
• Content integrity & retries: Computes a SHA-256 content hash for verification and records failures for later retry without re-fetching.

Quick Start

Run the scrape-firecrawl pipeline on a documentation base URL with your include/exclude path regexes so it generates topics/<topic_id>/20-sources/ with content.md and _meta.md for every captured page.