firecrawl-crawl

Bulk extract content from an entire website section by crawling all reachable URLs from a starting page. The crawl automatically follows links to a specified depth or limit while extracting page content in bulk.

You can restrict web crawling to specific documentation paths by applying include and exclude path filters. This targets exact sections like /docs/ and prevents the crawler from ingesting unrelated site areas.

The best way to scrape a multi-page site for downstream processing is running a link-following bulk ingestion crawl. It gathers comprehensive coverage of a site area and outputs JSON results for immediate use.

This web crawling approach supports concurrency and delay tuning to manage request rates. It also offers optional synchronous waiting with progress reporting to monitor the bulk extraction status.

You need to configure depth and limit controls for documentation ingestion to prevent unbounded crawling. Setting a reasonable limit ensures the bulk scrape completes efficiently within your desired site boundaries.

Your bulk scrape includes pages outside the target site area because path filtering is not applied. You must explicitly set include-paths or exclude-paths to scope the crawl and restrict coverage to your desired section.