firecrawl-crawl

To bulk extract website content, you can crawl linked web pages across an entire site in one operation. This process follows links to retrieve content from multiple pages and outputs the extracted results to a JSON file.

Yes, you can limit web crawling to a specific documentation section by applying URL path filtering. You can scope crawls by targeting specific paths like /docs, setting page count limits, and defining link depth.

The best way to collect pages for a knowledge base is using scoped web crawling. This approach retrieves content from multiple linked pages concurrently, applying path filtering and depth limits to gather targeted site sections efficiently.

Yes, you need the Firecrawl CLI to execute this website scraping workflow. The tool relies on the CLI to run concurrent extraction, asynchronous job polling, and progress reporting for your crawling tasks.

Asynchronous job polling during website scraping works by tracking the progress of concurrent extraction tasks. It allows you to monitor multi-page crawling workflows and report progress until all targeted pages are extracted into JSON output.