firecrawl-crawl

Bulk web scraping crawls target sites by following links up to a configurable max-depth, extracting and mapping all pages under a specific section. You can scope the process using include or exclude path filters to target only relevant directories.

Yes, site mapping supports configurable depth control to limit how deep the crawler follows links, alongside include and exclude path filters to restrict crawling to specific URL patterns and exclude irrelevant pages.

Speed up web scraping for large portals by configuring concurrent workers, which process multiple pages simultaneously. The crawler provides progress tracking to monitor the bulk extraction as it scans the site.

Concurrent crawling works directly with both depth limits and path filtering. You can configure concurrent workers to accelerate extraction while simultaneously applying include or exclude path filters to scope the crawl boundaries.

The best way to export crawled site content is to output the discovered URLs and extracted page data directly to a JSON file. This format allows for structured storage and immediate use in further content mapping or analysis workflows.