What problem does it solve? Extracting content from many pages on the same website one URL at a time is slow and tedious. This Skill automates bulk crawling of an entire site or a scoped section (like /docs), following links with depth, path, and concurrency controls. ## Core Features & Use Cases - Bulk Site Crawling: Follow links from a starting URL and extract content from every discovered page, with configurable depth and page limits. - Path Scoping: Use --include-paths and --exclude-paths to crawl only the sections you need, such as all pages under /docs. - Async Job Control: Run crawls with --wait and --progress for immediate results, or poll a job ID for long-running crawls. - Use Case: You need the full content of a product's documentation section for analysis. Run a crawl scoped to /docs with a page limit, wait for completion, and save the structured JSON output to a local file. ## Quick Start Ask the AI to crawl a website's docs section with Firecrawl, limiting it to 50 pages under /docs, and save the extracted content to a local JSON file.
