skills.rest
leonardoacosta
leonardoacostaCOMMUNITY
Explore all skills by this author →
GitHub

firecrawl-crawl

Crawls websites following links to bulk-extract content from multiple pages.

Updated Aug 2, 2026
Search Tags:#cli#firecrawl#web crawling#content extraction#site scraping#bulk extraction
One-click install
npx skills add https://github.com/leonardoacosta/agents --skill firecrawl-crawl-leonardoacosta
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: firecrawl-crawl
Source: https://github.com/leonardoacosta/agents/tree/main/skills/firecrawl-crawl
Command: npx skills add https://github.com/leonardoacosta/agents --skill firecrawl-crawl-leonardoacosta

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Extracting content from many pages across a website one URL at a time is slow and error-prone. This Skill automates bulk extraction by crawling a site or section, following links up to a configurable depth and limit, and saving the results to a local JSON file. ## Core Features & Use Cases - Bulk Site Crawling: Crawl an entire site or a scoped section (e.g., everything under /docs) with depth, path, and page-count limits. - Async Job Management: Run crawls synchronously with --wait or receive a job ID for asynchronous status polling. - Credit Awareness: Check credit usage before large crawls, since crawling consumes credits per page. - Use Case: You need all pages of a product's documentation section for offline analysis. Run a crawl scoped with --include-paths /docs and a page limit, then read the saved output under .firecrawl/. ## Quick Start Ask the agent to crawl the documentation section of a site with a page limit and save the results to a local file.

Frequently Asked Questions about firecrawl-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl an entire website with Firecrawl?▼

Run firecrawl crawl with the target URL, optionally adding --max-depth, --limit, and --wait to control scope and block until completion. Results are saved with -o to a JSON file such as .firecrawl/crawl.json.

How do I crawl only a specific section of a site?▼

Use the --include-paths option to scope the crawl, for example --include-paths /docs to crawl only pages under the docs path. This limits credit consumption to the pages you actually need.

Does Firecrawl crawl require authentication?▼

Yes, the crawl command requires authentication and has no keyless free tier. Without credentials, the CLI prompts an interactive login before the crawl can start.

What is the difference between Firecrawl crawl and scrape?▼

Crawl follows links to extract content from many pages across a site, while scrape extracts content from a single page. Use firecrawl-map to discover URLs first if you are unsure which pages exist.

How do I check the status of a running Firecrawl crawl?▼

If you started a crawl without --wait, the CLI returns a job ID. Pass that job ID to firecrawl crawl to poll its status until it reaches a terminal state.

Why is my Firecrawl crawl consuming so many credits?▼

Crawling consumes credits per page, so unscoped crawls of large sites add up quickly. Check firecrawl credit-usage before large crawls and scope with --include-paths, --limit, or --max-depth.

Related Skills

Explore More Software Engineering Skills→
olostepolostep

map

Discover and list all URLs on a website with keyword and pattern filtering.

Community
Intermediate
BEIRUXBEIRUX

tavily-cli

Perform LLM-optimized web search and content extraction with JSON output.

Official
Intermediate
NVIDIANVIDIA
3.4k

monitor

Monitor submitted compute jobs and report their execution status changes.

Official
Advanced
CerisonAutomationCerisonAutomation

firecrawl

Scrape and crawl websites into clean Markdown for research.

Community
Intermediate