skills.rest
ironkid90
ironkid90COMMUNITY
Explore all skills by this author →
GitHub

firecrawl-crawl

Crawl websites with depth limits and path filtering to extract content.

Updated Jun 15, 2026
Search Tags:#web scraping#website crawling#bulk scraping#site content extraction#concurrent extraction
One-click install
npx skills add https://github.com/ironkid90/lucky5-v8 --skill firecrawl-crawl-ironkid90
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: firecrawl-crawl
Source: https://github.com/ironkid90/lucky5-v8/tree/main/plugins/skills/firecrawl-crawl
Command: npx skills add https://github.com/ironkid90/lucky5-v8 --skill firecrawl-crawl-ironkid90

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires firecrawl, and includes scripts (resource) components.

What problem does it solve?

This Skill addresses the need to efficiently extract content from an entire website or specific sections, as well as bulk-scrape multiple pages following links.

Core Features & Use Cases

  • Bulk Content Extraction: Extracts all pages from a website or specific section.
  • Concurrent Extraction: Handles depth limits, path filtering, and concurrent extraction for optimal performance.
  • Use Case: Ideal for gathering content from numerous pages on the same site, such as all content under the /docs section of a website.

Quick Start

Use the firecrawl-crawl skill to crawl a section of a website. Example: firecrawl-crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

Frequently Asked Questions about firecrawl-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract all content from a specific section of a website?▼

Website content extraction for specific sections is done by specifying path filters like `--include-paths /docs` to limit the crawl to targeted site areas and output the results to a file.

What is concurrent website crawling and how does it handle large sites?▼

Concurrent website crawling simultaneously extracts multiple pages to optimize performance, using depth limits and path filtering to manage large-scale content retrieval efficiently.

Can I limit the depth of bulk web scraping to avoid crawling an entire domain?▼

Yes, bulk web scraping supports depth limits and path filtering, allowing you to restrict how deep the crawler traverses links and confine extraction to specific site sections.

Do I need Firecrawl installed to perform concurrent site content extraction?▼

Yes, Firecrawl is a required dependency for executing these bash scripts, providing the underlying engine for concurrent site content extraction and link-following bulk scraping operations.

What is the best way to bulk scrape multiple pages following links automatically?▼

The best way to bulk scrape pages is by running a concurrent crawl command with a defined page limit and wait option, automatically following links to gather all targeted site content into a JSON file.

Related Skills

Explore More Software Engineering Skills→
breverdbidderbreverdbidder
1

firecrawl-map

Discover and list all URLs on a website with filtering and sitemap options.

Community
Basic
abdullah1854abdullah1854
1

crawl-cli

Crawl websites and extract content into Markdown, JSON, or text.

Community
Basic
1thirteeng31thirteeng3

firecrawl-crawl

Crawl websites to bulk-extract pages and content with configurable depth and path filters.

Community
Intermediate
brandon-frysliebrandon-fryslie
4

firecrawl-crawl

Crawl websites with depth limits, path filtering, and concurrent extraction.

Community
Intermediate