skills.rest
iAziz786
iAziz786COMMUNITY
Explore all skills by this author →
Website𝕏@iAziz786GitHub

firecrawl-crawl

Crawl websites with depth limits and path filters via self-hosted Firecrawl.

3|Updated Jul 5, 2021
Search Tags:#firecrawl#concurrency#web-scraping#crawl#depth-control#site-mapping#bulk-extract
One-click install
npx skills add https://github.com/iAziz786/dotfiles --skill firecrawl-crawl-iaziz786
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: firecrawl-crawl
Source: https://github.com/iAziz786/dotfiles/tree/main/.agents/skills/firecrawl-crawl
Command: npx skills add https://github.com/iAziz786/dotfiles --skill firecrawl-crawl-iaziz786

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Bulk-extract content from entire websites or site sections using a self-hosted Firecrawl instance, enabling rapid collection of pages for analysis.

Core Features & Use Cases

  • Crawl all pages under a site or a docs subsection with depth and concurrency controls.
  • Apply include/exclude path filters to constrain the crawl to relevant sections.
  • Discover, map, and export page data for downstream processing.

Quick Start

Start a crawl on a target URL by running a command that specifies the base URL and optional include paths and a depth limit to begin extraction.

Frequently Asked Questions about firecrawl-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I bulk crawl an entire website or specific documentation section?▼

To bulk crawl a website, you specify a target base URL and configure optional include path filters with a depth limit. This initiates concurrent extraction across all matching pages to collect content rapidly.

Can I limit web scraping depth and filter paths to avoid crawling irrelevant site sections?▼

Yes, you can apply include and exclude path filters to constrain web scraping to relevant sections. You also enforce depth limits to control how deep the crawler traverses links from the base URL.

What's the best way to extract content from multiple pages concurrently for downstream analysis?▼

The best way to extract content concurrently is using a self-hosted Firecrawl instance with built-in concurrency controls. This enables scalable, repeatable bulk extraction of page data for downstream processing.

Do I need a self-hosted Firecrawl instance to map and export site pages?▼

Yes, a self-hosted Firecrawl instance is required to map and export site pages. The Skill uses this self-hosted deployment to execute depth-controlled, concurrent crawls and collect page data.

How does site mapping work when extracting pages under a specific docs path?▼

Site mapping works by discovering all reachable pages under a specified docs path and extracting their content. It applies path filtering and depth control to ensure only relevant pages are collected.

Can I scrape multiple pages from a single site without getting blocked or exceeding limits?▼

You can scrape multiple pages from a single site by configuring concurrency and enforcing depth limits. These controls ensure scalable, repeatable crawls while keeping extraction within manageable bounds.

Related Skills

Explore More Data & Analytics Skills→
iAziz786iAziz786
3

firecrawl

Searches, scrapes, and interacts with web pages via a local Firecrawl API.

Community
Advanced
shuanbao0shuanbao0
14

document-pro

Extract text, tables, and metadata from PDF, Word, PowerPoint, Excel, TXT, and Markdown documents.

Community
Advanced
sitharaj88sitharaj88
1

cf-ai

Deploy and manage AI workloads on Cloudflare's edge network.

Community
Advanced
firecrawlfirecrawl
72

firecrawl-build-scrape

Extract web page content as markdown, HTML, or structured data via Firecrawl's /scrape API.

Official
Intermediate