skills.rest
r1cA18
r1cA18COMMUNITY
Explore all skills by this author →
GitHub

firecrawl-crawl

Crawl websites by following links to extract page content with filters and limits.

4|Updated Nov 22, 2025
Search Tags:#automation#web scraping#content extraction#bulk data#crawling#site scraping
One-click install
npx skills add https://github.com/r1cA18/dotfiles --skill firecrawl-crawl
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: firecrawl-crawl
Source: https://github.com/r1cA18/dotfiles/tree/main/skills/firecrawl-crawl
Command: npx skills add https://github.com/r1cA18/dotfiles --skill firecrawl-crawl

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the process of extracting content from multiple pages of a website, saving significant time and effort compared to manual scraping.

Core Features & Use Cases

  • Bulk Content Extraction: Crawl and extract content from an entire website or specific sections.
  • Link Following: Automatically follows links to discover and extract content from related pages.
  • Filtering: Allows specifying paths to include or exclude, enabling targeted crawls.
  • Use Case: Extract all product descriptions from an e-commerce site's /products section, or gather all articles from a blog's /archive directory.

Quick Start

Use the firecrawl crawl skill to extract all pages under the /docs path from https://example.com/docs with a limit of 50 pages, saving the output to .firecrawl/crawl.json.

Frequently Asked Questions about firecrawl-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I bulk extract content from an entire website automatically?▼

Bulk content extraction is done by crawling a website and following links to discover and extract related pages up to a defined depth or page limit. This automates scraping to save significant time compared to manual effort.

Can I filter web scraping to only target specific sections of a site?▼

Web scraping can be filtered by specifying paths to include or exclude for targeted crawls. This allows you to extract content from specific sections like an e-commerce site's products area or a blog's archive directory.

Does site crawling support concurrent extraction and rate limiting?▼

Site crawling supports concurrent extraction and rate limiting to ensure efficient bulk data acquisition from web resources. These features manage request frequency and parallel processing during the crawl.

What is the best way to gather all articles from a blog's archive directory?▼

The best way to gather articles from a blog's archive is by crawling the specific directory path and following internal links. You can set a page limit to control the extraction scope and save the output to a JSON file.

How does link following work during website content extraction?▼

Link following during website content extraction works by automatically discovering and traversing links on visited pages. This enables the crawler to navigate related pages and extract content up to the defined crawl depth.

Related Skills

Explore More Software Engineering Skills→
s-hiraokus-hiraoku
21

mcp-firecrawl

Scrape websites, crawl pages, search, and extract structured data via Firecrawl tools.

Community
Basic
AlexKapadiaAlexKapadia
6

export-as-format

Convert a saved Pixie artefact to a user-chosen format via the server-side exporter.

Community
Advanced
LeandroBenjaminLLeandroBenjaminL

etl-pipelines

Automate ETL data pipelines using Pandas for batch processing.

Community
Intermediate
iAziz786iAziz786
3

firecrawl-crawl

Crawl websites with depth limits and path filters via self-hosted Firecrawl.

Community
Intermediate