skills.rest
PlaneInABottle
PlaneInABottleCOMMUNITY
Explore all skills by this author →
GitHub

crawl4ai

Crawl websites into clean Markdown or JSON via CLI, Python SDK, or Docker API.

1|Updated Aug 26, 2025
Search Tags:#automation#batch-processing#web-crawling#crawl4ai#python-sdk#docker-api#llm-extraction
One-click install
npx skills add https://github.com/PlaneInABottle/configs --skill crawl4ai-planeinabottle
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: crawl4ai
Source: https://github.com/PlaneInABottle/configs/tree/main/agents/.agents/skills/crawl4ai
Command: npx skills add https://github.com/PlaneInABottle/configs --skill crawl4ai-planeinabottle

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires packaging, crawl4ai, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Crawl4AI provides an open-source, LLM-friendly web crawler and scraper that turns websites into clean Markdown or JSON, enabling automation of data extraction, batch URL processing, and RAG pipelines.

Core Features & Use Cases

  • Supports CLI, Python SDK, and Docker API for versatile access
  • Deep crawling strategies (BFS, DFS, Best-First) with CSS/LLM-based extraction and Markdown/JSON outputs
  • Ready-to-use scripts and SDK references for batch processing, data extraction, and RAG pipelines

Quick Start

Initialize AsyncWebCrawler with a BrowserConfig and a CrawlerRunConfig, then call arun on a target URL to perform a basic crawl.

Frequently Asked Questions about crawl4ai

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate web crawling and data extraction for batch URL processing?▼

Web crawling automation for batch URL processing is achieved through a scalable code-driven interface supporting CLI, Python SDK, and Docker API. It orchestrates browser sessions, configurable timeouts, and robust error handling for production-grade workflows.

What is the best way to extract clean Markdown or JSON from websites for RAG pipelines?▼

Extracting clean Markdown or JSON from websites for RAG pipelines is done using an open-source, LLM-friendly web crawler. It supports deep crawling strategies with CSS and LLM-based extraction to turn diverse sites into structured data.

How do I perform deep crawling using BFS or DFS strategies with LLM-augmented extraction?▼

Deep crawling using BFS, DFS, or Best-First strategies with LLM-augmented extraction is supported natively. It allows configurable content filtering and extraction strategies across diverse sites to retrieve targeted data.

Does this web scraper support Docker API integration for automated data extraction?▼

Docker API integration for automated data extraction is fully supported alongside CLI and Python SDK access. This versatile interface enables scalable batch processing and browser session orchestration within containerized environments.

How do I initialize an AsyncWebCrawler to perform a basic crawl on a target URL?▼

To initialize an AsyncWebCrawler for a basic crawl, configure a BrowserConfig and a CrawlerRunConfig, then call arun on the target URL. This automates the extraction and processing workflow.

Related Skills

Explore More Software Engineering Skills→
TheBushidoCollectiveTheBushidoCollective
187

sentry-sdk-configuration

Configure Sentry SDK initialization and error tracking across frameworks.

Official
Intermediate
jackspacejackspace
16

markitdown

Convert PDFs, DOCX, PPTX, XLSX, HTML, EPUB, CSV, JSON, and images into Markdown.

Community
Intermediate
roach88roach88
2

scraping-websites

Crawl static and JavaScript-rendered websites and extract structured data.

Community
Intermediate
LuisElPotiLuisElPoti

tavily-cli

Perform LLM-optimized web search, URL mapping, and content extraction with JSON/file output.

Community
Intermediate