skills.rest
mbastakis
mbastakisCOMMUNITY
Explore all skills by this author →
GitHub

crawl4ai

Crawl web pages with Python and extract structured data as Markdown or JSON.

5|1|Updated Apr 22, 2025
Search Tags:#automation#markdown#JSON#Python#web#data-extraction#crawl
One-click install
npx skills add https://github.com/mbastakis/dotfiles --skill crawl4ai-mbastakis
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: crawl4ai
Source: https://github.com/mbastakis/dotfiles/tree/main/dot-config/opencode/skill/crawl4ai
Command: npx skills add https://github.com/mbastakis/dotfiles --skill crawl4ai-mbastakis

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires crawl4ai, packaging, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Crawl4AI provides a scalable toolkit for automated web crawling, data extraction, and content processing, enabling teams to quickly gather structured information from diverse sites with minimal manual effort.

Core Features & Use Cases

  • Deterministic single- and multi-URL crawls, including JavaScript-heavy pages, with customizable wait and navigation controls.
  • Rich Markdown generation and structured data extraction via CSS-based schemas or LLM-based strategies.
  • Batch processing, session management, site-wide crawling, and plugin-like extraction workflows for knowledge bases and documentation pipelines.

Quick Start

Install Crawl4AI and its Python package, then create a BrowserConfig and CrawlerRunConfig, and use AsyncWebCrawler to perform a crawl with arun(). Example:

  • Install: pip install crawl4ai
  • Basic usage: from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
  • Then run a simple crawl with crawler arun(url, config)

Frequently Asked Questions about crawl4ai

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured data from a JavaScript-rendered website?▼

To extract structured data from JavaScript-rendered pages, you can use a Python web crawler with configurable wait and navigation controls. It handles dynamic content loading and outputs clean Markdown or JSON for your data pipelines.

What is the best way to convert web pages to Markdown for a knowledge base?▼

The best way to convert web pages to Markdown for a knowledge base is using an automated crawling toolkit that supports rich Markdown generation. You can configure extraction strategies to pull deterministic content from single or multiple URLs repeatably.

Can I use Python to batch crawl multiple URLs and output JSON?▼

Yes, you can use Python to batch crawl multiple URLs and output JSON. The toolkit supports multi-URL batch processing, session management, and structured data extraction via CSS-based schemas or LLM-based strategies to gather JSON efficiently.

Does automated web crawling support configurable browser and run settings?▼

Automated web crawling supports configurable browser and run settings through dedicated configuration objects. You can define specific browser environments and crawler execution parameters to ensure repeatable data collection workflows across different sites.

How do you handle site-wide crawling for documentation pipelines?▼

Site-wide crawling for documentation pipelines is handled using batch processing and plugin-like extraction workflows. This allows you to programmatically gather and process content across an entire site, outputting structured data suitable for knowledge bases.

Related Skills

Explore More Software Engineering Skills→
alexjxalexjx

crawl4ai

Crawl URLs and convert webpages to LLM-friendly Markdown.

Community
Advanced
CodingHeaderCodingHeader

web-scrape

Extract web page content into markdown, JSON, or text outputs.

Community
Intermediate
basher83basher83
5

crawl4ai

Crawl JavaScript-heavy websites and extract structured data into Markdown.

Community
Intermediate
olostepolostep

batch

Scrape up to 10,000 URLs in parallel with configurable output formats.

Community
Intermediate