crawl4ai

Crawl URLs and convert webpages to LLM-friendly Markdown.

Updated Dec 18, 2025

Search Tags:#automation #markdown #llm #playwright #web-crawling #web-content #crawl4ai

Ready to run

npx skills add https://github.com/alexjx/skills --skill crawl4ai-alexjx

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl4ai
Source: https://github.com/alexjx/skills/tree/main/skills/crawl4ai
Command: npx skills add https://github.com/alexjx/skills --skill crawl4ai-alexjx

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a robust web crawling pipeline that converts web pages to clean, LLM-friendly markdown for AI processing, enabling reliable extraction and downstream analysis.

Core Features & Use Cases

Isolated Python environment: Uses uv to manage execution and dependencies, ensuring clean, reproducible runs.
URL-to-markdown conversion: Crawls pages and outputs markdown suitable for ingestion into LLM workflows.
Rich outputs: Generates markdown, HTML, screenshots, PDFs, and MHTML archives for downstream tasks.

Quick Start

From the skill directory, run uv run python scripts/crawl.py "https://example.com" to crawl a page and generate markdown.

Frequently Asked Questions about crawl4ai

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I convert a webpage to markdown for AI processing?▼

Web crawling with Crawl4AI converts URLs to clean, LLM-friendly markdown using AsyncWebCrawler and DefaultMarkdownGenerator. Run `uv run python scripts/crawl.py "https://example.com"` from the skill directory to crawl a page and generate markdown output suitable for AI workflows.

Can I scrape multiple URLs at once and extract content for LLM ingestion?▼

Yes, Crawl4AI accepts a list of URLs and produces markdown for each, enabling batch web content extraction. The crawler outputs raw_markdown that's formatted specifically for language model processing and downstream analysis.

What do I need to set up before crawling websites with this tool?▼

You need uv installed to manage an isolated Python environment via `uv sync`. Crawl4AI uses BrowserConfig and CrawlerRunConfig to handle dependencies and execution, ensuring reproducible runs without manual dependency management.

Does this web crawler support headless browsing and customizable timeouts?▼

Yes, Crawl4AI supports `--no-headless` mode for visible browser execution and `--timeout` to set request limits. Additional options include `--include-links` and `--include-images` to control markdown output richness.

What output formats does web crawling produce besides markdown?▼

Beyond markdown, Crawl4AI generates HTML, screenshots, PDFs, and MHTML archives from crawled pages. These rich outputs enable flexible downstream processing and storage options for extracted web content.

How does Crawl4AI differ from other web scraping tools for AI workflows?▼

Crawl4AI specifically optimizes for LLM-ready output with clean markdown generation and AI-focused configuration options. Its integrated browser automation, isolated execution environment, and purpose-built markdown formatting distinguish it from general-purpose scrapers.