crawl4ai

Crawl websites and extract structured data using CSS, XPath, and LLM-based strategies.

3|1|Updated Feb 2, 2026

Search Tags:#data extraction #LLM #RAG #AI #web scraping #markdown generation #crawling

Ready to run

npx skills add https://github.com/HouseGarofalo/claude-code-base --skill crawl4ai-housegarofalo

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl4ai
Source: https://github.com/HouseGarofalo/claude-code-base/tree/main/.claude/skills/crawl4ai
Command: npx skills add https://github.com/HouseGarofalo/claude-code-base --skill crawl4ai-housegarofalo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates the process of crawling websites and extracting valuable information, transforming raw web content into structured, AI-ready formats.

Core Features & Use Cases

Intelligent Web Crawling: Navigates websites, follows links, and extracts content using AI-driven adaptive crawling.
Data Extraction: Extracts data using CSS selectors, XPath, or advanced LLM-based strategies, outputting clean markdown, HTML, or structured JSON.
RAG Integration: Built to create knowledge bases from web content for retrieval-augmented generation pipelines.
Use Case: Automatically gather all product details from an e-commerce site, extract key information from technical documentation, or build a comprehensive knowledge base from a set of research papers online.

Quick Start

Use the crawl4ai skill to crawl the URL 'https://example.com' and print the extracted markdown content.

Frequently Asked Questions about crawl4ai

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract web data into markdown for RAG pipelines?▼

You can extract web data into markdown for RAG pipelines by using AI-powered crawling to navigate websites and output clean, structured markdown formatted specifically for retrieval-augmented generation. It supports adaptive crawling to gather content efficiently.

What is the best way to scrape structured data from multiple URLs?▼

The best way to scrape structured data from multiple URLs is using LLM-based schema extraction, which processes multiple URLs simultaneously and outputs structured JSON. This automates navigating sites and extracting key information without manual parsing rules.

Can I use CSS selectors and XPath for web scraping alongside AI extraction?▼

Yes, you can use CSS selectors and XPath for web scraping alongside AI extraction techniques. The tool supports traditional CSS and XPath extraction methods as well as advanced LLM-based strategies to handle complex data structures.

How does AI-powered crawling handle dynamic website content?▼

AI-powered crawling handles dynamic website content by using adaptive crawling techniques that intelligently navigate websites and follow links. This ensures accurate data extraction even when content structures change or require interaction to load.

What output formats are supported when extracting web content?▼

Supported output formats when extracting web content include clean markdown, raw HTML, and structured JSON. These formats allow direct integration into data analysis workflows or AI applications requiring structured information.

Do I need an LLM to extract data from technical documentation?▼

You do not strictly need an LLM to extract data from technical documentation, as CSS and XPath selectors are supported. However, LLM-based extraction provides advanced schema generation for complex or unstructured technical documents.