crawl4ai

Crawl web pages and extract Markdown or structured JSON data via crwl CLI.

5|3|Updated Jan 6, 2026

Search Tags:#markdown #CLI #SPA #web-scraping #structured-data #crawl

Ready to run

npx skills add https://github.com/durandom/skills --skill crawl4ai-durandom

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawl4ai
Source: https://github.com/durandom/skills/tree/main/skills/crawl4ai
Command: npx skills add https://github.com/durandom/skills --skill crawl4ai-durandom

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Web crawling and data extraction can be tedious when content is hidden behind SPA rendering or scattered across sites. This skill uses the crwl CLI to retrieve web pages and produce clean Markdown, extract structured data, and batch process multiple URLs.

Core Features & Use Cases

Web content extraction: convert pages to Markdown, including SPA-rendered content.
Structured data: extract product names, prices, links using JSON schema.
Batch crawling: process lists of URLs with output per URL.

Quick Start

Use the crwl CLI to crawl a single site and save Markdown output. crwl crawl -o md https://example.com Save into a file: crwl crawl -o md -O output.md https://example.com Handle SPA sites with a pre-made config: crwl crawl -C configs/spa.yaml -bc -o md "https://spa-site.com"

Frequently Asked Questions about crawl4ai

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract markdown from a website that uses JavaScript rendering?▼

Web scraping JavaScript-heavy SPAs requires a browser-configured crawler to render content before extraction. Using the crwl CLI with a pre-made SPA config and browser flag retrieves dynamically loaded pages and converts them to clean Markdown output.

Can I extract structured data like product prices and links from multiple URLs?▼

Structured data extraction from multiple URLs is supported through batch crawling with JSON schema definitions. The crwl CLI processes lists of URLs and outputs structured data such as product names, prices, and links per URL.

What is the best way to batch crawl a list of URLs and save the output?▼

Batch crawling multiple URLs is handled by the crwl CLI, which processes lists of links and generates separate output files for each URL. You can save the extracted Markdown or structured JSON data directly to specified files.

Does web scraping with this approach work on static sites without JavaScript?▼

Static site crawling works natively with the crwl CLI without needing browser configurations. You can retrieve web pages and produce clean Markdown output by running a simple crawl command targeting the static URL.

Why does my SPA web scraping return empty or incomplete content?▼

SPA web scraping often returns empty content when JavaScript rendering is not triggered before extraction. Applying a pre-made crawler config and enabling the browser flag in the crwl CLI ensures dynamic page elements load fully before content is retrieved.