skills.rest
durandom
durandomCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

crawl4ai

Crawl web pages and extract Markdown or structured JSON data via crwl CLI.

5|3|Updated Jan 6, 2026
Search Tags:#markdown#CLI#SPA#web-scraping#structured-data#crawl
One-click install
npx skills add https://github.com/durandom/skills --skill crawl4ai-durandom
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: crawl4ai
Source: https://github.com/durandom/skills/tree/main/skills/crawl4ai
Command: npx skills add https://github.com/durandom/skills --skill crawl4ai-durandom

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Web crawling and data extraction can be tedious when content is hidden behind SPA rendering or scattered across sites. This skill uses the crwl CLI to retrieve web pages and produce clean Markdown, extract structured data, and batch process multiple URLs.

Core Features & Use Cases

  • Web content extraction: convert pages to Markdown, including SPA-rendered content.
  • Structured data: extract product names, prices, links using JSON schema.
  • Batch crawling: process lists of URLs with output per URL.

Quick Start

Use the crwl CLI to crawl a single site and save Markdown output. crwl crawl -o md https://example.com Save into a file: crwl crawl -o md -O output.md https://example.com Handle SPA sites with a pre-made config: crwl crawl -C configs/spa.yaml -bc -o md "https://spa-site.com"

Frequently Asked Questions about crawl4ai

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract markdown from a website that uses JavaScript rendering?▼

Web scraping JavaScript-heavy SPAs requires a browser-configured crawler to render content before extraction. Using the crwl CLI with a pre-made SPA config and browser flag retrieves dynamically loaded pages and converts them to clean Markdown output.

Can I extract structured data like product prices and links from multiple URLs?▼

Structured data extraction from multiple URLs is supported through batch crawling with JSON schema definitions. The crwl CLI processes lists of URLs and outputs structured data such as product names, prices, and links per URL.

What is the best way to batch crawl a list of URLs and save the output?▼

Batch crawling multiple URLs is handled by the crwl CLI, which processes lists of links and generates separate output files for each URL. You can save the extracted Markdown or structured JSON data directly to specified files.

Does web scraping with this approach work on static sites without JavaScript?▼

Static site crawling works natively with the crwl CLI without needing browser configurations. You can retrieve web pages and produce clean Markdown output by running a simple crawl command targeting the static URL.

Why does my SPA web scraping return empty or incomplete content?▼

SPA web scraping often returns empty content when JavaScript rendering is not triggered before extraction. Applying a pre-made crawler config and enabling the browser flag in the crwl CLI ensures dynamic page elements load fully before content is retrieved.

Related Skills

Explore More Data & Analytics Skills→
ivanrvpereiraivanrvpereira
17

crwl

Extract readable web page content and structured JSON data from URLs via the crwl CLI.

Community
Intermediate
profmikegreeneprofmikegreene
1

plan-ingestion

Parses pilot plans and hypotheses into structured JSON or tabular data for downstream analysis.

Community
Intermediate
zocomputerzocomputer
45

web-scraper

Crawl websites and extract structured data using CSS selectors, JavaScript rendering, and LLM extraction.

Official
Advanced
chestnut0698chestnut0698

web-scraping

Extract data from static and dynamic web pages using Python tools.

Community
Advanced