brettdaviesbrettdaviesCommunityยท1 Agent Skills Included

crawl4ai-skill

Scrape JavaScript-heavy sites and extract structured web data

Scrapes JavaScript-heavy pages and SPAs with a headless browser, then converts them into clean markdown or structured JSON. Eliminates brittle copy-paste scraping, per-request LLM extraction costs, and manual handling of logins, proxies, and bot detection. Provides ready-made scripts, templates, and recipes for batch crawling, reusable CSS schemas, and authenticated sessions.
npx skills add brettdavies/crawl4ai-skill --all -g -y

All Skills in This Repository (1)

Pure Emerald Level Indicators

Frequently Asked Questions

FAQPage Schema
How to install crawl4ai-skill?โ–ผ

Run `npx skills add brettdavies/crawl4ai-skill --all -g -y` in your terminal to install the skill globally. You also need the Crawl4AI Python library installed via `pip install crawl4ai` followed by `crawl4ai-setup`.

How to scrape JavaScript-heavy websites?โ–ผ

This skill renders pages in a headless browser with network-idle waiting, so content loaded by JavaScript is fully captured. A single command like `crwl <url> -o markdown` returns clean markdown.

How to extract structured data without paying LLM costs?โ–ผ

Generate a reusable CSS schema once with the bundled script, then apply it to any number of pages for fast, deterministic extraction with zero per-request LLM cost.

Does crawl4ai-skill work with Claude Code and Cursor?โ–ผ

Yes. It follows the universal SKILL.md standard and loads in Claude Code, Codex, Cursor, OpenCode, Cline, and other compatible hosts.

Can it crawl many URLs or handle logins?โ–ผ

Yes. Batch scripts crawl multiple URLs concurrently, and session templates support authenticated, multi-step flows with proxies and anti-detection options.

Related Repositories in Software Engineering

View All in Software Engineeringโ†’