skills.rest
lodekeeper
lodekeeperCOMMUNITY
Explore all skills by this author →
GitHub

web-scraping

Scrape web content with tiered HTTP and headless browser automation.

4|1|Updated Feb 20, 2026
Search Tags:#automation#data extraction#web scraping#cloudflare#anti-bot#spa rendering
One-click install
npx skills add https://github.com/lodekeeper/dotfiles --skill web-scraping-lodekeeper
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/lodekeeper/dotfiles/tree/main/skills/web-scraping
Command: npx skills add https://github.com/lodekeeper/dotfiles --skill web-scraping-lodekeeper

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl_cffi, scrapling, trafilatura, camoufox, rebrowser-playwright, playwright-stealth, nodriver, extruct, w3lib, readability-lxml, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a robust solution for extracting web content when standard methods fail, overcoming challenges like JavaScript rendering, Cloudflare protection, and login requirements.

Core Features & Use Cases

  • Tiered Scraping: Automatically escalates through multiple tools (curl_cffi, DynamicFetcher, Camoufox) based on site complexity.
  • Anti-Bot Bypass: Designed to handle Cloudflare and other common anti-scraping measures.
  • Structured Data Extraction: Utilizes libraries like Trafilatura for clean text extraction and can handle JSON-LD, Microdata, and OpenGraph.
  • Use Case: Extracting product details from an e-commerce site that uses dynamic JavaScript rendering and Cloudflare protection.

Quick Start

Use the web-scraping skill to scrape the content from https://example.com.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a website that uses Cloudflare protection?▼

To scrape a website with Cloudflare protection, you can use a tiered approach that escalates from simple HTTP requests to headless browser automation tools like Camoufox to bypass anti-bot measures.

What's the best way to extract text content from a JavaScript-rendered SPA?▼

The best way to extract text from a JavaScript-rendered SPA is using headless browser automation that renders the page, followed by libraries like Trafilatura to extract clean text content.

Can I extract structured data like JSON-LD and OpenGraph from web pages?▼

Yes, you can extract structured data formats like JSON-LD, Microdata, and OpenGraph from web pages using specialized extraction libraries that parse HTML markup.

How does a tiered web scraping approach handle different site complexities?▼

A tiered web scraping approach automatically escalates through multiple tools, starting with basic HTTP requests and advancing to headless browsers based on the target site's complexity and anti-bot defenses.

Does this web scraping method support extracting clean main text content?▼

Yes, this web scraping method supports extracting clean main text content by utilizing Trafilatura, which removes boilerplate and navigation elements to isolate the primary readable text.

When should I escalate from simple HTTP requests to headless browser automation?▼

You should escalate from simple HTTP requests to headless browser automation when a website requires JavaScript rendering for content display or employs anti-scraping measures like Cloudflare protection.

Related Skills

Explore More Software Engineering Skills→
Chris-Chai-MinjaeChris-Chai-Minjae

scrapling

Automate web scraping with HTTP fetching, stealth browser automation, and spider crawling.

Community
Advanced
Difficult-BurgerDifficult-Burger
8

nlp-dataset-analyzer

Analyze NLP datasets in JSON, JSONL, or CSV and generate a Markdown quality report.

Community
Advanced
acaprinoacaprino
6

grabber-development

Build production-grade Python web scrapers with stealth and anti-bot defenses.

Community
Advanced
ggombeeggombee
13

crawler

Generate Playwright web crawler code templates for scraping tasks.

Community
Intermediate