web-scraping

Name: web-scraping
Availability: InStock
Author: Lodekeeper

Scrape web content with tiered HTTP and headless browser automation.

4|1|Updated Feb 20, 2026

Search Tags:#automation #data extraction #web scraping #cloudflare #anti-bot #spa rendering

Ready to run

npx skills add https://github.com/lodekeeper/dotfiles --skill web-scraping-lodekeeper

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/lodekeeper/dotfiles/tree/main/skills/web-scraping
Command: npx skills add https://github.com/lodekeeper/dotfiles --skill web-scraping-lodekeeper

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl_cffi, scrapling, trafilatura, camoufox, rebrowser-playwright, playwright-stealth, nodriver, extruct, w3lib, readability-lxml, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a robust solution for extracting web content when standard methods fail, overcoming challenges like JavaScript rendering, Cloudflare protection, and login requirements.

Core Features & Use Cases

Tiered Scraping: Automatically escalates through multiple tools (curl_cffi, DynamicFetcher, Camoufox) based on site complexity.
Anti-Bot Bypass: Designed to handle Cloudflare and other common anti-scraping measures.
Structured Data Extraction: Utilizes libraries like Trafilatura for clean text extraction and can handle JSON-LD, Microdata, and OpenGraph.
Use Case: Extracting product details from an e-commerce site that uses dynamic JavaScript rendering and Cloudflare protection.

Quick Start

Use the web-scraping skill to scrape the content from https://example.com.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scrape a website that uses Cloudflare protection?▼

To scrape a website with Cloudflare protection, you can use a tiered approach that escalates from simple HTTP requests to headless browser automation tools like Camoufox to bypass anti-bot measures.

What's the best way to extract text content from a JavaScript-rendered SPA?▼

The best way to extract text from a JavaScript-rendered SPA is using headless browser automation that renders the page, followed by libraries like Trafilatura to extract clean text content.

Can I extract structured data like JSON-LD and OpenGraph from web pages?▼

Yes, you can extract structured data formats like JSON-LD, Microdata, and OpenGraph from web pages using specialized extraction libraries that parse HTML markup.

How does a tiered web scraping approach handle different site complexities?▼

A tiered web scraping approach automatically escalates through multiple tools, starting with basic HTTP requests and advancing to headless browsers based on the target site's complexity and anti-bot defenses.

Does this web scraping method support extracting clean main text content?▼

Yes, this web scraping method supports extracting clean main text content by utilizing Trafilatura, which removes boilerplate and navigation elements to isolate the primary readable text.

When should I escalate from simple HTTP requests to headless browser automation?▼

You should escalate from simple HTTP requests to headless browser automation when a website requires JavaScript rendering for content display or employs anti-scraping measures like Cloudflare protection.