skills.rest
DiegoHeer
DiegoHeerCOMMUNITY
Explore all skills by this author →
GitHub

web-scraping

Extract content from anti-bot websites using Trafilatura, Requests, and Playwright fallbacks.

3|Updated Jun 18, 2025
Search Tags:#playwright#trafilatura#requests#web-scraping#data-cleaning#content-extraction#anti-bot
One-click install
npx skills add https://github.com/DiegoHeer/realty-alerts --skill web-scraping-diegoheer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/DiegoHeer/realty-alerts/tree/main/.claude/skills/web-scraping
Command: npx skills add https://github.com/DiegoHeer/realty-alerts --skill web-scraping-diegoheer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Extracting content from websites that deploy anti-bot defenses, dynamic rendering, or paywalls is time-consuming and fragile; this skill provides resilient, multi-method scraping to obtain usable text and metadata.

Core Features & Use Cases

  • Multi-backend scraping: Trafilatura, Requests, and Playwright with stealth to maximize success across diverse sites.
  • Content extraction and metadata: capture article text and titles, clean noise, and preserve structure for downstream processing.
  • Use Case: gather product pages or news articles from bot-protected sites and compile the main content into a unified dataset.

Quick Start

Fetch the main content from a target URL using the best available backend, then save the extracted text and title for further processing.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape content from websites with anti-bot defenses?▼

Scraping content from websites with anti-bot defenses is handled using a cascading scraper that tries Trafilatura, Requests, and Playwright in order, with automatic fallbacks to maximize extraction success.

Can I extract text from dynamically rendered or paywalled pages?▼

Extracting text from dynamically rendered or paywalled pages is supported by using Playwright with stealth, which acts as a fallback backend to capture content that simpler request methods fail to load.

What is the best way to scrape product listings and news articles?▼

The best way to scrape product listings and news articles is using a multi-backend approach that captures article text and titles, cleans noise, and preserves structure for downstream processing into a unified dataset.

Does this web scraping method clean noise and preserve structure?▼

This web scraping method cleans noise and preserves structure by capturing article text and titles through Trafilatura, ensuring the extracted content is usable for downstream processing.

Why does my web scraper fail on bot-protected sites?▼

Web scrapers fail on bot-protected sites because simple requests get blocked; this tool solves that by implementing a cascading fallback system across Trafilatura, Requests, and Playwright to ensure resilient extraction.

Are there ethical scraping guidelines included with content extraction?▼

Ethical scraping guidance is included with the content extraction process, ensuring that gathering data from bot-protected sites, social posts, and product pages is done responsibly.

Related Skills

Explore More Software Engineering Skills→
olostepolostep

map

Discover and list all URLs on a website with keyword and pattern filtering.

Community
Intermediate
alibabaalibaba
10.5k

pdf-extractor

Extract text, tables, and metadata from PDF documents into JSON.

Official
Intermediate
moonmingmoonming
1

a6-plugin-limit-req

Configure Apache APISIX limit-req plugin for leaky bucket rate limiting.

Community
Intermediate
antgroupantgroup
16

web-pilot

Automate web searches and extract page content via Playwright and Chromium.

Official
Advanced