skills.rest
ValorInvestigator
ValorInvestigatorCOMMUNITY
Explore all skills by this author →
GitHub

web-scraping

Extract web content using cascading requests and Playwright stealth scraping.

1|Updated Feb 23, 2026
Search Tags:#yt-dlp#playwright#trafilatura#web-scraping#anti-bot#instaloader#poison-pill-detection
One-click install
npx skills add https://github.com/ValorInvestigator/claude-plugin-toolkit --skill web-scraping-valorinvestigator
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/ValorInvestigator/claude-plugin-toolkit/tree/main/skills/web-scraping
Command: npx skills add https://github.com/ValorInvestigator/claude-plugin-toolkit --skill web-scraping-valorinvestigator

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill orchestrates a cascade of scraping techniques so you can reliably harvest public web content even when facing anti-bot defenses, paywalls, or undocumented APIs that block straightforward HTTP requests.

Core Features & Use Cases

  • Cascading scrapers: Start with lightweight Trafilatura or requests extraction and automatically fall back to Playwright stealth browsing when initial downloads fail.
  • Anti-bot and paywall awareness: Rotate headers, detect blocking patterns, and classify poison pills to avoid wasted effort on blocked domains.
  • Social media and API harvesting: Use yt-dlp, instaloader, and reverse-engineering tips for YouTube, TikTok, Instagram, and hidden API endpoints that power downstream research.

Quick Start

Use the web-scraping skill to fetch article text from a URL while detecting any paywalls or anti-bot signals.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape web content when anti-bot blocks or paywalls stop standard HTTP requests?▼

Scrape web content past anti-bot blocks by cascading from lightweight Trafilatura requests to Playwright stealth rendering, automatically rotating headers and detecting poison pills to bypass paywalls and ensure reliable extraction.

Can I extract metadata and monitor feeds from social media platforms like Instagram or TikTok?▼

Extract social media metadata and monitor feeds using integrated instaloader and yt-dlp components, which target platform-specific endpoints to download content and uncover undocumented APIs for research workflows.

What is the best way to handle poison-pill detection during web scraping?▼

Handle poison-pill detection during web scraping by classifying blocking patterns and rotating headers to identify blocked domains early, avoiding wasted extraction effort and ensuring ethical harvesting of public data.

How do I switch from lightweight extraction to stealth browser rendering when a scrape fails?▼

Switch to stealth browser rendering by orchestrating a fallback cascade that automatically launches Playwright stealth browsing when initial lightweight Trafilatura or requests extraction fails against anti-bot defenses.

Does this web scraping approach work for uncovering undocumented APIs?▼

Uncover undocumented APIs by applying reverse-engineering tips within the scraping cascade, allowing you to extract hidden endpoints and monitor feed content that powers downstream intelligence and research workflows.

Related Skills

Explore More Data & Analytics Skills→
pingmepipingmepi

pm-stage-01-brief

Generate a structured 7-section product brief from a business statement.

Community
Advanced
amychina12amychina12

paper-standardizer

Standardize raw academic paper data into analysis-ready datasets with Python codebooks.

Community
Advanced
pynbj1001pynbj1001
2

scrapling-official

Scrapes JavaScript-rendered and anti-bot protected websites using adaptive parsing and stealth browser automation.

Community
Advanced
kevinnftkevinnft
13

web-scraping

Extract structured data from static HTML, JS-rendered pages, and hybrid sites.

Community
Intermediate