skills.rest
AlexAlvarezAlmendros
AlexAlvarezAlmendrosCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

web-scraping

Scrape web content with anti-bot bypass and dynamic rendering.

Updated Aug 27, 2026
Search Tags:#data extraction#api#web scraping#yt-dlp#playwright#anti-bot#instaloader
One-click install
npx skills add https://github.com/AlexAlvarezAlmendros/HomeScrapper --skill web-scraping-alexalvarezalmendros
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/AlexAlvarezAlmendros/HomeScrapper/tree/main/.github/skills/.agents/skills/web-scraping
Command: npx skills add https://github.com/AlexAlvarezAlmendros/HomeScrapper --skill web-scraping-alexalvarezalmendros

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, trafilatura, playwright, yt-dlp, instaloader, fake_useragent, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill tackles the challenge of extracting valuable content from websites, even when faced with anti-bot measures, paywalls, or dynamic content loading.

Core Features & Use Cases

  • Multi-Strategy Scraping: Employs a cascade of methods (requests, Trafilatura, Playwright) with automatic fallbacks for robust data retrieval.
  • Anti-Bot Bypass: Utilizes Playwright with stealth mode and rotating user agents to circumvent detection.
  • Content Extraction: Extracts main article content, titles, and can handle JavaScript-rendered pages.
  • Poison Pill Detection: Identifies and flags paywalls, captchas, and rate limits.
  • Social Media Scraping: Includes patterns for YouTube (metadata, video/audio download, transcripts) and Instagram (post data, media download) using yt-dlp and instaloader.
  • Undocumented API Discovery: Provides methods for reverse-engineering and utilizing hidden APIs.
  • Use Case: Scrape product details from an e-commerce site that heavily relies on JavaScript, ensuring you get all product information despite anti-scraping measures.

Quick Start

Use the web-scraping skill to extract the main content and title from the URL 'https://example.com'.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape JavaScript-rendered web pages that block standard requests?▼

Scraping JavaScript-rendered pages reliably requires Playwright with stealth mode and rotating user agents to bypass anti-bot detection. This method renders dynamic content that standard requests cannot process, ensuring successful data extraction.

What is the best way to extract main article text while avoiding paywalls?▼

Extracting main article text while avoiding paywalls involves using Trafilatura for static content alongside poison pill detection to identify and flag paywalls or rate limits. This ensures you retrieve actual content instead of subscription prompts.

Can I download YouTube videos and transcripts using yt-dlp for web scraping?▼

Yes, you can use yt-dlp for YouTube scraping to retrieve video metadata, download video or audio files, and extract transcripts. This handles specific platform requirements for reliable media data retrieval.

How do I find and use undocumented APIs for data extraction?▼

Finding and using undocumented APIs involves reverse-engineering network requests to discover hidden endpoints, then utilizing those APIs directly for data extraction. This method bypasses front-end rendering entirely for more efficient scraping.

Does Playwright work with Instagram data retrieval alongside instaloader?▼

Instagram data retrieval is supported through instaloader to fetch post data and download media files. Playwright handles broader dynamic content rendering, while instaloader targets specific platform scraping patterns for reliable extraction.

Why does web scraping fail when encountering rate limits and captchas?▼

Web scraping fails on rate limits and captchas because sites deploy anti-bot measures to block automated access. The framework includes poison pill detection to identify these blocks and rotating user agents to circumvent detection.

Related Skills

Explore More Software Engineering Skills→
MaxMiksaMaxMiksa
2.3k

web-scraping

Automate web extraction with Trafilatura, Requests, Playwright stealth, and yt-dlp or instaloader for sites and social media.

Community
Advanced
asapMakiasapMaki

gastrohem-media-processor

Transcribe audio files and OCR images from Gastrohem WhatsApp folders.

Community
Advanced
pynbj1001pynbj1001
2

scrapling-official

Scrapes JavaScript-rendered and anti-bot protected websites using adaptive parsing and stealth browser automation.

Community
Advanced
ThrookieThrookie
2

playwright-scraper

Automate stealth web scraping of protected sites using Playwright and stealth plugins.

Community
Intermediate