intelligent-web-scraper

Learn and reuse web scraping patterns across list and detail pages.

33|6|Updated Jan 24, 2026

Search Tags:#Playwright #series #pagination #web-scraping #crawl4ai #learned-patterns #anti-blocking

Ready to run

npx skills add https://github.com/Yrzhe/claude-skills --skill intelligent-web-scraper

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: intelligent-web-scraper
Source: https://github.com/Yrzhe/claude-skills/tree/main/plugins/intelligent-web-scraper/skills/intelligent-web-scraper
Command: npx skills add https://github.com/Yrzhe/claude-skills --skill intelligent-web-scraper

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires crawl4ai, playwright, websockets, pydantic, aiohttp, beautifulsoup4, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a self-learning, adaptive web-scraping agent that analyzes page structure, handles pagination and lazy-loaded content, follows detail links for complete data, and remembers successful patterns to speed up future tasks.

Core Features & Use Cases

Self-learning pattern memory: automatically captures and reuses domain-specific selectors, delays, and anti-blocking strategies.
Robust page navigation: auto-detects pagination types, scroll loading, and detail-link traversal to capture full content.
Resilient workflows: adaptive delays, user-agent rotation, and CAPTCHA handling notifications.
Series discovery: identifies and aggregates multi-article series from a single entry point.
Use Case: Automate extraction of article lists and their detail pages across blogs with lazy loading and anti-bot protections.

Quick Start

Run the wrapper on a target URL to start crawling and observe the agent learn patterns over time using the provided scripts.

Frequently Asked Questions about intelligent-web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scrape lazy-loaded content and pagination without getting blocked?▼

Web scraping lazy-loaded content and pagination requires adaptive delays and user-agent rotation. This agent auto-detects scroll loading and pagination types, applying anti-blocking strategies to capture full content reliably.

Can I reuse learned web scraping selectors across different sites?▼

Yes, reusing learned web scraping selectors is possible through a centralized experience database. The agent captures domain-specific selectors, delays, and strategies, storing them to speed up future crawling tasks continually.

What is the best way to extract multi-article series from a single entry point?▼

Extracting multi-article series is best handled by series discovery mechanisms. The scraper identifies and aggregates linked articles from a single entry point, automatically following detail links to capture complete data.

Does Playwright work well for crawling detail pages with anti-bot protections?▼

Playwright works effectively for crawling detail pages with anti-bot protections when paired with adaptive delays. The agent uses it to handle resilient workflows, including user-agent rotation and CAPTCHA handling notifications.

How to crawl list pages and detail pages using an adaptive web scraper?▼

To crawl list pages and detail pages, run the provided wrapper script on a target URL. The adaptive web scraper analyzes page structure, follows detail links, and learns patterns over time to automate extraction.

Why does my web scraper fail on websites with dynamic scroll loading?▼

Web scrapers fail on dynamic scroll loading when they lack robust page navigation logic. This agent handles scroll loading by auto-detecting lazy content patterns and applying adaptive delays to retrieve full page structure.