skills.rest
LeoYeAI
LeoYeAICOMMUNITY
Explore all skills by this author →
Website𝕏@LeoYe_AIGitHub

openclaw-ultra-scraping

Scrape web data with anti-bot bypass and dynamic content rendering.

22|7|Updated Mar 5, 2026
Search Tags:#automation#web scraping#dynamic content#crawling#anti-bot#headless browser
One-click install
npx skills add https://github.com/LeoYeAI/openclaw-ultra-scraping --skill openclaw-ultra-scraping
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openclaw-ultra-scraping
Source: https://github.com/LeoYeAI/openclaw-ultra-scraping/tree/main
Command: npx skills add https://github.com/LeoYeAI/openclaw-ultra-scraping --skill openclaw-ultra-scraping

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scrapy, playwright, beautifulsoup4, api-python-client, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill enables AI assistants to perform robust web scraping and crawling by overcoming anti-bot measures and handling site redesigns seamlessly.

Core Features & Use Cases

  • Anti-Bot Bypass: Effortlessly bypass Cloudflare, Turnstile, CAPTCHAs, and other anti-detection mechanisms.
  • Adaptive Element Tracking: Maintain selector accuracy despite webpage reorganization or redesigns.
  • Large-Scale Crawling: Conduct multi-page crawling with concurrency, pause/resume, and proxy rotation.
  • Precise Data Extraction: Use CSS and XPath selectors, with support for JavaScript-rendered pages via headless browsers.
  • Use Case: Collect product data from protected e-commerce sites or aggregate content from dynamic SPA websites automatically.

Quick Start

Ask your AI assistant to scrape product details from a protected site using the stealth mode with adaptive element tracking.

Frequently Asked Questions about openclaw-ultra-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape dynamic content from JavaScript-rendered SPA websites?▼

You can scrape dynamic content from JavaScript-rendered pages using headless browsers. This Skill leverages Playwright to execute JavaScript, render single-page application content, and extract precise data using CSS and XPath selectors.

What's the best way to bypass Cloudflare and CAPTCHAs when web scraping?▼

To bypass anti-bot systems like Cloudflare, Turnstile, and CAPTCHAs during web scraping, this Skill uses stealth mode with custom fetchers. It circumvents anti-detection mechanisms automatically to ensure uninterrupted data collection.

Can I use scrapy for large-scale crawling with proxy rotation and pause/resume capabilities?▼

Yes, you can conduct large-scale crawling with scrapy. This Skill supports multi-page concurrency, proxy rotation, and pause/resume capabilities to handle scalable data extraction workflows efficiently.

How do I maintain web scraping accuracy when a target website redesigns its layout?▼

To maintain scraping accuracy when target websites redesign their layout, this Skill uses adaptive element tracking. It automatically adjusts to webpage reorganizations to keep your CSS and XPath selectors functioning correctly.

Does web scraping with playwright require anti-bot bypass for protected e-commerce sites?▼

Yes, scraping protected e-commerce sites with Playwright often requires anti-bot bypass. This Skill integrates stealth modes with headless browser rendering to evade detection while collecting product data.

Why does web scraping fail on sites with dynamic content and anti-bot protection?▼

Web scraping fails on dynamic sites with anti-bot protection because standard requests cannot execute JavaScript or pass security checks like Cloudflare. This Skill overcomes these barriers using headless browsers and stealth fetchers.

Related Skills

Explore More Content & Communication Skills→
nathanvalenathanvale
2

playwright-scraper

Automate web scraping with Playwright using semantic selectors and error handling.

Community
Advanced
h4vzzh4vzz
33

data-cleaning

Clean raw datasets into analysis-ready tables using pandas, pyjanitor, and great_expectations.

Community
Advanced
zhouziyue233zhouziyue233
6

scrapling

Automate web data collection with Python-based fetching, parsing, and crawling.

Community
Advanced
tedtv1007-ctrltedtv1007-ctrl

cloudflare-crawl

Fetch rendered HTML from dynamic websites via the Cloudflare Browser Rendering API.

Community
Intermediate