skills.rest
vivekgoquest
vivekgoquestCOMMUNITY
Explore all skills by this author →
GitHub

scrapling

Scrape websites using HTTP, dynamic browser, and stealth fetching with Cloudflare bypass.

Updated Aug 22, 2026
Search Tags:#data extraction#python#web scraping#crawling#scrapling#cloudflare bypass#stealth browser
One-click install
npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill scrapling-vivekgoquest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scrapling
Source: https://github.com/vivekgoquest/hermes-agent-stable/tree/main/optional-skills/research/scrapling
Command: npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill scrapling-vivekgoquest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scrapling.

What problem does it solve? Extracting data from websites often fails when pages require JavaScript rendering or are protected by anti-bot systems like Cloudflare Turnstile, leaving standard HTTP scrapers and built-in extraction tools unable to retrieve the content. ## Core Features & Use Cases - Three Fetching Strategies: Choose between fast HTTP requests (Fetcher), JS-rendered browser automation (DynamicFetcher), or stealth mode with Cloudflare solving (StealthyFetcher) depending on the target site's defenses. - Spider Framework: Crawl multiple pages with link following, concurrent requests, multi-session routing, and pause/resume checkpoints. - CLI and Python APIs: Extract pages directly from the command line into HTML, Markdown, text, or JSON, or write Python scripts with CSS, XPath, and regex element selection. - Use Case: A researcher needs to collect product listings from a Cloudflare-protected e-commerce site. They use StealthyFetcher with solve_cloudflare enabled to bypass the challenge, then parse prices and titles with CSS selectors into structured JSON. ## Quick Start Use the scrapling skill to scrape the quotes from https://quotes.toscrape.com and save them to a JSON file.

Frequently Asked Questions about scrapling

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a Cloudflare-protected website in Python?▼

Use Scrapling's StealthyFetcher with solve_cloudflare=True to bypass Cloudflare Turnstile challenges. Options like block_webrtc and hide_canvas reduce browser fingerprinting, though solving adds 5-15 seconds per fetch.

How to scrape JavaScript-rendered pages without a full browser framework?▼

Use Scrapling's DynamicFetcher, which runs a headless browser to execute JavaScript and return the rendered DOM. You can wait for specific selectors, block resources for speed, and inject custom Playwright page actions.

What is the difference between Fetcher, DynamicFetcher, and StealthyFetcher?▼

Fetcher makes plain HTTP requests for static pages and APIs. DynamicFetcher renders JavaScript in a real browser for SPAs. StealthyFetcher adds anti-fingerprinting and Cloudflare solving for protected sites.

Why does Scrapling fail with browser errors after pip install?▼

DynamicFetcher and StealthyFetcher require browser binaries that pip does not install. Run 'scrapling install' after installing the package to download the required browser dependencies.

Does Scrapling support multi-page crawling with sessions?▼

Yes, the Spider framework supports concurrent requests, download delays, link following, and routing requests to different session types like FetcherSession or AsyncStealthySession. It also supports pause and resume via checkpoint directories.

What are the limitations of Cloudflare bypass scraping?▼

Cloudflare solving adds 5-15 seconds per request and stealth mode runs a real browser, limiting concurrency. Users must also respect robots.txt and website Terms of Service, as scraping may be restricted by law or site policy.

Related Skills

Explore More Software Engineering Skills→
ovisanovisan
2

firecrawl-map

Discovers and lists all URLs on a website with optional search filtering.

Community
Basic
juricamigac-from-hrjuricamigac-from-hr

scrape-webpage

Scrape webpage content, metadata, and images into migration-ready JSON and assets.

Official
Advanced
ScrapingBeeScrapingBee
107

scrapingbee-cli

Scrape, crawl, and extract structured data from websites via the ScrapingBee API.

Official
Advanced
mazreanmazrean

building-static-site-rss-feeds

Generate RSS 2.0, Atom 1.0, and JSON feeds for static sites.

Community
Intermediate