skills.rest
matthew-johnson
matthew-johnsonCOMMUNITY
Explore all skills by this author →
GitHub

scrapling

Automate web scraping with HTTP fetching, JavaScript rendering, and Cloudflare bypass.

Updated Apr 20, 2026
Search Tags:#python#cli#web scraping#cloudflare#spider#stealth#scrapling
One-click install
npx skills add https://github.com/matthew-johnson/hermes-agent --skill scrapling-matthew-johnson
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scrapling
Source: https://github.com/matthew-johnson/hermes-agent/tree/main/optional-skills/research/scrapling
Command: npx skills add https://github.com/matthew-johnson/hermes-agent --skill scrapling-matthew-johnson

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Scrapling addresses the challenge of extracting data from websites that employ anti-bot measures and client-side rendering by providing HTTP fetching, dynamic JS rendering, stealth browser automation, and a spider framework accessible via CLI and Python.

Core Features & Use Cases

  • HTTP fetching for static pages and APIs with fast results
  • Dynamic rendering to handle JavaScript-heavy pages and SPAs
  • Stealth + Cloudflare bypass to access protected sites with reduced risk of detection
  • Spider framework for multi-page crawling and link following
  • CLI & Python APIs for easy integration into data pipelines and automation
  • Real-world use case: build a dataset by crawling product listings across pages and extract structured fields into CSV/JSON.

Quick Start

Install Scrapling and run scrapling to fetch a page from a URL.

Frequently Asked Questions about scrapling

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a website that uses Cloudflare anti-bot protection?▼

To scrape JavaScript-rendered SPAs, you need dynamic rendering to execute client-side scripts. This Skill handles dynamic JS rendering to extract fully loaded content from JavaScript-heavy pages and single-page applications via Python or CLI.

Can I crawl multiple pages and follow links automatically in Python?▼

Yes, you can integrate HTTP fetching and data extraction into existing Python automation pipelines. This Skill provides CLI and Python APIs designed for easy integration into data pipelines, allowing structured field extraction into CSV or JSON formats.

What is the best way to extract structured data from dynamic web pages into JSON?▼

The best way to extract structured data from dynamic web pages is using a spider framework with dynamic rendering. This Skill allows crawling product listings across pages and extracting structured fields directly into CSV or JSON datasets.

Does web scraping with stealth bypass work for static pages and APIs too?▼

Yes, stealth web scraping works for both static pages and APIs. This Skill provides HTTP fetching for fast static page and API results, alongside dynamic rendering and stealth bypass for protected or JavaScript-heavy sites.

Related Skills

Explore More Software Engineering Skills→
Chris-Chai-MinjaeChris-Chai-Minjae

scrapling

Automate web scraping with HTTP fetching, stealth browser automation, and spider crawling.

Community
Advanced
disusereddisusered

jq

Parse, filter, and transform JSON data with jq queries.

Community
Intermediate
tangzheng202202tangzheng202202
1

cronjob-scheduling-patterns

Schedule cron-like tasks with timezone-aware specific-time, duration-based, and one-time patterns.

Community
Intermediate
superfhpsuperfhp

scrapling

Extract website data using HTTP requests, stealth browser automation, and spider crawling.

Community
Intermediate