skills.rest
Prorise-cool
Prorise-coolCOMMUNITY
Explore all skills by this author →
GitHub

web-scraping-playbook

Analyze website rendering and anti-scraping mechanisms to plan data extraction strategies.

24|3|Updated Jan 24, 2026
Search Tags:#automation#data extraction#web scraping#site reconnaissance#anti-scraping#scraping strategy
One-click install
npx skills add https://github.com/Prorise-cool/prorise-claude-skills --skill web-scraping-playbook
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping-playbook
Source: https://github.com/Prorise-cool/prorise-claude-skills/tree/main/.claude/skills/scraping-specialist/references/domains/general-web-scraping
Command: npx skills add https://github.com/Prorise-cool/prorise-claude-skills --skill web-scraping-playbook

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you devise the most efficient and robust strategy for scraping any website, overcoming common obstacles like anti-scraping measures and complex site structures.

Core Features & Use Cases

  • Site Reconnaissance: Analyze website rendering, data sources, and navigation.
  • Strategy Selection: Prioritize methods from sitemaps and APIs to browser automation.
  • Anti-Scraping Mitigation: Plan for handling rate limits, Cloudflare, and other protections.
  • Use Case: You need to scrape product data from a new e-commerce site. This Skill will guide you through identifying the best data source (API vs. HTML), handling pagination, and preparing for potential blocks, before you write any code.

Quick Start

Use the web-scraping-playbook skill to determine the best strategy for scraping example.com.

Frequently Asked Questions about web-scraping-playbook

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is the best way to scrape a website with anti-scraping measures like Cloudflare?▼

To handle rate limiting and anti-scraping protections, develop a comprehensive scraping strategy that prioritizes lightweight HTTP requests, hidden APIs, and sitemaps over heavy browser automation, reducing the footprint that triggers rate limits and Cloudflare blocks.

How do I extract data from dynamic websites without using browser automation?▼

You can extract data from dynamic websites by performing site reconnaissance to find hidden APIs or data entry points that load dynamic content, then prioritizing these lightweight HTTP requests over browser automation for efficient data extraction.

When should I use sitemaps and APIs instead of browser automation for web scraping?▼

Use sitemaps and APIs instead of browser automation when site reconnaissance reveals accessible data sources, as prioritizing these lightweight HTTP requests delivers efficient data extraction while avoiding complex rendering and anti-scraping triggers.

How do I bypass Cloudflare and rate limits when scraping e-commerce product data?▼

To handle Cloudflare and rate limits when scraping e-commerce product data, devise a strategy that analyzes rendering methods and prioritizes lightweight data sources like APIs over browser automation, effectively mitigating anti-scraping mechanisms before writing code.

Does site reconnaissance help with scraping strategy selection for arbitrary websites?▼

Yes, site reconnaissance helps scraping strategy selection by analyzing arbitrary websites' rendering methods, data sources, and navigation, allowing you to prioritize efficient extraction methods ranging from sitemaps and APIs to browser automation.

Related Skills

Explore More Software Engineering Skills→
olostepolostep

map

Discover and list all URLs on a website with keyword and pattern filtering.

Community
Intermediate
zhayujiezhayujie
46.4k

web-fetch

Fetch web page HTML and extract title and readable text.

Community
Basic
atomeamatomeam
2

dynamic-ui-automation-javascript

Execute JavaScript and hover events for dynamic UI elements in Playwright.

Community
Intermediate
magnus919magnus919
40

flaresolverr-cli

Execute browser-backed HTTP requests to bypass Cloudflare and DDoS-GUARD anti-bot protections.

Community
Intermediate