skills.rest
nicoreincarnate-oss
nicoreincarnate-ossCOMMUNITY
Explore all skills by this author →
GitHub

web-scraping

Select and orchestrate web-scraping strategies across production site architectures.

Updated Mar 21, 2026
Search Tags:#apify#web-scraping#api-discovery#crawlee#dom-scraping#traffic-interception
One-click install
npx skills add https://github.com/nicoreincarnate-oss/objective-hertz --skill web-scraping-nicoreincarnate-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/nicoreincarnate-oss/objective-hertz/tree/main/.agent/skills/web-scraper
Command: npx skills add https://github.com/nicoreincarnate-oss/objective-hertz --skill web-scraping-nicoreincarnate-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill helps organizations automate web-scraping by selecting the optimal strategy (traffic interception, sitemap, API, or DOM scraping) and orchestrating end-to-end workflows.

Core Features & Use Cases

  • Adaptive reconnaissance with phased gates (0-5) that skip unnecessary work and minimize resource use.
  • Framework-aware data discovery and production-grade guidance, including API-first patterns and anti-blocking strategies.
  • Production-ready patterns and modular organization for easy migration to Apify Actors or Crawlee crawlers.

Quick Start

Ask Claude to activate adaptive scraping on a target site and follow the multi-phase workflow.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose the best web-scraping strategy for JavaScript-rendered pages?▼

To choose the best web-scraping strategy for JavaScript-rendered pages, this skill applies adaptive reconnaissance to evaluate traffic interception, sitemaps, APIs, and DOM scraping, selecting the optimal method to minimize resource use.

What is traffic interception for web scraping and when should I use it?▼

Traffic interception for web scraping is a strategy that captures data from network requests rather than parsing DOM elements. It is used when sites load content dynamically via internal APIs, bypassing complex JavaScript rendering.

How do I automate web scraping workflows with anti-bot protections?▼

You automate web scraping workflows with anti-bot protections by following this skill's phased, end-to-end orchestration, which applies framework-aware discovery and production-grade anti-blocking strategies across varying site architectures.

Can I use this web scraping workflow with Apify Actors and Crawlee crawlers?▼

Yes, you can use this workflow with Apify Actors and Crawlee crawlers. The skill provides production-ready patterns and modular organization specifically designed for easy migration to these frameworks.

What's the best way to scrape APIs instead of DOM elements?▼

The best way to scrape APIs instead of DOM elements is using an API-first discovery pattern. This skill identifies internal site APIs during its reconnaissance phase, intercepting traffic to extract structured data directly.

Related Skills

Explore More Software Engineering Skills→
olostepolostep

map

Discover and list all URLs on a website with keyword and pattern filtering.

Community
Intermediate
h4vzzh4vzz
33

data-cleaning

Clean raw datasets into analysis-ready tables using pandas, pyjanitor, and great_expectations.

Community
Advanced
acaprinoacaprino
6

grabber-development

Build production-grade Python web scrapers with stealth and anti-bot defenses.

Community
Advanced
kevinnftkevinnft
13

web-scraping

Extract structured data from static HTML, JS-rendered pages, and hybrid sites.

Community
Intermediate