skills.rest
peterbamuhigire
peterbamuhigireCOMMUNITY
Explore all skills by this author →
Website𝕏@peterbamuGitHub

web-scraping-foundations

Select data sources, parsers, and politeness controls for web-scraping tasks.

1|Updated Apr 25, 2026
Search Tags:#headless#foundation#data-extraction#web-scraping#structured-data#decision-tree#politeness
One-click install
npx skills add https://github.com/peterbamuhigire/digital-research-skills --skill web-scraping-foundations
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping-foundations
Source: https://github.com/peterbamuhigire/digital-research-skills/tree/main/skills/web-scraping-foundations
Command: npx skills add https://github.com/peterbamuhigire/digital-research-skills --skill web-scraping-foundations

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

The Web Scraping Foundations skill provides a single entry point for web-scraping tasks, encoding a decision tree, structured-data shortcuts, and orchestration rules across politeness, troubleshooting, and browser automation.

Core Features & Use Cases

  • The decision tree (API → JSON XHR → HTML → headless) and structured-data shortcuts reduce effort and increase reliability.
  • On-demand references for politeness, troubleshooting, and browser automation keep knowledge decoupled from execution.
  • Companion skill for Python scaling when crawls exceed ~1,000 URLs.

Quick Start

Load this skill and start a web-scraping task by following the decision tree to choose the appropriate data source, parser, and politeness controls.

Frequently Asked Questions about web-scraping-foundations

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between API, HTML scraping, and headless browsers for web scraping?▼

Reliable web scraping enforces robots.txt handling, implements backoff strategies, and follows load-order discipline. These politeness controls prevent server overload, avoid IP blocks, and ensure policy-compliant data extraction across both API and HTML scraping workflows.

When do I need a headless browser for web scraping structured data?▼

Structured data shortcuts in web scraping reduce extraction effort by leveraging existing JSON-LD, microdata, or API responses directly. This bypasses complex HTML parsing, increasing reliability and speeding up data acquisition when target sites expose structured formats.

How do I handle errors and troubleshooting in a web scraping task?▼

Web scraping troubleshooting applies an error taxonomy loaded through on-demand references to categorize failures across data sources and parsers. This structured approach isolates issues in politeness controls, browser automation, or parsing logic for precise resolution.

Can I use this web scraping approach for large crawls exceeding 1,000 URLs?▼

For large web scraping crawls exceeding roughly 1,000 URLs, a companion skill for Python scaling is required. The foundational decision tree handles source selection and politeness, while the companion skill manages the orchestration of high-volume data extraction.

Related Skills

Explore More Software Engineering Skills→
ssmleossmleo

govfolio-polite-fetching

Fetch data from official sources with bounded concurrency and politeness.

Community
Intermediate
robotics-playgroundrobotics-playground

bmad-analyst

Analyze datasets to generate structured insights and recommended actions.

Official
Basic
FOUNDUPSFOUNDUPS
2

qwen_cli_refactor

Extract logical command modules from monolithic Python CLI files.

Official
Advanced
MonichreMonichre

open-researcher

Gather credible open-source research sources with ranked summaries and citations.

Community
Basic