skills.rest
kileroppo
kileroppoCOMMUNITY
Explore all skills by this author →
GitHub

web-scraper

Extract structured datasets from web pages into Markdown, JSON, or CSV.

Updated Aug 23, 2026
Search Tags:#data extraction#json#data validation#web scraping#csv#pagination#javascript rendering
One-click install
npx skills add https://github.com/kileroppo/AI-Coding-Guidance-Skills --skill web-scraper-kileroppo
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/kileroppo/AI-Coding-Guidance-Skills/tree/main/skills/web-scraper
Command: npx skills add https://github.com/kileroppo/AI-Coding-Guidance-Skills --skill web-scraper-kileroppo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Web data often arrives as messy HTML, dynamic JavaScript pages, or paginated listings, making manual copy-paste slow and error-prone; this Skill turns that web content into clean, structured outputs you can analyze or import.

Core Features & Use Cases

  • Multi-strategy scraping: intelligently uses static fetching, browser automation for JS-rendered pages, and API/cURL approaches when available to reliably access the target data.
  • Extraction across common data types: supports tables, lists, product/pricing info, contacts, FAQs, events, jobs, and custom field schemas.
  • Pagination, monitoring, and comparison: follows paginated navigation, handles infinite scroll/load-more patterns (browser mode), and can detect changes across runs (diff mode).
  • Clean structured delivery: validates results, assigns confidence (HIGH/MEDIUM/LOW), and exports in Markdown table (default), JSON, or CSV with a consistent delivery envelope.
  • Use case: extract pricing plan tiers and feature sets from multiple SaaS pages into a single comparison table, even when some pages require JS rendering or have separate API/download endpoints.

Quick Start

Ask the AI to scrape a given URL for a specific data target and return the results as a Markdown table with pagination enabled.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured data from a website that requires JavaScript rendering?▼

To extract structured data from a website that requires JavaScript rendering, the scraper intelligently switches to browser automation mode to fully load dynamic page content before pulling the targeted records. This ensures accurate data capture from JS-heavy pages by fully rendering the DOM elements first.

What's the best way to scrape paginated listings and infinite scroll pages into a single dataset?▼

The best way to scrape paginated listings and infinite scroll pages is to use automated navigation that follows pagination links or triggers load-more patterns sequentially. The scraper aggregates all extracted records across pages and validates them into a single structured dataset before final export.

Can I export scraped web data directly to JSON or CSV format?▼

Yes, you can export scraped web data directly to JSON or CSV format. The scraper transforms and validates extracted records, allowing you to choose your preferred output format, with Markdown tables serving as the default delivery envelope for immediate analysis or system import.

Does the web scraper validate the accuracy of extracted contacts and product pricing details?▼

Yes, the web scraper validates the accuracy of extracted contacts and product pricing details by applying validation checks and assigning a HIGH, MEDIUM, or LOW confidence score to the extracted dataset. This ensures the final output meets quality standards before it is delivered to the user.

Why does static web scraping fail on some sites, and how is it handled?▼

Static web scraping fails on some sites because the target data is loaded dynamically via JavaScript or protected behind API endpoints. When static fetching fails, the scraper uses a safe fallback escalation strategy, automatically switching to browser automation or cURL-based API approaches to ensure reliable data access.

Related Skills

Explore More Data & Analytics Skills→
chicagopeabodydev-sudochicagopeabodydev-sudo

pydantic-skill

Define structured data contracts for LLM outputs using Pydantic BaseModel.

Community
Intermediate
cmetechcmetech

data-analyst

Ingest CSV, Excel, or JSON files and analyze them with SQL to produce dashboards, reports, or datasets.

Community
Advanced
ikki-daliikki-dali

hojokin-navi

Consolidate subsidy data from J-Grants and prefectural sources into Supabase.

Community
Advanced
zhayujiezhayujie
46.4k

web-fetch

Fetch web page HTML and extract title and readable text.

Community
Basic