skills.rest
vuongtu1k9-ship-it
vuongtu1k9-ship-itCOMMUNITY
Explore all skills by this author →
GitHub

web-scraper

Extract web page content into structured JSON with configurable YAML pipelines.

Updated Apr 9, 2026
Search Tags:#llm#scraping#web#data-extraction#content-processing#yaml-config#paywall-detection
One-click install
npx skills add https://github.com/vuongtu1k9-ship-it/Kotlin --skill web-scraper-vuongtu1k9-ship-it
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/vuongtu1k9-ship-it/Kotlin/tree/main/skills/web-scraper
Command: npx skills add https://github.com/vuongtu1k9-ship-it/Kotlin --skill web-scraper-vuongtu1k9-ship-it

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Web data is scattered and irregular; extracting consistent text, metadata, and entities across sites requires multiple, ad-hoc steps. This Skill provides a robust, configurable extraction pipeline that surfaces clean, structured data from diverse pages with minimal manual tuning.

Core Features & Use Cases

  • Cascade extraction: starts with static HTML and escalates to JS-rendered content when needed to maximize data fidelity.
  • Metadata and paywall detection: captures publication dates, authors, keywords, and flags paywalled content to handle gracefully.
  • LLM-based entity extraction (optional): derives people, organizations, locations, and events from clean text using a configurable workflow.
  • Configurable, YAML-driven: uses declarative selectors and pipeline settings to adapt to new sites without code changes.
  • Output formats: produces Python scripts, YAML configs, and JSON results suitable for further analytics.

Quick Start

Use the web-scraper on a target URL to generate a JSON file containing the article text and associated metadata.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract web content into structured data from multiple sites?▼

Web scraping extracts structured data from diverse pages using a cascade of static HTML and JS-rendered strategies. A YAML-configurable pipeline handles single-page scrapes and bulk crawls without manual tuning.

What is the best way to extract entities from scraped web text?▼

Entity extraction derives people, organizations, locations, and events from clean text using an optional LLM-based workflow. This processes web content after boilerplate removal to ensure high-fidelity results.

Can I configure web scraping rules without writing code?▼

YAML-driven configuration allows you to declare selectors and pipeline settings to adapt extraction to new sites without code changes. This declarative approach manages static and JS-rendered content extraction.

Does this web scraper handle JavaScript-rendered pages and paywalls?▼

The scraper escalates from static HTML to optional Playwright rendering for JS-rendered content. It also includes paywall detection to gracefully handle restricted pages and capture available metadata.

What output formats are generated by the web scraping pipeline?▼

The extraction pipeline produces Python scripts, YAML configs, and JSON results containing article text and associated metadata. These outputs are suitable for further analytics and downstream processing.

When should I not use a cascade extraction approach for web scraping?▼

Cascade extraction maximizes accuracy by escalating from lightweight to heavyweight strategies, which may add overhead. If a site is strictly static HTML without paywalls or JS rendering, a simpler scraper might suffice.

Related Skills

Explore More Software Engineering Skills→
mepukamepuka

effect-layers-services

Define service interfaces and bind live/test implementations through layers.

Community
Intermediate
aws-samplesaws-samples
90

file-processing

Process and analyze CSV, JSON, and text data files with automated cleaning and transformation.

Official
Advanced
droxeydroxey
1

web-scraper

Extract clean article text and structured news metadata from URLs into JSON.

Community
Advanced
skinred78skinred78
1

Firebase + Cloud Run Deployment Skill

Deploy Flask/Python apps to Google Cloud Run with Firebase Hosting.

Community
Advanced