skills.rest
tusosos
tusososCOMMUNITY
Explore all skills by this author →
GitHub

web-scraper

Extract data from web pages using CSS selectors and output JSON or CSV.

Updated Apr 3, 2026
Search Tags:#automation#etl#data-extraction#data-pipeline#web-scraping#css-selectors
One-click install
npx skills add https://github.com/tusosos/manus-knowledge-base --skill web-scraper-tusosos
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/tusosos/manus-knowledge-base/tree/main/skills/web-scraper
Command: npx skills add https://github.com/tusosos/manus-knowledge-base --skill web-scraper-tusosos

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Web pages often present data in unstructured HTML, making extraction and ingestion into databases labor-intensive. This Skill automates the collection, normalization, and loading of data from websites into structured formats like JSON or CSV, enabling reliable analytics and data pipelines.

Core Features & Use Cases

  • HTML parsing and CSS-selector based extraction to gather items such as product names, prices, links, and tables from both static and JS-rendered pages.
  • Data transformation and deduplication to normalize fields and remove duplicates for robust ETL workflows.
  • Use Case: Build a scalable scraper that aggregates catalog data across sites and loads it into a data warehouse for analysis.

Quick Start

Ask me to scrape a catalog page and save the results to a CSV file.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract data from HTML pages and convert it to JSON or CSV?▼

To extract data from HTML pages, identify target URLs and specific elements like prices or links, then use CSS selectors to parse the content and emit structured JSON or CSV files for downstream loading.

Can I scrape JavaScript-rendered web pages using CSS selectors?▼

Yes, you can scrape JavaScript-rendered web pages by configuring the crawler to use a headless browser instead of static HTML requests, allowing CSS selectors to extract dynamically generated data elements.

How do I build a data pipeline to scrape and deduplicate product catalogs?▼

Build a data pipeline by fetching catalog pages, applying data transformation and deduplication to normalize fields and remove duplicates, then emitting a clean, load-ready payload for databases.

Does web scraping support pagination for large ETL workflows?▼

Yes, web scraping supports pagination to crawl across multiple pages, extracting and normalizing items sequentially to aggregate catalog data for robust ETL workflows and data warehouse loading.

What is the best way to automate web scraping for analytics and database ingestion?▼

The best way to automate web scraping for analytics is to automate collection and normalization of unstructured HTML into structured formats like JSON or CSV, providing a reliable load-ready payload for databases.

Do I need to handle data transformation before loading scraped web data into a warehouse?▼

You do not need external transformation before loading, because the scraping process handles data transformation and deduplication internally, normalizing fields and removing duplicates to provide a clean payload.

Related Skills

Explore More Data & Analytics Skills→
quazfentonquazfenton

web-scraper

Crawl websites and extract structured data using tiered scraping tools.

Community
Advanced
TurnaboutHeroTurnaboutHero
12

data-wizard

Automate ETL and data transformation for CSV, JSON, and database sources using Python and pandas.

Community
Advanced
CatsofsufferingCatsofsuffering
5

data-engineering

Manage and validate data pipelines and streaming workflows across orchestration and quality tools.

Community
Advanced
xiaohei-infoxiaohei-info
28

web-reading-router

Route URL reading requests across platform-specific tools and generic extraction methods.

Community
Intermediate