skills.rest
guia-matthieu
guia-matthieuCOMMUNITY
Explore all skills by this author →
GitHub

web-scraper

Extract structured data from websites using CSS selectors and predefined schemas.

145|28|Updated Jan 31, 2026
Search Tags:#automation#data extraction#web scraping#lead generation#competitor analysis#web data
One-click install
npx skills add https://github.com/guia-matthieu/clawfu-skills --skill web-scraper-guia-matthieu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/guia-matthieu/clawfu-skills/tree/main/skills/automation/web-scraper
Command: npx skills add https://github.com/guia-matthieu/clawfu-skills --skill web-scraper-guia-matthieu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires beautifulsoup4, requests, lxml, pandas, click, and includes scripts (resource) components.

What problem does it solve?

This Skill automates the process of extracting specific information from websites, turning unstructured web content into usable data for analysis and other applications.

Core Features & Use Cases

  • Data Extraction: Scrape elements like pricing, product details, contact information, and links from web pages.
  • Structured Output: Organize extracted data into formats like JSON or CSV for easy integration into other tools.
  • Use Case: Automatically scrape competitor pricing information from multiple product pages to inform your own pricing strategy.

Quick Start

Use the web-scraper skill to scrape all h1, h2, and p elements from https://example.com.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured data from websites using CSS selectors?▼

To extract structured data using CSS selectors, the web scraper fetches HTML content with requests and parses specific elements using BeautifulSoup. You define predefined schemas to target elements like pricing or contact information for automated extraction.

What's the best way to automate competitor price monitoring from web pages?▼

Automating competitor price monitoring involves scraping product pages using predefined CSS selector schemas. The extracted pricing data is structured into JSON or CSV formats, enabling easy integration into your pricing strategy analysis.

Can I use BeautifulSoup to scrape contact information for lead generation?▼

Yes, BeautifulSoup can scrape contact information for lead generation by parsing targeted HTML elements. By defining specific CSS selectors for contact details, the scraper outputs structured data ready for integration into your CRM tools.

How do I export scraped web data into JSON or CSV formats?▼

Exporting scraped web data into JSON or CSV formats is handled automatically during the extraction process. The scraper organizes parsed HTML elements into structured outputs using pandas, facilitating seamless data integration.

Do I need lxml and requests to parse HTML content for a content audit?▼

Yes, you need requests to fetch the web pages and BeautifulSoup with lxml to parse the HTML content for a content audit. These libraries work together to extract targeted elements like h1, h2, and p tags efficiently.

What are the limitations of using predefined schemas for web scraping?▼

Using predefined schemas for web scraping limits flexibility when target websites change their HTML structure or CSS selectors. This approach requires updating your schemas manually to maintain accurate data extraction from dynamic web pages.

Related Skills

Explore More Marketing & Sales Skills→
Gimpy666Gimpy666
1

scraped_docs

Convert Tribot JavaDoc HTML pages into per-class Markdown files and a consolidated reference.

Community
Intermediate
ThomasRohdeThomasRohde
1

ecm-import-mapper

Maps CSV exports to ECM Studio import artifacts using a mapping JSON.

Community
Intermediate
Prorise-coolProrise-cool
24

structured-web-data-extractor

Extract structured web data into CSV, JSON, or Markdown tables using Playwright MCP.

Community
Intermediate
appautomatonappautomaton
18

html-to-markdown

Converts HTML or live web pages into structured Markdown with metadata and link inventories.

Official
Intermediate