skills.rest
l3mpire
l3mpireOFFICIAL
Explore all skills by this author →
GitHub

website-scraper

Extract structured data from web pages into a clean CSV file.

9|3|Updated Apr 13, 2026
Search Tags:#csv#sitemap#pagination#requests#data-extraction#web-scraping#beautifulsoup
One-click install
npx skills add https://github.com/l3mpire/claude-skills --skill website-scraper
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: website-scraper
Source: https://github.com/l3mpire/claude-skills/tree/main/website-scraper
Command: npx skills add https://github.com/l3mpire/claude-skills --skill website-scraper

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manually extracting structured data from websites is slow, error-prone, and hard to scale; this Skill automates locating, cleaning, deduplicating, and exporting web data so you get a ready-to-use CSV without manual copy‑paste work.

Core Features & Use Cases

  • Flexible Page Handling: Handles single pages, paginated listings, sitemap crawls, and domain-wide crawls to capture lists of items, articles, products, jobs, or contacts.
  • Robust Extraction & Cleaning: Uses HTML analysis or a requests+BeautifulSoup fallback, normalizes URLs/emails, strips HTML, removes duplicates, and performs basic QA on sparsity and encoding.
  • Responsible Scraping: Includes robots.txt checks, rate-limit/backoff handling, Cloudflare/CAPTCHA detection guidance, and explicit user confirmation for disallowed paths.
  • Use Case: Extract all job postings from a company's careers pages into a CSV with columns job_title, location, apply_url, and published_date.

Quick Start

Scrape the provided URL to extract the specified fields (for example: title, price, email) and save the cleaned results to scraped-data.csv.

Frequently Asked Questions about website-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured data from multiple web pages into a CSV file?▼

To extract structured data into a CSV file, this process handles single pages, paginated listings, sitemap crawls, and domain-wide crawls. It applies HTML fetching, BeautifulSoup fallback, deduplication, and UTF-8-sig CSV export.

What is the best way to scrape paginated listings and sitemaps for product data?▼

Scraping paginated listings and sitemaps for product data requires automated pagination and sitemap discovery. This approach locates items, normalizes URLs, strips HTML, removes duplicates, and exports clean CSV data.

Does web scraping with BeautifulSoup handle robots.txt checks and rate-limiting?▼

Yes, responsible web scraping with BeautifulSoup includes robots.txt checks, rate-limit and backoff handling, and explicit user confirmation for disallowed paths. It also provides Cloudflare and CAPTCHA detection guidance.

Can I extract job postings and contact information from a domain-wide crawl?▼

Yes, you can extract job postings and contacts from a domain-wide crawl. The extraction normalizes emails and URLs, performs basic QA on sparsity and encoding, and saves the cleaned results to a CSV file.

Why does web scraping fail on complex pages and how do I fix it?▼

Web scraping fails on complex pages due to dynamic content or HTML structure issues. Using a requests and BeautifulSoup fallback with configurable headers helps analyze and extract structured data that standard HTML fetching misses.

Related Skills

Explore More Data & Analytics Skills→
rodydavisrodydavis
58

json-to-html-table-with-lit

Generate dynamic HTML tables from JSON data using Lit web components.

Community
Intermediate
alongor666alongor666
1

backend-data-processor

Process vehicle insurance Excel/CSV data into clean, merged datasets.

Community
Intermediate
HokageZHokageZ
2

document-parsers

Parse and extract text, tables, and metadata from PDF, DOCX, HTML, and Markdown files.

Community
Advanced
sky770825sky770825

playwright-scraper-skill

Automate web scraping of dynamic websites with Playwright and anti-bot bypass.

Community
Intermediate