website-scraper

What problem does it solve?

Manually extracting structured data from websites is slow, error-prone, and hard to scale; this Skill automates locating, cleaning, deduplicating, and exporting web data so you get a ready-to-use CSV without manual copy‑paste work.

Core Features & Use Cases

• Flexible Page Handling: Handles single pages, paginated listings, sitemap crawls, and domain-wide crawls to capture lists of items, articles, products, jobs, or contacts.
• Robust Extraction & Cleaning: Uses HTML analysis or a requests+BeautifulSoup fallback, normalizes URLs/emails, strips HTML, removes duplicates, and performs basic QA on sparsity and encoding.
• Responsible Scraping: Includes robots.txt checks, rate-limit/backoff handling, Cloudflare/CAPTCHA detection guidance, and explicit user confirmation for disallowed paths.
• Use Case: Extract all job postings from a company's careers pages into a CSV with columns job_title, location, apply_url, and published_date.

Quick Start

Scrape the provided URL to extract the specified fields (for example: title, price, email) and save the cleaned results to scraped-data.csv.