website-scraping
CommunityExtract structured data from any website with a generic playbook.
Authormoonlight-lupin
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill automates the process of extracting structured data from websites, enabling users to transform unstructured web content into usable datasets.
Core Features & Use Cases
- Website Reconnaissance: Identifies the most suitable extraction tool based on the website's structure.
- Data Extraction: Extracts structured data from websites using various techniques like static HTML parsing, JSON-LD, JSON-in-page, and API reversal.
- Anti-bot Handling: Implements strategies to overcome anti-bot defenses like User-Agent filtering, stealth, and proxy rotation.
- Output Formatting: Generates clean JSONL output with a manifest file for metadata and raw payloads for debugging.
Quick Start
Run the 'website-scraping' skill with the following command: website-scraping run <url>.
Dependency Matrix
Required Modules
urllibrequestsplaywrightbeautifulsoup4lxmlpdfplumberpandas
Components
scriptsreferencesassets
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: website-scraping Download link: https://github.com/moonlight-lupin/agent-skills/archive/main.zip#website-scraping Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.