skills.rest
alirezarezvani
alirezarezvaniCOMMUNITY
Explore all skills by this author →
Website𝕏@RezaRezvaniBlnGitHub

universal-scraping-architect

Build validated web scraping pipelines using Firecrawl or local Python extraction.

25.3k|3.6k|Updated Oct 19, 2025
Search Tags:#Pandas#Validation#Firecrawl#crawling#BeautifulSoup#Data extraction#Web scraping
One-click install
npx skills add https://github.com/alirezarezvani/claude-skills --skill universal-scraping-architect
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: universal-scraping-architect
Source: https://github.com/alirezarezvani/claude-skills/tree/main/engineering/universal-scraping-architect/skills/universal-scraping-architect
Command: npx skills add https://github.com/alirezarezvani/claude-skills --skill universal-scraping-architect

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires firecrawl, pandas, requests, beautifulsoup4, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Web scraping projects often produce brittle one-off scripts that break on layout changes, leak API keys, ignore robots.txt, and ship unvalidated data. This Skill designs complete extraction pipelines with intelligent routing between API-driven and local approaches, enforced validation, and token-budget tracking.

Core Features & Use Cases

  • Three Extraction Modes: Route between Firecrawl API (dynamic JS sites, bulk crawling), local Python with BeautifulSoup and pandas (static pages, private files), or a hybrid pipeline combining both.
  • Mandatory Validation Gate: Run the stdlib-only validate_extraction.py script on every JSON output before delivery, checking for empty results and malformed structures.
  • Proactive Safety Triggers: Automatically flags hardcoded API keys, private data sent to external APIs, and missing pagination logic.
  • Use Case: You need to extract pricing tables from 50 product pages. The Skill routes to Firecrawl for discovery, checkpoints progress across pages, validates the JSON output, and delivers a clean CSV with a summary log.

Quick Start

Ask the agent to scrape a target URL and deliver a validated CSV, mentioning whether the data is public or sensitive so it can route between Firecrawl and local Python extraction.

Frequently Asked Questions about universal-scraping-architect

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a JavaScript-heavy website with Python?▼

Use the Firecrawl API mode for JS-heavy single-page applications, since it renders dynamic content server-side and returns clean markdown or structured JSON. Load your key from the FIRECRAWL_API_KEY environment variable and call scrape_url with the formats you need.

Firecrawl vs BeautifulSoup: which should I use for web scraping?▼

Use Firecrawl for public URLs, dynamic JS sites, search-first discovery, and bulk domain crawling. Use BeautifulSoup with requests and pandas for static HTML pages, local files like PDF or Excel, and private data that should never leave your machine.

How do I extract an HTML table into a pandas DataFrame?▼

Locate the table with BeautifulSoup using a stable id or data attribute selector, then pass it to pd.read_html to get a DataFrame. Normalize column names to snake_case, drop empty rows, and validate required columns before saving to CSV.

Does this scraping approach respect robots.txt?▼

Yes, both runner templates check robots.txt using the stdlib urllib.robotparser module before fetching and abort if the path is disallowed. They also send an identifying User-Agent and use exponential backoff with capped retries.

Why does my scraped JSON output fail validation?▼

The validate_extraction.py script exits with an error when the JSON is malformed and a warning when it parses but is empty. Fix the extraction logic or selectors and re-run until the validator returns status ok before delivering data.

When should I avoid sending scraped data to an external API?▼

Avoid external APIs when the source data is private or sensitive, since sending local files to a third-party service creates privacy and compliance risk. Use the local Python mode with BeautifulSoup and pandas so data never leaves your machine.

Related Skills

Explore More Software Engineering Skills→
Kastalien-ResearchKastalien-Research

summarize

Summarize or extract text from URLs, podcasts, and local files via CLI.

Official
Intermediate
Wesley1600Wesley1600

data-analysis

Analyze CSV, JSON, and Parquet datasets with exploratory analysis and visualizations.

Community
Advanced
mutsumi-yamamotomutsumi-yamamoto

data-clean

Cleanse, normalize, and deduplicate datasets for ETL pipelines.

Community
Advanced
bifrost-proxybifrost-proxy
112

site-cookie-login

Automate website logins with Puppeteer and save cookies to an environment variable file.

Community
Advanced