scraper-review

Review web scrapers for blocking risks, silent breakage, and data quality gaps.

15|3|Updated Jul 9, 2026
One-click install
npx skills add https://github.com/thefear078/cursor-kit-for-ai --skill scraper-review-thefear078
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scraper-review
Source: https://github.com/thefear078/cursor-kit-for-ai/tree/main/plugins/automation/skills/scraper-review
Command: npx skills add https://github.com/thefear078/cursor-kit-for-ai --skill scraper-review-thefear078

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Web scrapers fail in three predictable ways: they get blocked by anti-bot systems, break silently when sites change their markup, and produce dirty data that pollutes downstream pipelines. This Skill audits scraper code against all three failure modes before they cause production incidents. ## Core Features & Use Cases - Reliability Audit: Checks fetch-layer concerns like timeouts, exponential backoff, 429/Retry-After handling, and session reuse, plus parse-layer risks like fragile generated CSS selectors versus stable data-* anchors. - Data Quality Contracts: Enforces validated extraction schemas (e.g., Pydantic models), quarantine of parse failures with raw HTML snapshots, and canary checks that fail loudly when expected element counts drop to zero. - Blocking & Politeness Review: Evaluates rate limiting, robots.txt compliance, proxy rotation escalation, and whether an official API or JSON endpoint should replace scraping entirely. - Use Case: A scraper that worked for months suddenly returns empty product lists. Run this review to detect the blocked-but-200 response, identify the changed selectors, and add validation so future breakage alerts instead of silently writing nulls. ## Quick Start Review my Playwright scraper for the product listing pages and tell me where it will get blocked or break silently.

Frequently Asked Questions about scraper-review

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I review a web scraper for reliability?▼

Review the fetch layer for timeouts, exponential backoff with jitter, and 429/Retry-After handling, then the parse layer for stable selectors and per-field validation. Finally check the output layer for upserts on natural keys and run summaries that alert on volume anomalies.

When should I use Playwright instead of plain HTTP requests for scraping?▼

Use Playwright only when content is JavaScript-rendered and unavailable in the initial HTML. Plain httpx plus a parser is roughly 10x faster and less fragile for static pages, so check for a JSON blob in a script tag before launching a browser.

Why does my scraper return empty results without errors?▼

Silent empty results usually mean the site changed its markup or you are blocked but receiving a 200 response. Add a canary check that fails loudly when parsed element counts drop below the expected baseline, and save raw HTML snapshots for debugging.

How do I avoid getting blocked while scraping a website?▼

Start with the lowest escalation rung: slow down with 1-5s delays plus jitter, cap concurrency, reuse sessions, and send realistic headers. Only rotate proxies or switch to a real browser if those fail, and prefer an official API whenever one exists.

What data quality checks should a scraper have?▼

Validate every extracted field against a schema such as a Pydantic model, treat missing required fields as record-level errors, and quarantine bad rows with their raw HTML. Emit a run summary of pages fetched, parsed, failed, and rows written to catch anomalies.