skills.rest
gizix
gizixCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

spider-generator

Generate Scrapy spider templates for static, dynamic, and API content.

2|2|Updated Nov 4, 2025
Search Tags:#scaffolding#generator#web-scraping#crawling#scrapy#spider
One-click install
npx skills add https://github.com/gizix/cc_projects --skill spider-generator
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spider-generator
Source: https://github.com/gizix/cc_projects/tree/main/scrapy-template/.claude/skills/spider-generator
Command: npx skills add https://github.com/gizix/cc_projects --skill spider-generator

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill guides generating well-structured Scrapy spiders, applying industry best practices for error handling, rate limiting, and data extraction patterns to accelerate data gathering with fewer bugs.

Core Features & Use Cases

  • Basic Spider templates with proper settings and data extraction patterns.
  • CrawlSpider templates with rules-based navigation for complex sites.
  • API Spider templates for JSON or API-based scraping.
  • Built-in guidance on error handling, retries, and polite crawling.

Quick Start

Use the spider-generator to scaffold a new spider named "my_spider" for "example.com", with start URL "https://example.com/products" and customize selectors as needed.

Frequently Asked Questions about spider-generator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate Scrapy spiders that follow best practices?▼

Scrapy spider generation scaffolds well-structured spiders with built-in error handling, rate limiting, and data extraction patterns. The generator creates spider classes, parse methods, and appropriate pipelines following industry standards to reduce bugs and accelerate development.

What types of spiders can I scaffold with this generator?▼

The generator supports basic spiders for static content, CrawlSpiders with rules-based navigation for complex sites, Playwright/Selenium rendering spiders for dynamic JavaScript content, and API spiders for JSON or API-backed scraping.

Can I use this to scrape dynamic websites that require JavaScript rendering?▼

Yes. The generator includes Playwright and Selenium rendering spider templates designed specifically for dynamic content, adapting templates to your target website type and rendering requirements.

How do I customize the generated spider for my target website?▼

Provide the spider name, target domain, start URL, and configure CSS or XPath selectors for data extraction. The generator adapts configurable settings and structured templates to your project requirements and website structure.

What best practices does the generator enforce?▼

The generator enforces error handling, retry logic, polite crawling with rate limiting, and structured data extraction patterns. It configures Scrapy settings to prevent overloading target servers and implements industry-standard spider architecture.

Does the generator handle API-based content scraping?▼

Yes. The generator includes API spider templates for scraping JSON or API-backed content, separate from templates designed for HTML-based websites, with parsing methods adapted to API response structures.

Related Skills

Explore More Software Engineering Skills→
jadeclijadecli
1

improve-spider

Patch failing CSS/XPath selectors in Scrapy spiders from improvement logs.

Official
Intermediate
timequitytimequity
6

spark-basics

Creates SparkSessions and performs PySpark ETL with CSV, Parquet, JSON, and Delta.

Official
Basic
fgarofalo56fgarofalo56
6

scrapy

Build custom Scrapy spiders to crawl websites and extract data.

Community
Advanced
chestnut0698chestnut0698

web-scraping

Extract data from static and dynamic web pages using Python tools.

Community
Advanced