polite-fetching
CommunityLawful, durable web scraping
Authorssmleo
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill addresses the challenge of performing lawful, durable web scraping, ensuring compliance with website terms of service and robots.txt policies.
Core Features & Use Cases
- Conditional GETs: Executes GET requests based on ToS/robots.txt compliance.
- Min-Interval + Concurrency Control: Implements minimum interval and single concurrency for polite network interactions.
- User-Agent Identification: Utilizes a unique, identifiable User-Agent string with contact information.
- Logging: Records politeness incidents to a secure log (SAF).
- Anti-Pattern Avoidance: Protects against common scraping anti-patterns like parallel hammering and ignoring HTTP 429 responses.
- Learnings: Provides historical insights and best practices for scraping, including handling 403 errors and using fallback strategies.
Quick Start
Activate the polite-fetching skill when initiating a network fetch of a source to ensure compliance with website policies.
Dependency Matrix
Required Modules
None requiredComponents
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: polite-fetching Download link: https://github.com/ssmleo/govfolio/archive/main.zip#polite-fetching Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.