Enforce robots.txt and terms-of-service compliance during web scraping.
npx skills add https://github.com/ssmleo/govfolio --skill polite-fetching
Please help me install this Agent Skill. Skill: polite-fetching Source: https://github.com/ssmleo/govfolio/tree/main/agents/skills/polite-fetching Command: npx skills add https://github.com/ssmleo/govfolio --skill polite-fetching
This Skill addresses the challenge of performing lawful, durable web scraping, ensuring compliance with website terms of service and robots.txt policies.
Activate the polite-fetching skill when initiating a network fetch of a source to ensure compliance with website policies.
High-intent search queries and answers about installing and using this skill.
To ensure web scraping complies with robots.txt and terms of service, execute conditional GETs based on site policies, use a unique User-Agent string with contact information, and maintain single concurrency with minimum intervals between requests.
Common web scraping anti-patterns include parallel hammering and ignoring HTTP 429 responses. Avoid these by implementing minimum interval pacing, single concurrency control, and fallback strategies while logging politeness incidents.
Handle HTTP 403 and 429 errors during data retrieval by applying fallback strategies, respecting retry-after directives from 429 responses, and ensuring your User-Agent identification includes clear contact information.
Polite web scraping for large scale data retrieval requires conditional GETs, strict min-interval pacing, and single concurrency control to prevent parallel hammering and ensure durable, lawful network interaction.
A User-Agent string for compliant web scraping should include unique identification and contact information to maintain transparency and allow server administrators to reach you regarding network interaction policies.
Your web scraper may get blocked despite following robots.txt if it exhibits anti-patterns like parallel hammering or ignoring HTTP 429 responses, violating broader terms of service through impolite network interaction.
MaryHu-YRScans website data to generate GDPR and CCPA compliant privacy policies.
ovachieverQuery the Metabolomics Workbench REST API for metabolite structures and study metadata.
wyre-technologyConfigures HTTP, SNMP, and TCP monitors for network devices via Atera.
Sim-hub22Integrate Resend email service with Convex for asynchronous email delivery.
