Scrape public web pages with robots.txt checks and rate limiting.
npx skills add https://github.com/Wike-CHI/acquisition-agent --skill scrape-wike-chi
Please help me install this Agent Skill. Skill: scrape Source: https://github.com/Wike-CHI/acquisition-agent/tree/main/skills/scrape Command: npx skills add https://github.com/Wike-CHI/acquisition-agent --skill scrape-wike-chi
Prevents unlawful or harmful web scraping by enforcing robots.txt checks, terms-of-service review, rate limiting, and privacy-aware data handling so teams can collect public web data without exposing themselves to legal or reputational risk.
Check robots.txt for the target domain, create a polite HTTP session with a contact email, and fetch pages with at least a 2 second delay while handling 429 responses and exponential backoff.
High-intent search queries and answers about installing and using this skill.
Ethical web scraping requires parsing robots.txt before fetching, minimizing PII collection, and maintaining audit trails for GDPR and CCPA compliance. This skill automates those compliance checks and applies polite rate limiting to reduce legal and reputational risk.
The best way to handle rate limiting and 429 responses is using exponential backoff with randomized jitter. This skill enforces a polite request discipline with built-in delays, backoff strategies, and session headers to ensure compliant data collection.
To set up a compliant HTTP session for price monitoring, configure session headers to include a valid contact email address. This skill establishes polite HTTP sessions and enforces at least a 2-second delay between requests to respect server limits.
No, this scraping approach is designed exclusively for publicly accessible web pages where data is not behind authentication. It targets tasks like market research and public listing aggregation while ensuring terms-of-service and robots.txt compliance.
A contact email in session headers is needed to identify your scraper and allow site administrators to reach you before blocking your IP. This convention supports transparent, ethical web scraping and is automatically enforced during compliant data collection.
You should not use automated web scraping when target site terms-of-service explicitly prohibit it, or when pages require authentication. This skill prioritizes privacy-aware data handling and automatically guides you to consult site terms before scraping protected paths.
ssmleoEnforce robots.txt and terms-of-service compliance during web scraping.
math-incFetch and aggregate Polymarket market data from public REST APIs.
mwathibenImplement data privacy compliance across GDPR, CCPA, and HIPAA.
mindspore-aiGuide compliant web scraping with Beautiful Soup, Selenium, or Scrapy.
