skills.rest
hung-phan
hung-phanCOMMUNITY
Explore all skills by this author →
GitHub

web-crawler

Design distributed web crawlers with rate limits and robots.txt compliance.

Updated Jun 5, 2026
Search Tags:#data extraction#indexing#performance optimization#web crawling#robots.txt#distribute system design
One-click install
npx skills add https://github.com/hung-phan/system-skills --skill web-crawler-hung-phan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-crawler
Source: https://github.com/hung-phan/system-skills/tree/main/skills/system-review/references/interview-templates/web-crawler
Command: npx skills add https://github.com/hung-phan/system-skills --skill web-crawler-hung-phan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill guides users in the design and operation of distributed web crawlers, helping them avoid common pitfalls and optimize for efficiency, politeness, and scalability.

Core Features & Use Cases

  • Distributed Design: Offers a blueprint for a scalable and efficient distributed crawling system with detailed architecture and component roles.
  • Politeness Policies: Provides guidelines on respecting robots.txt and rate-limiting to avoid being banned by sites.
  • Content Deduplication: Explains methods for deduplication to ensure no unnecessary crawling or indexing of duplicate content.
  • Robots.txt Handling: Discusses parsing and adhering to robots.txt rules for ethical crawling.
  • Fetcher & Storage Solutions: Describes strategies for robust fetching and efficient storage of raw content.
  • Recrawl Scheduling: Details adaptive scheduling to handle dynamic content while saving bandwidth.
  • Use Case: Use this Skill to design a scalable web crawler that indexes a large portion of the web efficiently without violating site terms.

Quick Start

Implement a web crawler that processes 1B pages per month and maintains freshness with recrawls every 30 days.

Frequently Asked Questions about web-crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design a distributed web crawling system that scales to billions of pages?▼

Design distributed web crawling systems by blueprinting scalable architecture with defined component roles to efficiently crawl billions of web pages while maintaining per-host rate limits and respecting robots.txt.

How does content deduplication work in a distributed web crawler?▼

Content deduplication in web crawling works by identifying and filtering duplicate data at both the URL and content levels, preventing unnecessary crawling and indexing to save bandwidth and storage.

What is the best way to implement per-host rate limiting and robots.txt parsing for web crawling?▼

Implement per-host rate limiting and robots.txt parsing by adhering to defined politeness policies, ensuring your web crawler respects site rules and avoids being banned during large-scale data extraction.

Can I use this approach to crawl a large portion of the web without violating site terms?▼

Yes, you can use this approach to index a large portion of the web ethically, as it strictly respects robots.txt rules and applies per-host rate limiting to ensure efficient and polite data extraction.

What strategies are used for raw content fetching and storage in distributed crawling systems?▼

Strategies for raw content fetching and storage in distributed crawling systems involve robust fetching mechanisms paired with efficient storage solutions to reliably manage large volumes of extracted web data.

Related Skills

Explore More Software Engineering Skills→
CommonCapitalCommonCapital
2

static-website-asset-paths

Convert absolute asset paths to relative in HTML for proxy compatibility.

Community
Intermediate
aliyunaliyun
17

hologres-knowledge-base

Build Hologres knowledge bases with vector and full-text indexes.

Official
Advanced
hung-phanhung-phan

distributed-job-scheduler

Guide distributed job scheduler design with leader election, retries, and idempotency.

Community
Advanced
obulaiobulai
1

obul-riveter

Scrape web pages and extract structured data via a managed API gateway.

Official
Intermediate