crawling-a-site
OfficialEfficiently crawl a website and capture all reachable pages as Markdown.
Authorxberg-io
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill enables users to systematically crawl a website, following links and capturing every reachable page, which is useful for web scraping, SEO, and content auditing.
Core Features & Use Cases
- Deep Crawl: Follows links across a domain with specified depth and maximum page count.
- Concurrency and Rate Limiting: Manages the number of concurrent requests and limits the rate of requests to avoid overloading servers.
- Respect Robots.txt: Honours the website's robots.txt file to comply with crawling policies.
- Output Selection: Outputs the results in Markdown format for easy reading and analysis.
- Use Case: Ideal for quickly gathering content from a website or building a sitemap for SEO purposes.
Quick Start
Use the crawlberg crawl command to start crawling 'https://example.com' with a depth of 3, capturing up to 200 pages, at a rate of 8 concurrent requests.
Dependency Matrix
Required Modules
requestsbeautifulsoup4lxml
Components
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: crawling-a-site Download link: https://github.com/xberg-io/plugins/archive/main.zip#crawling-a-site Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.