skills.rest
xberg-io
xberg-ioOFFICIAL
Explore all skills by this author →
Website𝕏@kreuzberg_devGitHub

crawling-a-site

Crawl websites with depth, page limits, concurrency, and robots.txt compliance.

26|2|Updated Jun 8, 2026
Search Tags:#SEO#web crawling#content auditing#site scraping#Markdown output
One-click install
npx skills add https://github.com/xberg-io/plugins --skill crawling-a-site
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: crawling-a-site
Source: https://github.com/xberg-io/plugins/tree/main/plugins/crawlberg/.cursor-plugin/skills/crawling-a-site
Command: npx skills add https://github.com/xberg-io/plugins --skill crawling-a-site

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables users to systematically crawl a website, following links and capturing every reachable page, which is useful for web scraping, SEO, and content auditing.

Core Features & Use Cases

  • Deep Crawl: Follows links across a domain with specified depth and maximum page count.
  • Concurrency and Rate Limiting: Manages the number of concurrent requests and limits the rate of requests to avoid overloading servers.
  • Respect Robots.txt: Honours the website's robots.txt file to comply with crawling policies.
  • Output Selection: Outputs the results in Markdown format for easy reading and analysis.
  • Use Case: Ideal for quickly gathering content from a website or building a sitemap for SEO purposes.

Quick Start

Use the crawlberg crawl command to start crawling 'https://example.com' with a depth of 3, capturing up to 200 pages, at a rate of 8 concurrent requests.

Frequently Asked Questions about crawling-a-site

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a website and save all pages as Markdown?▼

To crawl a website and capture pages as Markdown, you specify a target URL, crawl depth, and maximum page count. The tool follows links across the domain and outputs the gathered content in Markdown format for easy reading.

Does this web scraper respect robots.txt and rate limiting?▼

Yes, the web scraper respects robots.txt to comply with crawling policies. It also features concurrency controls and rate limiting to manage request volume, preventing server overload during deep crawls.

Can I limit the depth and number of pages when scraping a site?▼

Yes, you can limit scraping depth and the total number of pages captured. Specifying these parameters controls the crawl scope, making it efficient for targeted content auditing or large site mapping without excessive requests.

What is the best way to extract site content for SEO auditing?▼

The best way to extract site content for SEO auditing is a deep crawl that follows links across a domain. Capturing reachable pages as Markdown provides a standardized format to analyze content structure and gather SEO data.

Why does my site crawl stop before reaching all pages?▼

A site crawl stops before reaching all pages when it hits the specified maximum page limit or depth threshold. These constraints are configured to manage concurrency and rate limiting, ensuring the crawl remains efficient.

Related Skills

Explore More Software Engineering Skills→
mindmorassmindmorass
2

site-crawler

Crawl websites with robots.txt compliance and extract structured markdown content.

Community
Advanced
boringdataboringdata
2

content-ingestion

Map sitemaps, fetch selected URLs, and extract content as markdown with metadata.

Community
Intermediate
daanteijema-beepdaanteijema-beep

firecrawl-crawl

Crawl reachable URLs from a starting page and extract content in bulk.

Community
Intermediate
CrumbgrabberCrumbgrabber
3

webapp-testing

Automate local web application testing with Python Playwright scripts.

Community
Intermediate