skills.rest
leobrival
leobrivalCOMMUNITY
Explore all skills by this author →
GitHub

web-crawler

Crawl websites and generate JSON, Markdown, and HTML reports.

Updated Aug 27, 2026
Search Tags:#reporting#Rust#robots.txt#crawl#web crawler#LLM-ready
One-click install
npx skills add https://github.com/leobrival/serum-plugins-official --skill web-crawler
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-crawler
Source: https://github.com/leobrival/serum-plugins-official/tree/main/plugins/web-crawler/skills/web-crawler
Command: npx skills add https://github.com/leobrival/serum-plugins-official --skill web-crawler

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides a high-performance web crawler that maps site structure, discovers sitemaps, extracts links, and produces multi-format reports for quick analysis and integration with LLM/RAG pipelines.

Core Features & Use Cases

  • Performance: 60+ pages/sec crawl speed, minimal startup time, and low memory usage in Rust.
  • Robustness & Compliance: Robots.txt handling and rate-limited crawling to respect site policies.
  • Output & Intelligence: Generates JSON, Markdown (LLM-ready), HTML reports, and CSV/Links outputs; supports content filtering and stealth mode for realistic requests.
  • Use Case: Quickly map a new site to understand its structure, extract internal and external links, and prepare data for AI indexing.

Quick Start

To begin, run the prebuilt binary located at ~/.claude/skills/web-crawler/bin/rcrawler with a target URL, for example: ~/.claude/skills/web-crawler/bin/rcrawler https://example.com -w 20 -d 2

Frequently Asked Questions about web-crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a website and export the content as LLM-ready Markdown?▼

Crawling a website to export LLM-ready Markdown involves mapping site structure and extracting links into structured formats. This crawler outputs Markdown reports specifically formatted for immediate integration into RAG pipelines.

What is the best way to map large domains for SEO analysis and content inventory?▼

Mapping large domains for SEO analysis requires a high-performance crawler capable of 60+ pages per second. This Rust-based tool discovers sitemaps, respects robots.txt compliance, and outputs structured JSON and CSV reports for content inventory.

Can I use a web crawler that respects robots.txt and includes stealth mode?▼

Yes, this web crawler respects robots.txt and includes stealth mode. It performs rate-limited crawling to adhere to site policies while using stealth techniques to ensure requests appear realistic during site audits.

How do I configure crawl depth and performance controls for site auditing?▼

Configuring crawl depth and performance controls for site auditing is done via command-line arguments to the crawler binary. You specify worker counts and depth limits to directly manage crawl speed, scope, and resource usage.

Does the web crawler support JSON and CSV outputs alongside Markdown?▼

Yes, the web crawler supports JSON and CSV outputs alongside Markdown. It produces multi-format reports including JSON, HTML, and CSV/Links files to satisfy diverse site auditing, SEO analysis, and content filtering requirements.

Related Skills

Explore More Software Engineering Skills→
ekatasingh1107ekatasingh1107
2

seo-auditor

Audit technical, on-page, and off-page SEO signals and return prioritized fixes.

Community
Advanced
Postpartum-genushyacinthus29Postpartum-genushyacinthus29
8

dotnet-managedcode-markitdown

Convert documents to deterministic Markdown in .NET applications.

Community
Intermediate
mindmorassmindmorass
2

site-crawler

Crawl websites with robots.txt compliance and extract structured markdown content.

Community
Advanced
abdullah1854abdullah1854
1

crawl-cli

Crawl websites and extract content into Markdown, JSON, or text.

Community
Basic