Skills
.Work. You
Rest

ailey-tools-web-crawl

Community

Crawl, download, and map websites with control.

Software Engineering#automation#authentication#puppeteer#sitemap#crawler#robots.txt#web-crawler
Authorarmoin2018
Version1.0.0
Installs0

System Documentation

What problem does it solve?

Automates comprehensive website crawling and data collection with depth control, filtering, and authentication to streamline site archiving, sitemap generation, and data extraction.

Core Features & Use Cases

  • Depth-controlled crawling, pattern filtering, cross-site navigation, and file-type filtering for efficient site traversal.
  • Parallelization and rate limiting to optimize performance while respecting server load.
  • Authentication support (basic, bearer, form-based, and cookies) and robots.txt compliance for secure, scalable crawling across domains.
  • Real-world use cases include building sitemap archives, performing content migrations, and collecting pages and media for SEO analysis or research.

Quick Start

Configure START_URL and OUTPUT_DIR, then run the crawl command to begin collecting pages and metadata.

Dependency Matrix

Required Modules

None required

Components

Standard package

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: ailey-tools-web-crawl
Download link: https://github.com/armoin2018/ai-command-center/archive/main.zip#ailey-tools-web-crawl

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.