skills.rest
armoin2018
armoin2018COMMUNITY
Explore all skills by this author →
WebsiteGitHub

ailey-tools-web-crawl

Crawl websites with depth control, filtering, and authentication.

4|Updated Jan 2, 2026
Search Tags:#automation#authentication#puppeteer#sitemap#crawler#robots.txt#web-crawler
One-click install
npx skills add https://github.com/armoin2018/ai-command-center --skill ailey-tools-web-crawl
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ailey-tools-web-crawl
Source: https://github.com/armoin2018/ai-command-center/tree/main/plugins/aicc-web-crawl/skills/ailey-tools-web-crawl
Command: npx skills add https://github.com/armoin2018/ai-command-center --skill ailey-tools-web-crawl

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates comprehensive website crawling and data collection with depth control, filtering, and authentication to streamline site archiving, sitemap generation, and data extraction.

Core Features & Use Cases

  • Depth-controlled crawling, pattern filtering, cross-site navigation, and file-type filtering for efficient site traversal.
  • Parallelization and rate limiting to optimize performance while respecting server load.
  • Authentication support (basic, bearer, form-based, and cookies) and robots.txt compliance for secure, scalable crawling across domains.
  • Real-world use cases include building sitemap archives, performing content migrations, and collecting pages and media for SEO analysis or research.

Quick Start

Configure START_URL and OUTPUT_DIR, then run the crawl command to begin collecting pages and metadata.

Frequently Asked Questions about ailey-tools-web-crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a website with depth control and authentication?▼

Web crawling for SEO analysis involves mapping site structure and collecting metadata. This crawler extracts pages and media files while respecting configurable depth limits and rate limiting to optimize performance without overloading servers.

Can I download binary files and filter by file type during a site crawl?▼

The crawler respects robots.txt compliance during site traversal to ensure ethical scraping. It applies configurable rules for cross-site navigation and pattern filtering, allowing you to safely map domains without violating server restrictions.

Does this web crawler support parallelization and rate limiting?▼

To generate a sitemap and archive a site, configure the start URL and output directory, then run the crawl command. The crawler collects pages and metadata automatically, producing outputs suitable for sitemap generation and content migration.

What is the best way to automate website crawling for content migration?▼

Yes, you can perform cross-site browsing and multi-domain crawling. The crawler supports cross-site navigation with configurable rules, enabling comprehensive research and data collection across multiple domains during a single automated session.

Related Skills

Explore More Software Engineering Skills→
7a336e6e7a336e6e
1

managing-sessions-tokens

Manage JWT authentication sessions with rotating refresh tokens and revocation.

Community
Advanced
abdullah1854abdullah1854
1

crawl-cli

Crawl websites and extract content into Markdown, JSON, or text.

Community
Basic
leafcoderleafcoder
2

Browser Automation Expert

Automate browser interactions and web testing with Playwright and Puppeteer.

Community
Intermediate
andersnygaardandersnygaard
5

rule-making-skill

Analyze a directory and generate .claude/rules/ markdown files for Claude Code.

Community
Intermediate