crawl-sites
OfficialEffortlessly extract structured content from websites at scale.
Authorthe-agency-ai
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill automates the process of crawling websites and extracting structured content, reducing manual browsing efforts and enabling comprehensive data collection.
Core Features & Use Cases
- Site Content Extraction: Crawl specified sites and retrieve HTML content based on URL patterns.
- Customizable Scope: Target specific sites or entire configurations for targeted data harvesting.
- Use Case: Imagine monitoring multiple documentation sites for updates. Use this Skill to crawl them regularly and collect new articles or changes automatically.
Quick Start
Use the crawl-sites skill to scan your configured websites and gather all relevant content into your output folder.
Dependency Matrix
Required Modules
None requiredComponents
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: crawl-sites Download link: https://github.com/the-agency-ai/the-agency/archive/main.zip#crawl-sites Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.