Generate structured JSON crawl plans from sitemaps, llms.txt, or HTML targets.
npx skills add https://github.com/jadecli/researchers --skill crawl-plan
Please help me install this Agent Skill. Skill: crawl-plan Source: https://github.com/jadecli/researchers/tree/main/claude-code/.claude/skills/crawl-plan Command: npx skills add https://github.com/jadecli/researchers --skill crawl-plan
Planning crawl campaigns manually is time-consuming and error-prone; this skill automatically generates an organized crawl plan from sitemaps, llms.txt indices, or HTML targets.
Provide a target URL to generate a crawl plan.
High-intent search queries and answers about installing and using this skill.
The crawl plan supports input sources from sitemap XML, llms.txt indexes, or direct HTML pages. It validates these inputs with clear error messages to fit existing workflows.
A structured JSON crawl plan contains the target, total_pages, estimated_time_minutes, and grouped URLs. This payload is ready for automation and initiating new crawl campaigns.
You can prioritize pages when planning a web crawl because the skill classifies pages by type, including API references and documentation. It determines an optimal order for fast wins.
You do not need a sitemap to generate an estimated crawl time. The skill can analyze an llms.txt index or direct HTML target to produce the structured JSON plan.
The best way to automate crawl prioritization is providing a target URL to generate a crawl plan. The skill automatically classifies documentation and other page types to determine an optimal order.
ajayatwal1105-emersonAudit and generate XML sitemaps with URL health checks and index planning.
chainreactorsCrawl web targets to identify parameterized URLs, form targets, and JavaScript endpoints.
nathanvaleFetch web content and extract structured data via the Firecrawl API.
deomiarnGenerate a Markdown sitemap with page structure and pnpm commands.
