crawl

Crawls llms.txt-linked references and tracks changes for Claude Code documentation.

1|Updated Feb 17, 2026
One-click install
npx skills add https://github.com/hlibkoval/claudemd --skill crawl-hlibkoval
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: crawl
Source: https://github.com/hlibkoval/claudemd/tree/main/.claude/skills/crawl
Command: npx skills add https://github.com/hlibkoval/claudemd --skill crawl-hlibkoval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl, jq, grep, sort, comm.

What problem does it solve?

Automates maintenance of Claude Code reference documentation by crawling, downloading reference pages, and detecting changes to the claudemd plugin.

Core Features & Use Cases

  • Syncs llms.txt with skill-map.json to map new references to existing topics.
  • Downloads reference pages into references/ and cleans up orphaned files.
  • Detects and reports changes to facilitate automatic regeneration of skills.

Quick Start

Run the crawl to synchronize references and regenerate changed skills.

Frequently Asked Questions about crawl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate Claude Code documentation crawling and track reference changes?

Automate Claude Code documentation crawling by syncing llms.txt with skill-map.json, downloading reference pages, and detecting changes to trigger automatic skill regeneration.

What is the best way to sync llms.txt with skill-map.json for reference mapping?

Sync llms.txt with skill-map.json by crawling documentation sources to map new references to existing topics, ensuring your reference directory maintains accurate documentation coverage.

Do I need curl and jq to run documentation crawl automation in a Unix environment?

Yes, documentation crawl automation requires curl and jq in a Unix-like environment to fetch URLs, parse JSON responses, and manage downloaded reference files effectively.

How does orphan cleanup work when downloading reference pages into the references directory?

Orphan cleanup removes obsolete files from the references directory after crawling, ensuring that only currently valid downloaded reference pages remain and outdated documentation is eliminated.

Can I detect changes to the claudemd plugin automatically during a documentation crawl?

Yes, the crawl automation detects and reports changes to the claudemd plugin during the synchronization process, facilitating automatic regeneration of affected skills based on detected modifications.

Why does my reference crawl fail to parse JSON when updating Claude Code documentation?

Reference crawl JSON parsing failures typically occur when jq dependencies are missing or when documentation sources return invalid JSON, preventing proper synchronization between llms.txt and skill-map.json.