web-index

Builds local link routing indexes for documentation sites from llms.txt and sitemap.xml.

2|Updated Jun 8, 2026
One-click install
npx skills add https://github.com/HACK-WU/skills --skill web-index-hack-wu
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: web-index
Source: https://github.com/HACK-WU/skills/tree/main/skills/web-index
Command: npx skills add https://github.com/HACK-WU/skills --skill web-index-hack-wu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? When an AI repeatedly consults the same documentation site, it wastes effort rediscovering which page to open every session. This Skill captures that one-time exploration cost as a local routing table mapping tasks ("I want to...") to exact URLs with anchors, so future lookups hit the right page with a single fetch. ## Core Features & Use Cases - Site map extraction: Fetches candidate links via a four-level priority chain (llms.txt > sitemap.xml with recursive index support > manual sitemap > navigation fallback) using a zero-dependency Python script. - Task-oriented routing tables: Produces index.md plus topics/ partition files under .web-index/{site}/, where every entry maps a user task to a URL, anchor, and keywords—never mirrors page content. - Dual-mode operation: Consumption mode queries existing indexes and fetches directly; build mode runs the full six-stage pipeline (admission check, scope confirmation, map fetch, purpose annotation, persistence, delivery) only for unindexed sites. - Use Case: You tell the AI "I'll be working from the MinIO docs repeatedly—index this site." It crawls the sitemap, annotates 87 links with task descriptions, and writes .web-index/minio/. Two weeks later, a question about TLS certificates is answered by reading one topics file and fetching one anchored URL. ## Quick Start Ask the AI to build a web index for a documentation site you will reference repeatedly, for example: "Create a web index for https://min.io/docs/minio/linux since I'll consult it throughout this project."

Frequently Asked Questions about web-index

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a local index of a documentation website for an AI agent?

Run the fetch_site_map.py script against the site's URL with a scope prefix, then annotate each link with a task-oriented purpose and write routing tables into .web-index/{site}/. The script automatically tries llms.txt first, then sitemap.xml, before falling back to manual navigation extraction.

What is the difference between llms.txt and sitemap.xml for site indexing?

llms.txt is a Markdown file sites publish for LLMs, containing links with human-written descriptions and section headings, making it the highest-quality source. sitemap.xml is a standard XML URL list without descriptions, so purposes must be inferred from URL paths and navigation text.

Does the web index script require any Python packages to install?

No installation is needed. The script uses only the Python 3 standard library (urllib.request, xml.etree, re) and runs directly with python3 on version 3.8 or later, with no virtual environment or pip dependencies required.

When should I not build a web index for a site?

Skip indexing for one-time lookups, sites with only 1-3 pages, or sites behind logins and paywalls. Building an index costs a full exploration pass, so it only pays off when the site will be consulted at least twice or across sessions.

Why does the script exit with code 4 after fetching a sitemap?

Exit code 4 means the sitemap was found but filtering produced zero entries, almost always because the --scope prefix is wrong. Remove the scope or use a shorter path prefix and rerun the command once.

Does the web index stay up to date automatically?

No, indexes are one-time snapshots with no auto-refresh or link validation. When links go stale or the site is redesigned, you rerun the build process to overwrite the entire site index.