llms-txt-from-website

Generate llms.txt and llms-full.txt manifests from website documentation sources.

3|Updated Jan 14, 2014
One-click install
npx skills add https://github.com/prateek/dotfiles --skill llms-txt-from-website
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: llms-txt-from-website
Source: https://github.com/prateek/dotfiles/tree/main/.agents/skills/llms-txt-from-website
Command: npx skills add https://github.com/prateek/dotfiles --skill llms-txt-from-website

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Generate llms.txt and llms-full.txt for docs sites from a URL by discovering documentation sources, reusing existing manifests when possible, and assembling a curated set of sources for reliable downstream use.

Core Features & Use Cases

  • Reuse existing llms.txt/llms-full.txt when available to minimize redundant work.
  • Discover documentation sources by following "Edit this page" / GitHub links, cloning the docs repo, and extracting markdown sources.
  • Fall back to sitemap.xml or crawling when direct repo discovery is not possible, then generate a high-signal manifest.
  • Produce a structured llms-full.txt by concatenating and attaching source context for downstream models.

Quick Start

Provide a website URL and run the skill to generate llms.txt and llms-full.txt.

Frequently Asked Questions about llms-txt-from-website

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate llms.txt from a website?

Yes, it automatically detects and reuses an existing /llms.txt manifest on the target website to minimize redundant work. If an existing llms.txt is found during the website documentation crawling process, the skill incorporates it directly rather than regenerating the content from scratch.

Can I extract markdown documentation from a GitHub repo link?

When direct repo discovery fails, the skill falls back to parsing sitemap.xml or crawling the website to generate a high-signal manifest. This ensures website documentation can still be converted into a structured llms.txt format even without explicit GitHub source links.

What is the difference between llms.txt and llms-full.txt?

The llms.txt file is a curated manifest of documentation sources, while llms-full.txt is a structured concatenation of those sources with attached context. Generating llms-full.txt produces a comprehensive markdown file designed to provide maximum context for downstream LLM processing.

Do I need a sitemap.xml to crawl website docs?

No, a sitemap.xml is not required because the skill first attempts to discover the documentation source repo via GitHub links. Sitemap.xml parsing is only used as a fallback mechanism when direct repository discovery is not possible during the website crawling process.