llmstxt

An llms.txt file provides LLM context for web documentation by standardizing how agents discover and fetch LLM-ready information from a domain. Agents retrieve this file before parsing HTML to obtain curated, markdown-friendly context and reduce token waste.

To make website documentation LLM-friendly using markdown, author an llms.txt file at your domain root with an H1, summary blockquote, and optional file lists. Agents fetch this context first, using linked markdown pages to minimize HTML parsing.

Fetch expanded context files by requesting /llms-ctx.txt or /llms-ctx-full.txt from the target domain. These consolidated files reduce token waste and fragmentation by providing larger blocks of LLM-ready context without needing multiple page fetches.

The best way to map HTML URLs to clean markdown endpoints is to implement .md fallbacks like page.html.md or index.html.md. Agents attempt these markdown renderings to obtain cleaner input and avoid parsing complex HTML structures.

You should not rely on ad-hoc page scraping for LLM context discovery when a website provides standardized llms.txt or llms-ctx.txt files. Fetching these dedicated context files first ensures consistent, curated input and minimizes token waste.