Fetch clean readable web content from URLs via llms.txt and markdown fallback.
npx skills add https://github.com/JKHeadley/instar --skill smart-web-fetch-jkheadley
Please help me install this Agent Skill. Skill: smart-web-fetch Source: https://github.com/JKHeadley/instar/tree/main/skills/smart-web-fetch Command: npx skills add https://github.com/JKHeadley/instar --skill smart-web-fetch-jkheadley
This Skill reduces the token and cleanup cost of reading web pages by finding the cleanest available content source first instead of passing raw HTML into context.
Ask the assistant to fetch a URL with smart-web-fetch and return the cleanest readable version of the page.
High-intent search queries and answers about installing and using this skill.
To extract clean text from web pages for LLM summarization, fetch the URL to strip noisy HTML markup and retrieve only the readable content. This approach uses llms.txt discovery and markdown fallback to ensure token-efficient retrieval.
The best way to reduce token usage when fetching web content is to retrieve markdown-first sources instead of raw HTML. By checking for llms.txt files and stripping noisy markup, agents receive only the essential readable text.
Llms.txt discovery works by checking a domain for llms.txt or llms-full.txt files to retrieve curated, agent-friendly content. If unavailable, the fetch operation falls back to a markdown-oriented reader path before attempting raw HTML parsing.
Yes, you can use markdown extraction for documentation and product sites. The retrieval process applies a markdown-oriented reader path that is ideal for blogs and product pages, ensuring fast and clean content access.
Parsing raw HTML increases token costs because it includes noisy markup, scripts, and structural tags that consume context window space. Stripping this noise and retrieving markdown-first content provides token-efficient extraction for faster summarization.
You should not rely solely on markdown fallback for web fetching when a site explicitly provides an llms.txt file. Llms.txt discovery provides curated, agent-friendly content directly, making it the optimal explicit source selection before any fallback parsing occurs.
microsoftGenerate llms.txt and llms-full.txt discovery files from wiki pages.
dongzhuoyaoIndex and retrieve untruncated sections from long arXiv LaTeX papers.
0xBigBossFetches web pages and converts main content into clean markdown using CSS selectors.
zhayujieFetch web page HTML and extract title and readable text.
