defuddle

Extracting article body text from web pages removes ads, navigation, headers, footers, and boilerplate to output clean markdown. This reduces token waste by 40–60% and prepares content for wiki ingestion.

Cleaning cluttered HTML into markdown for a knowledge base is done by stripping cookie banners and related-section clutter, leaving only the relevant article body. This ensures efficient knowledge base ingestion without surrounding UI content.

Defuddle works with URL-based ingestion workflows for news, blogs, and documentation pages. It automatically extracts meaningful content from URLs and outputs cleaned markdown to stdout suitable for piping into an ingestion directory.

To reduce token usage when ingesting web articles, the extraction process keeps only relevant content and discards boilerplate. Typical web article ingestion is reduced by 40–60% tokens, improving readability and saving processing costs.

You can pipe cleaned markdown output directly into a wiki ingestion directory. The tool uses defuddle-cli with stdout suitable for piping, allowing you to run it on a URL and save the output into your raw ingestion folder automatically.

Web page content extraction for markdown cleaning should be avoided when pages lack a distinct article body or when surrounding UI content is the primary target. It is designed for pages where boilerplate dominates and meaningful article extraction is needed.