skills.rest
paulirish
paulirishCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

distill-page

Extract distilled webpage content as Markdown using Chromium's on-device ML page annotation.

4.4k|1.3k|Updated Jan 23, 2012
Search Tags:#content extraction#playwright#Markdown#CDP#Protobuf#Chromium#Web scraping
One-click install
npx skills add https://github.com/paulirish/dotfiles --skill distill-page
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distill-page
Source: https://github.com/paulirish/dotfiles/tree/main/agents/skills/distill-page
Command: npx skills add https://github.com/paulirish/dotfiles --skill distill-page

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires playwright, @bufbuild/protobuf, @bufbuild/buf, @bufbuild/protoc-gen-es, and includes scripts (resource) components.

What problem does it solve?

Converting full webpage HTML to Markdown with traditional parsers includes navigation menus, sidebars, ads, and footers, wasting context window tokens when feeding pages to LLMs. This Skill uses Chromium's experimental Page Content Annotation ML model to isolate only the main content, reducing output size by 50-70% compared to HTML-to-Markdown converters.

Core Features & Use Cases

  • ML-Based Content Extraction: Uses the Chrome DevTools Protocol method Page.getAnnotatedPageContent to identify semantic roles like ARTICLE and MAIN, automatically stripping navigation, footers, ads, and popups.
  • Structured Markdown Output: Preserves headings, lists, tables with captions, code blocks, links, and images, while wrapping asides in <aside> tags and collapsed content in <details> blocks.
  • Flexible Input Modes: Fetch a live URL via Playwright, or decode a base64-encoded protobuf payload from stdin without launching a browser.
  • Use Case: Feed a long technical blog post to an LLM agent for summarization; instead of 21KB of noisy HTML-derived Markdown, the agent receives an 11KB distilled version containing only the article body.

Quick Start

Ask the agent to extract the main article content from a URL as clean Markdown using the distill-page script, for example by running it against a blog post URL to get a noise-free Markdown version.

Frequently Asked Questions about distill-page

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract the main content of a webpage as Markdown?▼

Run the distill-page script with a URL: `node agents/skills/distill-page/scripts/distill-page.ts <URL>`. It launches Chromium via Playwright, calls the CDP method Page.getAnnotatedPageContent, and prints distilled Markdown to stdout.

How does Chromium ML page annotation compare to Turndown for HTML to Markdown?▼

Turndown maps HTML tags directly to Markdown syntax, preserving inline code and emphasis precisely. The Chromium ML approach discards tags and uses visual layout analysis, which strips sidebars, ads, and popups far better but loses fine-grained inline formatting distinctions.

Why does Page.getAnnotatedPageContent fail with a protocol error?▼

The method fails in old headless mode and on empty pages like about:blank. Launch Chromium with the new headless mode (`--headless=new` while Playwright headless is false) plus the flags `--enable-dom-distiller` and `--enable-features=OptimizationHints,PageContentAnnotation,OptimizationGuideModelDownloading`.

Can I decode an AnnotatedPageContent protobuf without launching a browser?▼

Yes. Pass a base64-encoded protobuf payload via stdin using the `--base64-stdin` flag. This bypasses Playwright entirely and decodes the payload directly into Markdown or raw JSON.

What are the limitations of ML-based page content extraction?▼

The model does not recognize blockquotes, collapses bold and italic into a single emphasis flag, and may lose inline code formatting if the code is not visually distinct. It is an experimental Chromium feature that may change or be removed in future versions.

Related Skills

Explore More Software Engineering Skills→
breverdbidderbreverdbidder
1

firecrawl-scrape

Extract clean markdown content from URLs with JavaScript rendering support.

Community
Basic
kreuzberg-devkreuzberg-dev
9.2k

chunking

Split extracted text into sized chunks for LLM context windows and RAG ingestion.

Official
Intermediate
YanBerdinYanBerdin
2

chrome-devtools

Automate Chrome browser navigation, interaction, and performance profiling via DevTools MCP.

Community
Advanced
awfixers-stuffawfixers-stuff

web-fetch

Convert HTTP/HTTPS web pages into clean Markdown using selectors and DOM parsing.

Official
Intermediate