skills.rest
paulirish
paulirishCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

fetch-as-markdown

Fetch web pages with Playwright and convert rendered HTML to Markdown.

4.4k|1.3k|Updated Jan 23, 2012
Search Tags:#content extraction#playwright#Markdown#Chromium#headless browser#HTML conversion#Web scraping
One-click install
npx skills add https://github.com/paulirish/dotfiles --skill fetch-as-markdown
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: fetch-as-markdown
Source: https://github.com/paulirish/dotfiles/tree/main/agents/skills/fetch-as-markdown
Command: npx skills add https://github.com/paulirish/dotfiles --skill fetch-as-markdown

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires playwright, markpaste, and includes scripts (resource) components.

What problem does it solve?

Many modern web pages render content client-side with JavaScript, so a plain HTTP fetch returns incomplete HTML. This Skill renders pages in a headless browser and converts the fully loaded content into clean Markdown for reading, archiving, or feeding into other tools.

Core Features & Use Cases

  • Headless Browser Rendering: Uses Playwright's Chromium to load pages and execute client-side JavaScript before capturing content.
  • HTML to Markdown Conversion: Converts the rendered page body into Markdown using markpaste with the turndown converter.
  • Flexible Dependency Resolution: Uses a local markpaste clone when available, otherwise falls back to the published npm package.
  • Use Case: Fetch a JavaScript-heavy documentation page or article and pipe the resulting Markdown into notes, a static site, or an LLM prompt.

Quick Start

Fetch the page at a given URL and output its content as Markdown to the terminal.

Frequently Asked Questions about fetch-as-markdown

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a web page to Markdown from the command line?▼

Run the script with a URL argument: node agents/skills/fetch-as-markdown/scripts/fetch-as-markdown.ts <URL>. It renders the page in headless Chromium and prints the converted Markdown to stdout.

How to scrape JavaScript-rendered pages that curl cannot fetch?▼

Use a headless browser like Playwright's Chromium, which executes client-side JavaScript before capturing the DOM. This skill waits for the load event, then extracts the fully rendered HTML for conversion.

What is the difference between Playwright and a plain HTTP request for scraping?▼

A plain HTTP request only retrieves the initial HTML, missing content injected by JavaScript. Playwright launches a real browser engine, so dynamic content rendered client-side is included in the captured page.

Does this require a local markpaste installation?▼

No. The script first tries importing markpaste from a sibling local clone, and if that fails it falls back to the published markpaste npm package. If neither is available, it exits with an error message.

Why does the script fail with a markpaste loading error?▼

The error occurs when neither the local markpaste clone nor the npm package can be imported. Install markpaste via npm or place a clone in the expected sibling directory, then rerun the script.

Related Skills

Explore More Software Engineering Skills→
TeamWiseFlowTeamWiseFlow
8.4k

rss-reader

Convert RSS/Atom feed URLs into markdown lists of recent entries.

Official
Intermediate
kreuzberg-devkreuzberg-dev
9.2k

chunking

Split extracted text into sized chunks for LLM context windows and RAG ingestion.

Official
Intermediate
izoizo

defuddle

Convert noisy web pages into clean Obsidian-ready Markdown with frontmatter.

Community
Intermediate
isymchychisymchych

find-rss-atom-feed

Scan page HTML and endpoints to return the best RSS or Atom feed.

Community
Intermediate