web-to-markdown

Convert URLs to Markdown using trafilatura and shot-scraper.

14|Updated May 28, 2024
One-click install
npx skills add https://github.com/cameronraysmith/vanixiets --skill web-to-markdown-cameronraysmith
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: web-to-markdown
Source: https://github.com/cameronraysmith/vanixiets/tree/main/modules/home/ai/skills/src/core/web-to-markdown
Command: npx skills add https://github.com/cameronraysmith/vanixiets --skill web-to-markdown-cameronraysmith

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trafilatura, shot-scraper, and includes scripts (resource) components.

What problem does it solve?

This Skill automates the process of converting webpages into clean, usable Markdown format, saving users time and effort in content capture and repurposing.

Core Features & Use Cases

  • Content Extraction: Downloads and extracts the main content from a given URL.
  • Format Conversion: Converts extracted HTML into well-formatted Markdown.
  • Handles Dynamic Content: Can process JavaScript-rendered pages using shot-scraper.
  • Use Case: When a user wants to save an article from a blog or news site for offline reading or later reference, this Skill can capture it as a clean Markdown file.

Quick Start

Convert the webpage at https://example.com/article to markdown and save it as output.md.

Frequently Asked Questions about web-to-markdown

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a webpage to Markdown format?

To convert a webpage to Markdown, this Skill downloads the URL, extracts main content from rendered HTML, and outputs a clean Markdown file. It handles both static and JavaScript-rendered pages for accurate content capture.

Can I extract main content from JavaScript-rendered pages into Markdown?

Yes, you can extract content from JavaScript-rendered pages into Markdown using shot-scraper for dynamic page rendering. Trafipatura then extracts the main content from the rendered HTML for conversion.

What is the best way to save web articles for offline reading as text files?

The best way to save web articles for offline reading is using a webpage to Markdown converter. It downloads the URL and extracts the main content into a well-formatted Markdown file for later reference.

Does this URL to Markdown converter work with dynamic content?

Yes, this URL to Markdown converter works with dynamic content by utilizing shot-scraper to render JavaScript pages. It then uses trafilatura to extract the main content and convert it into Markdown format.

How do I specify the output path when converting a URL to Markdown?

You can specify the output path when converting a URL to Markdown by providing the desired file path during the conversion process. The Skill then saves the extracted content as a Markdown file at that location.

What are the limitations of using trafilatura for web content extraction?

While trafilatura is effective for web content extraction, it primarily handles static HTML. For JavaScript-rendered pages, the Skill relies on shot-scraper to render the dynamic content before trafilatura extracts it.