web-scraper

Fetch HTML from a URL and convert it to clean Markdown.

1|Updated May 12, 2026
One-click install
npx skills add https://github.com/cocyuhao/my-ai-skills-library --skill web-scraper-cocyuhao
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/cocyuhao/my-ai-skills-library/tree/main/web-scraper
Command: npx skills add https://github.com/cocyuhao/my-ai-skills-library --skill web-scraper-cocyuhao

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, and includes scripts (resource) components.

What problem does it solve?

Fetching and reading web content often requires manual copy-paste and formatting. This Skill fetches a URL, strips noise, and renders the article in clean markdown for easy reading and note-taking.

Core Features & Use Cases

  • Fetch a single URL and convert HTML to readable markdown.
  • Remove navigation, ads, and other clutter while preserving headings, lists, and code blocks.
  • Use in workflows to wire URLs into note-taking, QA, summaries, or content pipelines.
  • Optional raw HTML output via a flag for debugging or specialized processing.

Quick Start

Fetch a page and convert it to markdown by running the fetch_url script with the target URL.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert HTML from a web page to clean markdown?

To convert HTML to markdown, fetch the target URL to retrieve the web page content, strip away navigation and ads, and render the remaining text as clean markdown. This preserves headings, lists, and code blocks for easy reading and note-taking.

What is the best way to scrape a single URL and remove web page clutter?

Scraping a single URL to remove web page clutter involves fetching the HTML content and stripping noise like ads and navigation. The process outputs a readable markdown format that retains essential structural elements such as headings and lists.

Does the web to markdown conversion work if the requests library is unavailable?

Yes, web to markdown conversion works if the requests library is unavailable. The script uses a urllib fallback mechanism to fetch HTML content, ensuring reliable URL retrieval and markdown conversion across different environments.

Can I get raw HTML output instead of markdown for debugging?

Yes, you can get raw HTML output instead of markdown for debugging. By using an optional flag, the script returns the unprocessed HTML content fetched from the URL, allowing for specialized processing or issue diagnosis.

How do I extract article text from a URL for a content pipeline?

To extract article text from a URL for a content pipeline, fetch the web page and convert the HTML into clean markdown. This provides structured text suitable for wiring into automated workflows like summaries, QA, or note-taking.

What are the limitations of fetching web pages and converting to markdown?

Limitations of fetching web pages and converting to markdown include reliance on basic error handling and support for single URL reads only. It extracts static HTML content and does not process dynamic JavaScript-rendered page elements.