web-scraper

Fetch web page content and convert HTML to clean Markdown.

333|83|Updated Jan 9, 2026
One-click install
npx skills add https://github.com/zephyrwang6/myskill --skill web-scraper-zephyrwang6
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/zephyrwang6/myskill/tree/main/web-scraper
Command: npx skills add https://github.com/zephyrwang6/myskill --skill web-scraper-zephyrwang6

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, and includes scripts (resource) components.

What problem does it solve?

This Skill helps users quickly access and convert web page content into readable Markdown, eliminating manual copy-paste and tedious formatting when researching online articles.

Core Features & Use Cases

  • Fetch and convert HTML pages to clean Markdown, preserving structure while removing navigation, ads, and other non-content sections.
  • Support single URL fetch with adjustable timeout and output length, plus easy extension to multiple URLs.
  • Real-world use: Save long articles for offline reading, summarization, or feeding extracted content into AI workflows.

Quick Start

Run the fetch script to retrieve a URL and convert it to Markdown: python3 scripts/fetch_url.py "https://example.com/article"

Options: --timeout <seconds> (default: 30) --max-length <chars> (default: 100000) --raw Output raw HTML instead of Markdown

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a web page to markdown for offline reading?

Use the fetch script to convert HTML pages to clean markdown: run `python3 scripts/fetch_url.py "https://example.com/article"`. The Skill fetches the page, strips navigation and ads, and outputs readable markdown while preserving structure for offline access or AI workflows.

Can I extract article text from URLs without manual copy-paste?

Yes. This Skill automates HTML-to-markdown conversion, eliminating manual formatting. It fetches web page content via requests library, parses HTML, and returns clean text extraction—ideal for research, summarization, or feeding extracted content into downstream processes.

What happens if the built-in web fetch tool is unavailable or restricted?

This Skill provides an alternative fetch mechanism using Python's requests library with fallback to urllib. It handles page retrieval when standard tools are unavailable, then converts the HTML to markdown for reliable content extraction.

How do I adjust timeout and output length when fetching URLs?

Pass `--timeout <seconds>` (default: 30) and `--max-length <chars>` (default: 100000) flags to the fetch script. These options control request duration and limit markdown output size, accommodating different network conditions and content volume needs.

Can I output raw HTML instead of markdown?

Yes. Use the `--raw` flag with the fetch script to retrieve unprocessed HTML. This option supports workflows requiring raw HTML parsing or custom post-processing before markdown conversion.