skills.rest
Jst-Well-Dan
Jst-Well-DanCOMMUNITY
Explore all skills by this author →
GitHub

web-fetch

Scrape web content and convert it to Markdown and PDF.

8|4|Updated Dec 15, 2025
Search Tags:#pdf#markdown#Playwright#web-scraping#web-fetch#WeChat#noise-removal
One-click install
npx skills add https://github.com/Jst-Well-Dan/Skill-Box --skill web-fetch-jst-well-dan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-fetch
Source: https://github.com/Jst-Well-Dan/Skill-Box/tree/main/content-pipeline/web-fetch
Command: npx skills add https://github.com/Jst-Well-Dan/Skill-Box --skill web-fetch-jst-well-dan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires crawl4ai, requests, beautifulsoup4, markdownify, playwright, markdown, reportlab, and includes scripts (resource) components.

What problem does it solve?

This Skill helps users scrape web content and convert it into clean Markdown or PDF formats, removing noisy elements like navigation, headers, footers, and ads. It supports general websites and WeChat (微信公众号) articles, with Playwright-based anti-bot handling for WeChat. By default, output includes both Markdown and PDF, unless you specify Markdown-only.

Core Features & Use Cases

  • Noise-free extraction using a CSS-selector cleaning strategy
  • Image preservation with alt text
  • Special handling for WeChat articles (lazy-loaded images, metadata extraction)
  • Clean Markdown output ready for translation or further processing
  • PDF conversion with clean reading style and CJK font support
  • Both MD and PDF outputs by default (configurable)

Quick Start

  • General web pages: python scripts/fetch_web_content.py <url> <output_filename.md>
  • WeChat articles: python scripts/fetch_weixin.py "<微信文章URL>" [output_filename.md]
  • Convert Markdown to PDF: python scripts/md_to_pdf.py <markdown_file>.md

Frequently Asked Questions about web-fetch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert web pages to Markdown and PDF?▼

Web-fetch scrapes web content and converts it to clean Markdown and PDF formats in a single workflow. It removes noisy elements like navigation, ads, and footers while preserving images with alt text, outputting both formats by default unless you specify Markdown-only.

Can I scrape WeChat articles with anti-bot protection?▼

Yes, web-fetch includes Playwright-based WeChat fetching with anti-bot bypass specifically for 微信公众号 articles. It handles lazy-loaded images, extracts metadata, and outputs clean Markdown or PDF with CJK font support.

How do I remove ads and noise from scraped web content?▼

Web-fetch uses CSS-selector-driven cleaning to automatically strip navigation, headers, footers, and ads during scraping. The result is publication-ready Markdown or PDF with only the core article content preserved.

Does web-fetch support image preservation when converting to PDF?▼

Yes, web-fetch preserves images during both Markdown and PDF conversion, maintaining alt text for accessibility. For WeChat articles, it specifically handles lazy-loaded images and renders them correctly in the PDF output.

What's the difference between using web-fetch for general websites versus WeChat?▼

Web-fetch uses crawl4ai for generic website scraping and Playwright-based fetching for WeChat articles. WeChat support includes anti-bot handling, lazy-image loading, and metadata extraction tailored to 微信公众号 structure.

Can I convert extracted Markdown to PDF separately?▼

Yes, web-fetch includes a dedicated md_to_pdf script for converting Markdown files to PDF. This allows you to separate extraction and PDF conversion into distinct steps with CJK font support.

Related Skills

Explore More Content & Communication Skills→
UniquecreteUniquecrete

web-scraper

Extract structured data from web pages using CSS schemas and crawling.

Community
Intermediate
lapaixkemsdortshlee-svglapaixkemsdortshlee-svg

markitdown

Convert binary and web-based files into clean Markdown for text processing.

Community
Intermediate
itda-skillsitda-skills

web-reader

Download web pages and YouTube videos into Markdown or JSON.

Official
Intermediate
HouseGarofaloHouseGarofalo
3

web-automation

Automate browser interactions and web scraping with Playwright, Puppeteer, Selenium, and Scrapy.

Official
Intermediate