content-extract

Extracts webpage content to Markdown using web_fetch probing with MinerU API fallback.

7|Updated May 13, 2026
One-click install
npx skills add https://github.com/DawnMoon1542/agents-skills --skill content-extract-dawnmoon1542
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: content-extract
Source: https://github.com/DawnMoon1542/agents-skills/tree/main/search-skills/content-extract
Command: npx skills add https://github.com/DawnMoon1542/agents-skills --skill content-extract-dawnmoon1542

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Webpages from platforms like WeChat (mp.weixin.qq.com) and Zhihu often block automated fetching with anti-bot pages, captchas, or dynamic rendering, leaving agents with unusable content. This Skill provides a unified URL-to-Markdown pipeline that probes cheaply first and falls back to the MinerU API, always returning a traceable result with source links. ## Core Features & Use Cases - Two-tier extraction workflow: Probes with low-cost web_fetch first, then falls back to the official MinerU API (MinerU-HTML model) when the probe fails or the domain is whitelisted. - Domain whitelist and heuristics: Ships reference rules for known anti-bot domains (WeChat, Zhihu, Xiaohongshu) and failure signals like captcha text, thin content, or 403 errors. - Traceable result contract: Every run returns a JSON contract with source_url, engine, markdown, artifact paths, and a sources list so downstream skills can verify provenance. - Use Case: Given a WeChat article URL that web_fetch cannot read, the Skill routes directly to MinerU and returns clean Markdown plus the local markdown_path and MinerU zip link for auditing. ## Quick Start Ask the agent to extract this WeChat article URL into Markdown and return the result with its source links.

Frequently Asked Questions about content-extract

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract a WeChat article to Markdown?

Pass the mp.weixin.qq.com URL to the extraction workflow, which skips probing for whitelisted domains and calls the MinerU API with the MinerU-HTML model. The result includes the Markdown body plus source links for traceability.

How to convert a webpage to Markdown when web_fetch is blocked?

When web_fetch returns 403 errors, captcha pages, or thin content, the workflow falls back to the MinerU document parsing API. A deterministic script wraps the MinerU call and returns a compact JSON contract with the extracted Markdown.

What signals indicate a webpage extraction failed?

Failure signals include HTTP 401/403/429 responses, interstitial text like captcha or 'open in WeChat client' prompts, and content under roughly 800 characters for article URLs. These heuristics trigger the MinerU fallback path.

Does this approach bypass login walls or captchas?

No, the Skill explicitly does not bypass logins or captchas, treating those as access-layer problems. If both web_fetch and MinerU fail, it reports the failure reason and suggests next steps such as providing a mirror URL or uploading exported HTML.

What are the limitations of MinerU-based webpage extraction?

MinerU may fail on heavily protected pages, and the upload-HTML fallback flow is not yet implemented. The Skill also requires the mineru-extract wrapper script to be installed as a sibling directory or located via environment variables.