web-content-fetcher

Name: web-content-fetcher
Availability: InStock
Author: adiytharpansa

Fetch Cloudflare-protected webpages as Markdown via fallback endpoints.

Updated Apr 14, 2026

Search Tags:#markdown #fetch #cloudflare #scraping #bypass #curl #web

Ready to run

npx skills add https://github.com/adiytharpansa/openclawuserland --skill web-content-fetcher-adiytharpansa

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: web-content-fetcher
Source: https://github.com/adiytharpansa/openclawuserland/tree/main/skills/web-content-fetcher
Command: npx skills add https://github.com/adiytharpansa/openclawuserland --skill web-content-fetcher-adiytharpansa

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When standard web_fetch or web_search cannot retrieve page content because of Cloudflare protections, scraping filters, or other blocking mechanisms, this Skill provides alternative endpoints to obtain the page as readable Markdown.

Core Features & Use Cases

Fallback endpoints: Sequentially try r.jina.ai, markdown.new, and defuddle.md to maximize the chance of retrieving content.
Multiple interfaces: Includes a bash script and a Python helper for quick CLI usage or integration into automation.
Use Case: Retrieve an article from a Cloudflare-protected news site for summarization or translation when direct scraping fails.

Quick Start

Use the web-content-fetcher to retrieve https://news.example.com/article/123 using the jina method to get Markdown output.

Frequently Asked Questions about web-content-fetcher

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I fetch content from a webpage that is blocked by Cloudflare protection?▼

To bypass Cloudflare protection and fetch blocked webpages, this Skill uses curl-based fallback access across r.jina.ai, markdown.new, and defuddle.md endpoints to retrieve the readable article text as Markdown. It applies when standard web fetching fails due to scraping defenses.

Why does standard web scraping fail to retrieve article text and how can I get it as Markdown?▼

Standard web scraping fails to retrieve article text due to filtering mechanisms and scraping defenses. You can retrieve the content as Markdown by sequentially trying alternative endpoints like r.jina.ai and markdown.new, which provide readable text for analysis when direct access is blocked.

Can I use curl to convert a protected webpage to Markdown for translation?▼

Yes, you can use curl to convert a protected webpage to Markdown for translation. The Skill provides a bash script and Python helper that apply curl-based fallback access to fetch blocked webpages and output readable article text suitable for summarization or translation.

What is the best way to retrieve content when web_fetch calls fail due to scraping filters?▼

The best way to retrieve content when web_fetch calls fail due to scraping filters is to sequentially try alternative endpoints. This Skill applies curl-based fallback access across r.jina.ai, markdown.new, and defuddle.md to maximize the chance of retrieving the page as readable Markdown.

Do I need to install dependencies to bypass web scraping defenses and fetch Markdown?▼

No dependencies are required to bypass web scraping defenses and fetch Markdown. The Skill operates independently using curl-based fallback access and provides a bash script and Python helper for quick CLI usage or automation integration without external dependencies.

What are the limitations when using fallback endpoints to fetch blocked webpages?▼

Limitations when using fallback endpoints to fetch blocked webpages include potential timeout handling issues and the need for sequential fallbacks. If r.jina.ai, markdown.new, and defuddle.md all fail to bypass the scraping defenses, the content cannot be retrieved.