skills.rest
gn00678465
gn00678465COMMUNITY
Explore all skills by this author →
GitHub

crawler

Fetch web pages and convert them to clean Markdown.

1|Updated Mar 2, 2026
Search Tags:#markdown#firecrawl#web scraping#content extraction#fallback#jina reader#cloudflare bypass
One-click install
npx skills add https://github.com/gn00678465/crawler-skill --skill crawler-gn00678465
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: crawler
Source: https://github.com/gn00678465/crawler-skill/tree/main/skills/crawler-skill
Command: npx skills add https://github.com/gn00678465/crawler-skill --skill crawler-gn00678465

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires firecrawl-py>=2.0, httpx>=0.27, scrapling>=0.2, html2text>=2024.2.26, curl_cffi>=0.7.0, playwright>=1.41.0, patchright>=1.41.0, browserforge>=1.1.0, msgspec>=0.18.0, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a robust solution for fetching web page content and converting it into clean Markdown format, overcoming common challenges like anti-scraping measures and API failures.

Core Features & Use Cases

  • 3-Tier Fallback Chain: Reliably fetches content by trying Firecrawl, Jina Reader, and Scrapling sequentially.
  • High-Quality Markdown Output: Converts HTML to well-formatted Markdown.
  • Bypasses Restrictions: Includes capabilities for CAPTCHA detection, Cloudflare bypass, and stealthy crawling.
  • Use Case: When a user asks to summarize a blog post from a site that typically blocks direct scraping or shows a CAPTCHA, this Skill will successfully retrieve the content and provide the summary.

Quick Start

Use the crawler skill to fetch the content from https://example.com and save it to reports/example.md.

Frequently Asked Questions about crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract text content from a web page that blocks direct HTTP requests?▼

To extract text content from restricted web pages, this skill uses a 3-tier fallback chain (Firecrawl, Jina Reader, Scrapling) to fetch the page and convert it to clean markdown. It handles bot protection and CAPTCHAs by escalating to a stealth browser.

Does this web scraping tool work on Medium articles and WeChat public accounts?▼

Yes, this web scraping tool works on Medium articles and WeChat public accounts. It is specifically designed to bypass geo-restrictions and paywalls, successfully fetching and converting the content into markdown using its fallback chain.

What is the best way to convert HTML to markdown for documentation sites with Cloudflare protection?▼

The best way to convert HTML to markdown for Cloudflare-protected sites is using a tool with stealth browser capabilities. This skill bypasses Cloudflare protections by falling back to Scrapling, rendering the page, and extracting clean markdown.

Why does simple HTTP fetching return an empty page or CAPTCHA?▼

Simple HTTP fetching returns an empty page or CAPTCHA because many sites deploy anti-bot measures to block automated access. This skill overcomes this by escalating requests through Firecrawl, Jina Reader, and a stealth browser until the content is retrieved.

When should I not use this web page to markdown converter?▼

You should not use this web page to markdown converter for building custom scrapers, checking HTTP status codes, parsing already-downloaded HTML files, or answering conceptual questions about scraping tools. It is designed specifically for fetching live web page text content.

Can I use Firecrawl and Jina Reader together for content extraction?▼

Yes, you can use Firecrawl and Jina Reader together for content extraction because this skill implements them sequentially. If Firecrawl fails to fetch the web page, it automatically falls back to Jina Reader, then to a Scrapling stealth browser.

Related Skills

Explore More Software Engineering Skills→
glfruitglfruit

web-content-fetcher

Extract clean Markdown content from web pages using Jina Reader, Scrapling, and html2text.

Community
Intermediate
Blake-JohnBlake-John

pymupdf

Convert PDF documents into Markdown, JSON, and text for LLM workflows.

Community
Intermediate
u9401066u9401066

pdf-reader

Extract text, figures, and citations from PDFs into Markdown.

Community
Intermediate
superfhpsuperfhp

scrapling

Extract website data using HTTP requests, stealth browser automation, and spider crawling.

Community
Intermediate