web-crawler-ocr

Crawl web pages and extract image text with Gemini OCR into markdown files.

Updated Jul 7, 2026
One-click install
npx skills add https://github.com/hivgb1-ai/do-better-workspace-v2 --skill web-crawler-ocr-hivgb1-ai
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: web-crawler-ocr
Source: https://github.com/hivgb1-ai/do-better-workspace-v2/tree/main/.claude/skills/web-crawler-ocr
Command: npx skills add https://github.com/hivgb1-ai/do-better-workspace-v2 --skill web-crawler-ocr-hivgb1-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, scrapling, curl_cffi, browserforge, playwright, trafilatura, aiohttp, certifi, python-dotenv, Pillow, and includes scripts (resource) components.

What problem does it solve? Extracting content from modern websites is difficult: many sites use JavaScript rendering, lazy-loaded images, and Cloudflare protection, while Claude's 5MB image limit blocks large image analysis. This Skill crawls any URL locally and produces a clean markdown file with OCR text inserted at each image position. ## Core Features & Use Cases - 3-Tier Fetching: Automatically escalates from plain HTTP to stealth browser (Camoufox) to Playwright with scrolling, handling static sites, SPAs, and Cloudflare-protected pages. - Structured Data Extraction: Pulls JSON-LD, Next.js RSC payloads, and meta tags from SPA sites (Korean e-commerce like Olive Young, Coupang, Musinsa) alongside trafilatura-cleaned content. - Gemini OCR Inline: Analyzes up to 15 images per page (20MB each, bypassing Claude's 5MB limit) and inserts OCR results directly at image positions in the output markdown. - Use Case: Provide a competitor's product page URL and receive a markdown file saved to your workspace containing the full page text, structured product data (price, brand, category), and OCR-extracted text from all product images. ## Quick Start Ask Claude to analyze a URL, for example: analyze this competitor site https://example.com and save it to my resources folder.

Frequently Asked Questions about web-crawler-ocr

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I crawl a website and extract text from its images?

Run the web-crawler.py script with a URL and output path. It fetches the page, extracts content with trafilatura, downloads images, and uses the Gemini API to OCR each image, inserting results inline in the output markdown.

How to scrape JavaScript-rendered SPA pages like Korean e-commerce sites?

Use the --wait and --scroll flags, for example: python3 web-crawler.py URL output.md --wait 5000 --scroll. This triggers Playwright rendering and page scrolling to load lazy-loaded content, and extracts structured data from JSON-LD and Next.js RSC payloads.

What do I do when a site blocks the crawler with Cloudflare?

Install the stealth dependencies with pip install -r requirements-stealth.txt and scrapling install, then use the --stealth flag. This routes the request through the Camoufox browser to bypass Cloudflare protection.

Does web crawling require an API key?

Crawling itself is fully local and needs no key. Only the OCR feature requires a GEMINI_API_KEY, set via environment variable or a .env file in the scripts directory. Use --no-ocr to skip OCR entirely.

What are the limitations of Gemini OCR for web images?

The Gemini free tier allows 15 requests per minute, each image can be up to 20MB, and the crawler processes a maximum of 15 images per page by default. The image limit is configurable with the --max-images option.