skills.rest
OpenSourceAGI
OpenSourceAGIOFFICIAL
Explore all skills by this author →
GitHub

ask-render-url-to-html

Guides selection and operation of JSDOM and Puppeteer page-rendering services for bot-protected pages.

74|11|Updated Jul 4, 2024
Search Tags:#web scraping#docker#puppeteer#jsdom#cloudflare bypass#page rendering#stealth plugin
One-click install
npx skills add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill ask-render-url-to-html-opensourceagi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ask-render-url-to-html
Source: https://github.com/OpenSourceAGI/qwksearch-research-agent/tree/main/skills/ask-render-url-to-html
Command: npx skills add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill ask-render-url-to-html-opensourceagi

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Plain HTTP fetches fail on JavaScript-rendered pages and Cloudflare bot-check interstitials, returning empty bodies or challenge pages instead of content. This Skill explains how to choose between and run the two self-hosted rendering services in the render-url-to-html package: a Dockerised Puppeteer proxy with stealth evasion and a JSDOM-based scraper with scripted automation steps. ## Core Features & Use Cases - Renderer Selection Guide: Decision table mapping situations (static HTML, JS rendering, Cloudflare bot checks, serverless sessions) to the right tool: plain fetch, scraper-jsdom, scraper-puppeteer, or the Cloudflare Worker html-renderer-api. - scraper-puppeteer Operation: Run the Koa proxy over puppeteer-extra with the stealth plugin in Docker, send URLs as query parameters, and receive fully rendered responses with hop-by-hop headers stripped. - scraper-jsdom Automation: Use the JSDomScraper class with click, type, and waitFor automation steps plus custom scripts to extract title, meta description, text, links, and script results. - Use Case: A scraper pipeline hits a Cloudflare interstitial on a target site. You deploy scraper-puppeteer via docker compose, route requests through it, and add a residential proxy via PROXY_URL when stealth alone is insufficient. ## Quick Start Ask the assistant to help you render a JavaScript-heavy page that returns a bot-check page, and it will guide you through picking and running the right rendering service.

Frequently Asked Questions about ask-render-url-to-html

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I render a JavaScript page that returns empty HTML?▼

Use the scraper-jsdom JSDomScraper class for pages needing script execution without evasion, or scraper-puppeteer for real Chromium rendering. JSDOM runs scripts but lacks layout and full browser APIs, so React or Vue pages may still need Puppeteer.

How do I bypass Cloudflare bot checks when scraping?▼

Run the scraper-puppeteer Docker service, which uses puppeteer-extra with the stealth plugin to defeat fingerprinting. If stealth alone fails, the issue is IP reputation, so set PROXY_URL to route through a residential proxy.

JSDOM vs Puppeteer for web scraping, which should I use?▼

Use JSDOM when pages need scripts but no bot evasion and you want a lightweight library call. Use Puppeteer when facing Cloudflare interstitials, when JSDOM returns empty bodies on framework-rendered pages, or when real browser behavior is required.

Why does scraper-puppeteer fail with Chromium not found outside Docker?▼

The executable path is hardcoded to /usr/bin/chromium-browser, which only exists inside the Docker image. Either install Chromium at that path, edit the executablePath option, or run the service via docker compose, which is the supported path.

Why does docker compose up fail with a missing network error?▼

The compose file declares the caddy network as external, so it must exist before startup. Run docker network create caddy first, then start the service with docker compose up -d --build.

What are the limitations of JSDOM-based scraping?▼

JSDOM executes scripts but is not a real browser: it has no layout engine, limited timer-driven rendering, and patchy web API support. Pages relying on API calls after load or complex framework rendering should be escalated to Puppeteer or the html-renderer-api service.

Related Skills

Explore More Software Engineering Skills→
PrimeIntellect-aiPrimeIntellect-ai
113

open_webpage

Fetch URLs and extract complete plain text from HTML and PDF documents.

Official
Intermediate
papachongpapachong
2

brave-search

Search Brave Search headlessly and extract page content as markdown.

Community
Intermediate
AgentComputerAIAgentComputerAI
5

proxy

Route Puppeteer web scraping through authenticated residential, datacenter, or mobile proxies.

Official
Advanced
VoDaiLoczVoDaiLocz
24

devops

Automates deployment and management of cloud infrastructure on Cloudflare, Docker, and GCP.

Community
Advanced