Verified:GitHub

ask-html-renderer-api

Render JavaScript-heavy and bot-protected pages through a Cloudflare Worker browser API.

74|11|Updated Jul 4, 2024

Search Tags:#cloudflare workers #web scraping #puppeteer #durable objects #cookie persistence #page rendering #captcha bypass

Ready to run

npx skills add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill ask-html-renderer-api-opensourceagi

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ask-html-renderer-api
Source: https://github.com/OpenSourceAGI/qwksearch-research-agent/tree/main/skills/ask-html-renderer-api
Command: npx skills add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill ask-html-renderer-api-opensourceagi

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Scraping pages that require JavaScript execution, sit behind logins, or are protected by Cloudflare challenges fails with plain HTTP clients. This Skill explains how to operate the html-renderer-api Cloudflare Worker, which renders pages in a real browser and returns HTML or JSON. ## Core Features & Use Cases - Page Rendering: Call /api/render to execute JavaScript and return rendered HTML or a JSON payload with title, cookies, and load time. - Authenticated Sessions: Use /api/login and /api/fetch with a shared sessionId so cookies persist in a Durable Object across requests. - Bot-Protection Bypass: Enable bypassCaptcha and 2captcha integration to pass Cloudflare challenges, reCAPTCHA, and Turnstile. - Use Case: A research agent needs content from a JS-heavy dashboard behind a login. It posts credentials to /api/login with sessionId "acct", then calls /api/fetch with the same sessionId to retrieve the rendered page. ## Quick Start Ask the agent to render a JavaScript-heavy URL through the html-renderer-api worker using your SCRAPER_API_KEY and return the page HTML.

Frequently Asked Questions about ask-html-renderer-api

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I render a JavaScript-heavy page with the html-renderer-api?▼

POST to /api/render with the target url and your SCRAPER_API_KEY as a Bearer token. Set format to "json" for structured output with title, html, and cookies, and tune waitUntil, timeout, and blockImages for load behavior.

How do I scrape a website behind a login with Cloudflare Workers?▼

POST credentials to /api/login with a sessionId, then call /api/fetch with the same sessionId. Cookies are saved into Durable Object storage after each render and reloaded on the next, so the session persists across requests.

Why does the render API return 401 even with the correct key?▼

The key must exactly match the SCRAPER_API_KEY binding, and authentication runs before routing on every route including /api/login and /api/fetch. The key can be sent as a Bearer header, query parameter, or POST body field.

Can the renderer bypass Cloudflare challenges and CAPTCHAs?▼

Yes, set bypassCaptcha to true to attempt Cloudflare-challenge bypass, and provide a twoCaptchaKey for reCAPTCHA or Turnstile. Tune challengeMatch, maxRetries, and challengeTimeout when the interstitial markup is unusual.

Why do cookies disappear between render requests?▼

Cookie persistence is scoped per sessionId, with one Durable Object instance and cookie jar per id. If you change or omit the sessionId, you get the "default" jar, so reuse a consistent sessionId to keep sessions alive.

Why does bun run start fail for html-renderer-api?▼

The package.json start script points at a .js file while the source is TypeScript, and the project is a Cloudflare Worker, not a Node app. Use wrangler dev or wrangler deploy with a wrangler config binding MYBROWSER, BROWSER_DO, and SCRAPER_API_KEY.