web-fetch-fallback

To fetch web content when WebFetch fails, you can use an automated routing script that classifies URLs and switches to alternative fetch methods like gh CLI, curl with pandoc, or agent-browser. This ensures reliable retrieval of GitHub files, documentation sites, and JS-rendered pages.

The best way to scrape JS-rendered pages as plain text is using agent-browser to execute page scripts and extract rendered content. This method bypasses static HTML parsing limits when fetching dynamic documentation or web applications.

Routing curl requests through an https proxy with a custom user-agent is supported by configuring the https_proxy environment variable and setting a custom user-agent string. This approach increases fetch reliability for static documentation sites and blogs.

The web-fetch-fallback script does not strictly require pandoc or lynx, but supports them as curl-based parsing alternatives to convert HTML to plain text. It uses URL-type detection to select the best available fetch path for static documentation sites.

Fetching a static docs site returns incomplete content when the primary WebFetch method is blocked or times out. Using a multi-method fallback approach with curl-based parsers like html2text or pandoc retrieves the full plain text accurately.