skills.rest
NousResearch
NousResearchOFFICIAL
Explore all skills by this author →
Website𝕏@nousresearchGitHub

blocked-page-recovery

Recover blocked, paywalled, or WAF-protected web pages from third-party archives.

239k|48.8k|Updated Jul 22, 2025
Search Tags:#content extraction#paywall bypass#provenance tracking#Wayback Machine#WAF bypass#Web scraping#archive recovery
One-click install
npx skills add https://github.com/NousResearch/hermes-agent --skill blocked-page-recovery
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: blocked-page-recovery
Source: https://github.com/NousResearch/hermes-agent/tree/main/skills/web/blocked-page-recovery
Command: npx skills add https://github.com/NousResearch/hermes-agent --skill blocked-page-recovery

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Fetching a URL often fails with 403/429 errors, paywalls, Cloudflare interstitials, or bot-detection walls. This Skill provides a fallback ladder to retrieve a usable copy of the page from third-party services without retry-looping on the same blocked URL.

Core Features & Use Cases

  • Multi-route recovery ladder: Tries Wayback Machine, archive.today (with domain rotation), and Jina Reader in order of cost and reliability.
  • Provenance tracking: Tags every recovered copy as snapshot or live so citations can include the archive date and avoid presenting stale data as current.
  • Fake-success rejection: Validates response bodies against byte floors, interstitial titles, and redirect-stub patterns to discard AMP caches, Google Cache interstitials, and rate-limit HTML.
  • Use Case: A research agent needs the text of a paywalled news article. Run the recovery script and receive the archived body plus its snapshot timestamp for proper citation.

Quick Start

Use the blocked-page-recovery skill to fetch the contents of this paywalled article URL and return the recovered text with its provenance.

Frequently Asked Questions about blocked-page-recovery

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fetch a page that returns 403 or Cloudflare challenge?▼

Run the recovery script against the URL. It queries the Wayback Machine available API, rotates archive.today domains, and optionally calls Jina Reader with JINA_API_KEY, returning the first validated body along with its provenance.

What is the best archive service for paywalled news articles?▼

archive.today often holds user-submitted copies of paywalled articles that Wayback lacks. Wayback Machine is preferred first because it returns explicit snapshot timestamps needed for citation.

Does Google Cache still work for blocked pages?▼

No. Since mid-2024, webcache.googleusercontent.com returns a Google Search interstitial with a JS redirect, not a cached page. The recovery script rejects these bodies automatically.

Why does AMP cache return a redirect stub instead of content?▼

AMP caches typically serve a small meta-refresh page pointing back at the original blocked URL. The script detects this pattern and rejects the body to prevent fetch loops.

When should I use a real browser instead of an archive?▼

Use a browser tool only when archives have no copy and the page is a JS-only SPA that requires client-side rendering. Archives are cheaper and provide timestamped provenance that browsers cannot.

Related Skills

Explore More Education & Research Skills→
TeamWiseFlowTeamWiseFlow
8.4k

rss-reader

Convert RSS/Atom feed URLs into markdown lists of recent entries.

Official
Intermediate
johngrandsonjohngrandson

ck:research

Synthesize technical research into structured reports with citations and executive summaries.

Community
Advanced
Focus-GTSFocus-GTS

content-freshness

Flag stale content across AEM Edge Delivery sites using query index dates and .plain.html signals.

Official
Intermediate
kevinnftkevinnft
13

web-scraping

Extract structured data from static HTML, JS-rendered pages, and hybrid sites.

Community
Intermediate