skills.rest
gooseworks-ai
gooseworks-aiOFFICIAL
Explore all skills by this author →
WebsiteGitHub

web-archive-scraper

Discover Wayback Machine snapshots via the CDX API and retrieve archived HTML content.

1.1k|200|Updated Mar 2, 2026
Search Tags:#archive#historical-data#data-extraction#web-scraping#content-fetching#wayback#cdx-api
One-click install
npx skills add https://github.com/gooseworks-ai/goose-skills --skill web-archive-scraper-gooseworks-ai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-archive-scraper
Source: https://github.com/gooseworks-ai/goose-skills/tree/main/skills/capabilities/web-archive-scraper
Command: npx skills add https://github.com/gooseworks-ai/goose-skills --skill web-archive-scraper-gooseworks-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, and includes scripts (resource) components.

What problem does it solve?

This Skill helps researchers and analysts quickly locate archived versions of websites on the Wayback Machine and retrieve their content for historical analysis.

Core Features & Use Cases

  • Archive discovery: search snapshots of a target URL, apply date and status filters, and deduplicate results.
  • Content retrieval: fetch archived HTML pages for offline analysis, QA, and evidence gathering.
  • Use Case: monitor how a competitor's product pages and testimonials evolve over time to inform strategy.

Quick Start

Provide a URL to fetch archived snapshots and optionally download the most relevant archived page content.

Frequently Asked Questions about web-archive-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find archived website snapshots from the Wayback Machine for historical analysis?▼

To find archived website snapshots, you query the Wayback Machine CDX API to discover historical data, apply date and status filters, and deduplicate results by day for accurate historical analysis.

How do I scrape historical data from the Wayback Machine using the CDX API?▼

You scrape historical data by querying the CDX API to discover archived snapshots, optionally fetching archived HTML content while respecting rate limits, and outputting the extracted data in JSON, CSV, or a readable summary.

Can I retrieve and download archived HTML content for offline analysis?▼

Yes, you can retrieve and download archived HTML content for offline analysis. The Skill fetches archived pages after discovering them via the CDX API, supporting QA and evidence gathering workflows.

Does this Wayback Machine scraper require any specific Python dependencies?▼

The Wayback Machine scraper requires the Python requests library to query the CDX API and fetch archived HTML content, operating as a script without additional heavy web scraping dependencies.

What is the best way to track how a competitor's product pages evolve over time?▼

The best way to track competitor product pages is to discover archived snapshots via the CDX API, retrieve their historical HTML content, and analyze the extracted data to monitor testimonial and page evolution.

Are there rate limits when fetching archived content from the Wayback Machine?▼

Yes, when fetching archived content from the Wayback Machine, the scraper respects rate limits to ensure stable data extraction and prevent query failures during bulk historical snapshot retrieval.

Related Skills

Explore More Data & Analytics Skills→
dawsonblockdawsonblock

github-wayback-recovery

Recover deleted GitHub content from Wayback Machine and CDX API snapshots.

Community
Advanced
AnastasiyaWAnastasiyaW
140

product-meaning-extractor

Extract core value propositions, emotional hooks, and competitive positioning from product data.

Community
Intermediate
mbastakismbastakis
5

crawl4ai

Crawl web pages with Python and extract structured data as Markdown or JSON.

Community
Intermediate
ArianHobson333ArianHobson333
7

web-archive-analysis

Query the Wayback Machine CDX API to identify technology migrations from historical snapshots.

Community
Intermediate