skills.rest
wgpsec
wgpsecOFFICIAL
Explore all skills by this author →
Website𝕏@wgpsecGitHub

gau-urlfetch

Collects historical URLs for target domains from Wayback Machine, Common Crawl, OTX, and URLScan.

1.7k|238|Updated Dec 7, 2019
Search Tags:#reconnaissance#attack surface#osint#gau#Wayback Machine#url collection#parameter discovery
One-click install
npx skills add https://github.com/wgpsec/AboutSecurity --skill gau-urlfetch
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gau-urlfetch
Source: https://github.com/wgpsec/AboutSecurity/tree/main/skills/tool/gau-urlfetch
Command: npx skills add https://github.com/wgpsec/AboutSecurity --skill gau-urlfetch

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Discovering hidden endpoints, parameters, and API paths on a target domain is difficult when relying only on live crawling. This Skill uses gau to aggregate known URLs from four passive historical data sources, expanding the attack surface without sending a single request to the target.

Core Features & Use Cases

  • Multi-source URL aggregation: Collects known URLs from Wayback Machine, Common Crawl, OTX, and URLScan, with optional subdomain inclusion and provider selection.
  • Pipeline-friendly filtering: Extracts parameterized URLs, JavaScript files, API endpoints, and sensitive paths (admin, backup, .env, .git) via grep pipelines, and feeds results into tools like httpx or dalfox.
  • Use Case: During reconnaissance on example.com, run gau with --subs to gather all historical URLs, filter for URLs containing parameters, and pipe them into a vulnerability scanner to find XSS or injection points.

Quick Start

Use the gau-urlfetch skill to collect all historical URLs including subdomains for target.com and save them to a file.

Frequently Asked Questions about gau-urlfetch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I collect historical URLs for a domain with gau?▼

Run gau followed by the target domain, such as gau target.com, to fetch known URLs from all default providers. Add --subs to include subdomains and -o urls.txt to save output to a file.

What data sources does gau use for URL collection?▼

gau aggregates four passive sources: Wayback Machine, Common Crawl, OTX AlienVault, and URLScan. You can restrict sources with the --providers flag, for example --providers wayback,commoncrawl.

How to find parameters and API endpoints from gau output?▼

Pipe gau output through grep to filter results: grep "=" extracts parameterized URLs, grep -E "/api/|/v[0-9]/" finds API endpoints, and grep "\.js$" lists JavaScript files. Combine with sort -u to deduplicate.

Can gau filter results by date range?▼

Yes, gau supports --from and --to flags with year-month values, such as gau --from 202301 --to 202612 target.com, to limit results to a specific historical window.

Does gau send requests to the target website?▼

No, gau is fully passive and only queries third-party archives like Wayback Machine and Common Crawl. To check which collected URLs are still live, pipe the output into httpx for active probing.

Related Skills

Explore More Software Engineering Skills→
AeonDaveAeonDave
15

subfinder

Enumerate subdomains for a target domain using passive OSINT sources.

Community
Intermediate
davila7davila7
30.5k

Red Team Tools and Methodology

Automates reconnaissance, subdomain enumeration, and vulnerability scanning workflows for bug bounty hunting.

Community
Intermediate
zebbernzebbern
5

endpoint-discovery

Discover web endpoints and historical URLs using Katana, Gospider, and GAU.

Community
Intermediate
cloudlinesscloudliness

oss-forensics

Collect multi-source evidence from GitHub repositories for OSS supply chain incidents.

Community
Advanced