skills.rest
dapperdivers
dapperdiversCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

Web Application Spider & Crawler

Crawl web applications to discover endpoints, forms, links, and API routes.

1|1|Updated Feb 8, 2026
Search Tags:#penetration testing#endpoint discovery#web crawler#api discovery#security research#javascript analysis
One-click install
npx skills add https://github.com/dapperdivers/roundtable-arsenal --skill web-application-spider-crawler
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Web Application Spider & Crawler
Source: https://github.com/dapperdivers/roundtable-arsenal/tree/main/pentest/web-spider
Command: npx skills add https://github.com/dapperdivers/roundtable-arsenal --skill web-application-spider-crawler

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl, grep, jq, python3, sed, awk, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill solves the challenge of manually discovering web application attack surfaces by automatically crawling websites and identifying pages, forms, links, and API endpoints.

Core Features & Use Cases

  • Endpoint Discovery: Crawl web applications to collect URLs, API routes, forms, and static assets with configurable depth and scope.
  • JavaScript Analysis: Extract potential API endpoints from JavaScript files using static analysis techniques.
  • Security Research Workflow: Help authorized penetration testers map application structure before vulnerability assessment and testing activities.

Quick Start

Use the web spider skill to crawl an authorized application and return discovered endpoints, forms, and API routes.

Frequently Asked Questions about Web Application Spider & Crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I discover web application endpoints and API routes for security research?▼

Web application endpoint discovery automates crawling websites to identify pages, forms, links, and API routes. This Skill performs automated crawling and static JavaScript analysis to map application structure before vulnerability assessment.

Can I extract API endpoints from JavaScript files during penetration testing?▼

Yes, JavaScript analysis extracts potential API endpoints from JavaScript files using static analysis techniques. This Skill applies static analysis to JavaScript files to discover hidden API routes and produce structured attack surface data.

What do I need to crawl web applications and map an attack surface?▼

Crawling web applications requires command-line web retrieval, JSON processing, URL parsing, and static JavaScript analysis capabilities. You need curl, grep, jq, python3, sed, and awk installed to run the automated crawling and analysis workflows.

When do I need automated endpoint discovery for web applications?▼

Automated endpoint discovery is needed during authorized penetration testing and application reconnaissance workflows. It solves the challenge of manually discovering web application attack surfaces by automatically crawling websites to identify pages, forms, links, and API endpoints.

Does this web crawler support configurable depth and scope for application reconnaissance?▼

Yes, this web crawler supports configurable depth and scope for application reconnaissance. It crawls web applications to collect URLs, API routes, forms, and static assets, returning discovered endpoints as structured attack surface data.

Why does static JavaScript analysis matter for API discovery workflows?▼

Static JavaScript analysis matters for API discovery because JavaScript files often contain hidden API endpoints not visible in the application UI. This Skill extracts potential API endpoints from JavaScript files to help authorized penetration testers map the complete attack surface.

Related Skills

Explore More Software Engineering Skills→
yun520-1yun520-1
38

log-analyzer

Parse and search application logs using grep, awk, jq, and Python3.

Community
Advanced
uphiagouphiago
1.1k

cors-chain-automation

Probe API endpoints for exploitable CORS misconfigurations and generate PoC files.

Community
Intermediate
giosuetedeschi-specgiosuetedeschi-spec

Burp Suite Web Application Testing

Intercept and modify HTTP traffic to test web application security.

Community
Advanced
AeonDaveAeonDave
15

dirsearch

Scan web servers for hidden directories and files with recursive scanning.

Community
Intermediate