Web Application Spider & Crawler
CommunityDiscover web attack surfaces faster
Software Engineering#penetration testing#endpoint discovery#web crawler#api discovery#security research#javascript analysis
Authordapperdivers
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill solves the challenge of manually discovering web application attack surfaces by automatically crawling websites and identifying pages, forms, links, and API endpoints.
Core Features & Use Cases
- Endpoint Discovery: Crawl web applications to collect URLs, API routes, forms, and static assets with configurable depth and scope.
- JavaScript Analysis: Extract potential API endpoints from JavaScript files using static analysis techniques.
- Security Research Workflow: Help authorized penetration testers map application structure before vulnerability assessment and testing activities.
Quick Start
Use the web spider skill to crawl an authorized application and return discovered endpoints, forms, and API routes.
Dependency Matrix
Required Modules
curlgrepjqpython3sedawk
Components
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: Web Application Spider & Crawler Download link: https://github.com/dapperdivers/roundtable-arsenal/archive/main.zip#web-application-spider-crawler Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 545,000+ vetted skills library on demand.