skills.rest
jayjpatel9717
jayjpatel9717COMMUNITY
Explore all skills by this author →
GitHub

web-crawling

Crawl websites to discover URLs, forms, and API endpoints for security reconnaissance.

Updated Apr 21, 2026
Search Tags:#penetration testing#web crawling#endpoint discovery#security reconnaissance#form discovery
One-click install
npx skills add https://github.com/jayjpatel9717/kurukshetra --skill web-crawling-jayjpatel9717
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-crawling
Source: https://github.com/jayjpatel9717/kurukshetra/tree/main/squads/pentest/agents/eklavya/skills/web-crawling
Command: npx skills add https://github.com/jayjpatel9717/kurukshetra --skill web-crawling-jayjpatel9717

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires crawl4ai, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the process of web crawling, enabling efficient reconnaissance and discovery of endpoints and forms for security assessment.

Core Features & Use Cases

  • Deep Crawl: Automatically crawl websites to discover URLs, forms, API endpoints, and JavaScript files.
  • Form Discovery: Extracts and details all forms on a website for potential injection testing.
  • API Endpoint Discovery: Identifies API endpoints for further analysis and testing.
  • Use Case: When preparing for a penetration test, use this Skill to quickly map the attack surface of a target website and gather intelligence on potential entry points.

Quick Start

Use the web-crawling skill to crawl the website 'https://example.com' and map all endpoints.

Frequently Asked Questions about web-crawling

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate web crawling for security reconnaissance?▼

Automate security reconnaissance web crawling by deploying crawl4ai to discover all URLs, forms, and API endpoints. This maps the target website attack surface and gathers intelligence on potential entry points.

What is endpoint discovery and how does it work during penetration testing?▼

Endpoint discovery identifies API endpoints and JavaScript files on a target website during penetration testing. It works by injecting JavaScript to extract and detail forms for potential injection testing.

Does automated web crawling support form discovery for injection testing?▼

Automated web crawling supports form discovery by extracting and detailing all forms on a given website. This provides the necessary intelligence for preparing to test potential injection points during security assessments.

Do I need crawl4ai to discover API endpoints on a target website?▼

You need crawl4ai to discover API endpoints on a target website. The Skill requires this dependency to perform automated crawling and JavaScript injection for accurate form and API discovery.

What is the best way to map the attack surface of a website before a penetration test?▼

The best way to map a website attack surface before a penetration test is to automate web crawling to quickly discover all URLs, forms, API endpoints, and JavaScript files for intelligence gathering.

Related Skills

Explore More Software Engineering Skills→
Harmeet10000Harmeet10000
7

crawl4ai

Crawl JavaScript-heavy and multi-URL sites with structured and markdown outputs.

Community
Advanced
dapperdiversdapperdivers
1

post-parameter-tester

Test POST parameters for SQL injection, XSS, and command injection vulnerabilities.

Community
Advanced
wgpsecwgpsec
1.6k

katana-crawl

Crawl web targets with JavaScript rendering to discover hidden endpoints and API surfaces.

Official
Advanced
salmanabdurrahmansalmanabdurrahman

client-spa-endpoints

Extract API endpoints, hostnames, and routes from JavaScript bundles via static analysis.

Community
Advanced