crawler

Automate website exploration and document discovered APIs into crawl blueprints.

3|Updated Nov 30, 2025

Search Tags:#automation #code generation #API discovery #web #crawl #Playwriter

Ready to run

npx skills add https://github.com/alpoxdev/claude-code --skill crawler

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: crawler
Source: https://github.com/alpoxdev/claude-code/tree/main/.claude/skills/crawler
Command: npx skills add https://github.com/alpoxdev/claude-code --skill crawler

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates the exploration of websites and documentation of discovered APIs, turning manual crawling into reproducible automation.

Core Features & Use Cases

Automated crawl planning: outline pages, selectors, and API endpoints.
Documentation generation: output ANALYSIS.md, SELECTORS.md, API.md, NETWORK.md based on live exploration.
Code scaffolding: produce CRAWLER.ts templates for both API-driven and DOM-based crawling.

Quick Start

To begin, create a session and navigate to the target site, then collect structure and API hints to generate a crawler skeleton.

Frequently Asked Questions about crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate website crawling and API discovery?▼

Automate website crawling and API discovery by using a Playwriter workflow to intercept network requests, handle authentication, and scrape DOM elements. This process generates a crawl blueprint, API documentation, and TypeScript starter code for local execution.

How do I generate TypeScript crawler templates from web pages?▼

Generate TypeScript crawler templates by running an automated exploration of target web pages. The workflow analyzes live DOM structures and intercepted API endpoints to scaffold CRAWLER.ts templates ready for local execution.

What is DOM-based scraping and API interception for documentation generation?▼

DOM-based scraping and API interception for documentation generation is the process of exploring live websites to capture network traffic and page selectors, producing structured ANALYSIS.md, SELECTORS.md, API.md, and NETWORK.md files.

Does the Playwriter workflow support automated crawl planning and authentication handling?▼

Yes, the Playwriter workflow supports automated crawl planning by outlining pages, selectors, and API endpoints. It handles authentication scenarios during exploration to produce reproducible crawl blueprints and starter code.

Can I document discovered APIs automatically during web scraping?▼

Document discovered APIs automatically during web scraping by intercepting network requests in real-time. The workflow outputs structured API.md and NETWORK.md documentation based on live exploration of the target site.

What is the best way to turn manual website crawling into reproducible automation?▼

Turn manual website crawling into reproducible automation by creating a session, navigating to the target site, and collecting structure and API hints. This generates a crawler skeleton with TypeScript templates and reference documentation.