crawler

Web scraping API discovery involves crawling target sites to systematically capture APIs, cookies, tokens, and headers. This tool automates that reconnaissance using Playwriter sessions and accessibility snapshots to document findings for reproducibility.

Documenting web crawling findings requires generating structured artifacts like ANALYSIS.md, SELECTORS.md, API.md, NETWORK.md, and CRAWLER.ts. This approach creates reproducible API mapping records and starter code under a dedicated site directory.

Yes, running a Playwriter session enables network request interception to map API endpoints, extract authentication details, and capture HTTP headers. It automatically generates crawler templates and structured documentation for the discovered site.

No external dependencies are required to generate crawler templates and document website APIs. You only need a Playwriter session environment to navigate the target site and trigger the automated artifact generation workflow.

To start automated crawler code creation, run a Playwriter session and navigate to your target site. The tool then builds the crawler documentation, network analysis, and CRAWLER.ts starter code for repeatable workflows.

Generating structured artifacts like SELECTORS.md and NETWORK.md during web reconnaissance ensures reproducibility and knowledge transfer. It captures API surfaces, authentication details, and HTTP headers in a standardized format for automated crawler workflows.