exploring-web-app

Provide a starting URL and the crawler follows same-origin internal links up to a configurable depth (default 2) and page cap (default 15). It deduplicates on post-redirect URLs, strips tracking query params, and skips file downloads, anchors, and destructive links.

Capture an authenticated session once as a storageState file, then launch the browser with storage_state_path so every crawled page is already authenticated. Deduplicate on post-redirect URLs so an expired session shows up as repeated /login records instead of fake pages.

Yes, which is why this crawl enforces a hard skip list covering logout, delete, reset, checkout, and admin action URLs. Skipped destructive links are recorded in the map as skipped rather than navigated, which matters most during authenticated crawls.

No, the tool does not respect robots.txt, so you must have authorization to crawl the target site. If the site returns 429 responses, lower the page cap rather than adding explicit sleeps, since the network-idle wait usually paces requests.

This happens when the session expires mid-crawl or pages redirect to /login. Every remaining URL records as the same login page with zero errors, so check final post-redirect URLs before concluding the site is healthy.

SPAs that change URLs via the History API may only expose currently-rendered links, so routes revealed only after user interaction can be missed. Infinite-scroll feeds are crawled only in their initial state, not exhaustively.