BrightData

Use progressive escalation: start with a simple fetch, then retry with Chrome browser headers via curl, then full browser automation with Playwright, and finally a residential proxy service like Bright Data that handles CAPTCHAs and advanced bot detection automatically.

For sites under 50 pages, scrape the starting URL, extract same-domain links, and batch-scrape discovered pages iteratively. For larger sites, use the Bright Data Crawl API with a depth limit and URL filter regex, then poll the snapshot endpoint for results.

Use Bright Data when lower tiers fail: sites with CAPTCHAs, advanced fingerprinting that detects headless browsers, or IP-based rate limiting requiring residential proxies. For simple public pages, WebFetch or curl is faster and free.

Static fetch methods like curl return empty content for SPAs because content loads via JavaScript. Browser automation with Playwright renders the full page including dynamic content, making it the appropriate tier for React, Vue, or Angular sites.

Light Crawl costs roughly $0.006 per page and is capped at 50 pages. Full Crawl via the Crawl API costs about $1.50 per 1,000 pages. The workflow requires user confirmation before crawls exceeding 20 pages or any Full Crawl.