BrightData

Multi-tier web scraping handles blocked requests by automatically escalating through four fallback layers: WebFetch, customized curl, Playwright browser automation, and Bright Data MCP. This tiered approach reliably retrieves content from URLs protected by bot detection.

Multi-page crawling with fallback tiers retrieves dynamic JavaScript content by first trying WebFetch, then escalating to Playwright browser automation if needed. It maps site links and returns the fetched content structured in markdown format.

Tiered fallback works by sequentially attempting four scraping methods: WebFetch first, then curl with Chrome headers, followed by Playwright browser automation, and finally Bright Data MCP. If a lower tier fails to retrieve content, the next tier automatically activates.

Bright Data MCP access is only required for the fourth and final fallback tier. The workflow first attempts scraping using WebFetch, curl with Chrome headers, and Playwright browser automation before escalating to Bright Data MCP for advanced bot protection.

You can override default scraping configurations by setting up an optional local customization directory. This allows you to tailor the WebFetch, curl, and Playwright tier behaviors to suit specific site requirements before escalating to Bright Data MCP.

WebFetch fails on sites using bot detection or heavy JavaScript rendering because it cannot execute dynamic scripts. When WebFetch and curl fail, the workflow escalates to Playwright browser automation to render the page and retrieve the actual content.