BrightData

Scrape URLs via four escalating tiers from WebFetch to Bright Data MCP.

12|Updated Aug 16, 2019

Search Tags:#data extraction #web scraping #content retrieval #bot detection #bright data #url scraping

Ready to run

npx skills add https://github.com/phatblat/dotfiles --skill brightdata-phatblat

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: BrightData
Source: https://github.com/phatblat/dotfiles/tree/main/.claude/skills/BrightData
Command: npx skills add https://github.com/phatblat/dotfiles --skill brightdata-phatblat

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill overcomes website blocking, bot detection, and CAPTCHAs to reliably scrape content from any URL, even those that are difficult to access.

Core Features & Use Cases

Progressive Scraping: Uses a four-tier strategy, starting with simple fetching and escalating to professional services like Bright Data MCP when needed.
Bot & CAPTCHA Bypass: Effectively handles advanced anti-scraping measures.
Use Case: You need to scrape product details from an e-commerce site that actively blocks automated requests. This Skill will use its advanced tiers to retrieve the data successfully.

Quick Start

Use the BrightData skill to scrape the content from https://example.com.

Frequently Asked Questions about BrightData

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scrape a website that blocks automated requests and uses bot detection?▼

Scraping a website with bot detection requires escalating access tiers from simple WebFetch to customized Curl, then Playwright browser automation, and finally Bright Data MCP for reliable content retrieval in markdown.

What is the best way to retrieve web content in markdown format from heavily protected URLs?▼

The best way to retrieve web content from protected URLs is a four-tier progressive scraping strategy: WebFetch, Curl with Chrome headers, Playwright browser automation, and Bright Data MCP, yielding markdown content.

Can I use Playwright browser automation to bypass CAPTCHAs during web scraping?▼

Playwright browser automation serves as the third tier in a progressive scraping workflow to bypass CAPTCHAs; if it fails, the system escalates to Bright Data MCP to ensure successful content retrieval.

Does Bright Data MCP work for scraping e-commerce product details behind anti-scraping measures?▼

Bright Data MCP works for scraping e-commerce product details by acting as the final tier in a progressive strategy, specifically targeting sites with active anti-scraping measures and CAPTCHAs for reliable data retrieval.

Why does WebFetch fail to retrieve content from certain URLs and how is it resolved?▼

WebFetch fails to retrieve content from certain URLs due to bot detection mechanisms blocking simple requests; this is resolved by escalating to customized Curl, Playwright browser automation, or Bright Data MCP.