skills.rest
phatblat
phatblatCOMMUNITY
Explore all skills by this author →
Website𝕏@phatblatGitHub

BrightData

Scrape URLs via four escalating tiers from WebFetch to Bright Data MCP.

12|Updated Aug 16, 2019
Search Tags:#data extraction#web scraping#content retrieval#bot detection#bright data#url scraping
One-click install
npx skills add https://github.com/phatblat/dotfiles --skill brightdata-phatblat
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: BrightData
Source: https://github.com/phatblat/dotfiles/tree/main/.claude/skills/BrightData
Command: npx skills add https://github.com/phatblat/dotfiles --skill brightdata-phatblat

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill overcomes website blocking, bot detection, and CAPTCHAs to reliably scrape content from any URL, even those that are difficult to access.

Core Features & Use Cases

  • Progressive Scraping: Uses a four-tier strategy, starting with simple fetching and escalating to professional services like Bright Data MCP when needed.
  • Bot & CAPTCHA Bypass: Effectively handles advanced anti-scraping measures.
  • Use Case: You need to scrape product details from an e-commerce site that actively blocks automated requests. This Skill will use its advanced tiers to retrieve the data successfully.

Quick Start

Use the BrightData skill to scrape the content from https://example.com.

Frequently Asked Questions about BrightData

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a website that blocks automated requests and uses bot detection?▼

Scraping a website with bot detection requires escalating access tiers from simple WebFetch to customized Curl, then Playwright browser automation, and finally Bright Data MCP for reliable content retrieval in markdown.

What is the best way to retrieve web content in markdown format from heavily protected URLs?▼

The best way to retrieve web content from protected URLs is a four-tier progressive scraping strategy: WebFetch, Curl with Chrome headers, Playwright browser automation, and Bright Data MCP, yielding markdown content.

Can I use Playwright browser automation to bypass CAPTCHAs during web scraping?▼

Playwright browser automation serves as the third tier in a progressive scraping workflow to bypass CAPTCHAs; if it fails, the system escalates to Bright Data MCP to ensure successful content retrieval.

Does Bright Data MCP work for scraping e-commerce product details behind anti-scraping measures?▼

Bright Data MCP works for scraping e-commerce product details by acting as the final tier in a progressive strategy, specifically targeting sites with active anti-scraping measures and CAPTCHAs for reliable data retrieval.

Why does WebFetch fail to retrieve content from certain URLs and how is it resolved?▼

WebFetch fails to retrieve content from certain URLs due to bot detection mechanisms blocking simple requests; this is resolved by escalating to customized Curl, Playwright browser automation, or Bright Data MCP.

Related Skills

Explore More Software Engineering Skills→
pynbj1001pynbj1001
4

BrightData

Scrape URLs through a four-tier workflow with automatic fallbacks and markdown output.

Community
Advanced
jackandkingjackandking

data-deduplication

Deduplicate datasets using exact, fuzzy, ID-based, and content similarity strategies.

Community
Intermediate
Quill-AgentQuill-Agent
3

scrapling

Automate web scraping with anti-bot bypass and JavaScript-rendered content.

Community
Advanced
GratefulJinx77GratefulJinx77

BrightData

Scrape web content with progressive escalation through WebFetch, Curl, Browser Automation, and Bright Data MCP.

Community
Advanced