skills.rest
brightdata
brightdataOFFICIAL
Explore all skills by this author →
WebsiteGitHub

scrape

Scrape webpages into clean markdown using Bright Data's Web Unlocker API.

245|35|Updated Jan 28, 2026
Search Tags:#automation#markdown#api#web#scrape#captcha#bot-detection
One-click install
npx skills add https://github.com/brightdata/skills --skill scrape-brightdata
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scrape
Source: https://github.com/brightdata/skills/tree/main/skills/scrape
Command: npx skills add https://github.com/brightdata/skills --skill scrape-brightdata

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl, jq, and includes scripts (resource) components.

What problem does it solve?

Web content extraction is tedious and error-prone when pages are dynamic or protected. This skill automates turning web pages into clean, readable Markdown by leveraging Bright Data's Web Unlocker API, bypassing bot detection and CAPTCHA challenges.

Core Features & Use Cases

  • Automated page scraping into Markdown-ready text for articles, product pages, and blogs.
  • Handles JavaScript-heavy sites and bypasses common anti-bot measures for reliable content extraction.
  • Use cases include content repurposing, knowledge base creation, and data collection for research.

Quick Start

Provide a URL to scrape and receive clean markdown content in return.

Frequently Asked Questions about scrape

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape JavaScript-heavy webpages into clean markdown?▼

Scraping webpages into clean markdown is handled by sending a target URL to the Bright Data Web Unlocker API, which returns formatted text content. This process bypasses bot detection and CAPTCHAs to reliably extract articles, product pages, and blogs.

What environment variables do I need to scrape webpages via the Web Unlocker API?▼

Scraping webpages via the Web Unlocker API requires setting the BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE environment variables. These credentials authenticate requests to Bright Data for bypassing bot defenses and extracting content.

Can I use this to bypass CAPTCHAs and bot detection for web scraping?▼

Bypassing CAPTCHAs and bot detection for web scraping is supported through Bright Data's Web Unlocker API. It handles anti-bot measures automatically, allowing reliable content extraction from protected, JavaScript-heavy pages without manual intervention.

Does the scrape skill work with curl and jq for web content extraction?▼

Web content extraction with curl and jq is fully supported, as they are core dependencies. Curl manages HTTP requests to the Web Unlocker API, while jq processes JSON responses to isolate the clean markdown output.

What is the best way to automate article and blog content extraction into markdown?▼

Automating article and blog content extraction into markdown is best achieved using Bright Data's Web Unlocker API. It converts any webpage into clean text suitable for automation pipelines, knowledge base creation, or research data collection.

Related Skills

Explore More Software Engineering Skills→
semaes111semaes111

search

Retrieve Google search results via Bright Data's SERP API with pagination.

Community
Intermediate
aivos-xieaivos-xie
1

llm-wiki

Parse Markdown documents into a structured knowledge base for cross-referencing and queries.

Community
Advanced
brightdatabrightdata
245

bright-data-best-practices

Codifies best practices for building production-ready Bright Data API integrations.

Official
Advanced
dexhunterdexhunter

aluvia-web-proxy

Access restricted websites using Aluvia mobile proxies and Brave Search API keys.

Community
Intermediate