scrape

Name: scrape
Availability: InStock
Author: Bright Data

Scrape webpages into clean markdown using Bright Data's Web Unlocker API.

245|35|Updated Jan 28, 2026

Search Tags:#automation #markdown #api #web #scrape #captcha #bot-detection

Ready to run

npx skills add https://github.com/brightdata/skills --skill scrape-brightdata

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: scrape
Source: https://github.com/brightdata/skills/tree/main/skills/scrape
Command: npx skills add https://github.com/brightdata/skills --skill scrape-brightdata

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl, jq, and includes scripts (resource) components.

What problem does it solve?

Web content extraction is tedious and error-prone when pages are dynamic or protected. This skill automates turning web pages into clean, readable Markdown by leveraging Bright Data's Web Unlocker API, bypassing bot detection and CAPTCHA challenges.

Core Features & Use Cases

Automated page scraping into Markdown-ready text for articles, product pages, and blogs.
Handles JavaScript-heavy sites and bypasses common anti-bot measures for reliable content extraction.
Use cases include content repurposing, knowledge base creation, and data collection for research.

Quick Start

Provide a URL to scrape and receive clean markdown content in return.

Frequently Asked Questions about scrape

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scrape JavaScript-heavy webpages into clean markdown?▼

Scraping webpages into clean markdown is handled by sending a target URL to the Bright Data Web Unlocker API, which returns formatted text content. This process bypasses bot detection and CAPTCHAs to reliably extract articles, product pages, and blogs.

What environment variables do I need to scrape webpages via the Web Unlocker API?▼

Scraping webpages via the Web Unlocker API requires setting the BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE environment variables. These credentials authenticate requests to Bright Data for bypassing bot defenses and extracting content.

Can I use this to bypass CAPTCHAs and bot detection for web scraping?▼

Bypassing CAPTCHAs and bot detection for web scraping is supported through Bright Data's Web Unlocker API. It handles anti-bot measures automatically, allowing reliable content extraction from protected, JavaScript-heavy pages without manual intervention.

Does the scrape skill work with curl and jq for web content extraction?▼

Web content extraction with curl and jq is fully supported, as they are core dependencies. Curl manages HTTP requests to the Web Unlocker API, while jq processes JSON responses to isolate the clean markdown output.

What is the best way to automate article and blog content extraction into markdown?▼

Automating article and blog content extraction into markdown is best achieved using Bright Data's Web Unlocker API. It converts any webpage into clean text suitable for automation pipelines, knowledge base creation, or research data collection.