salesforce-developer-site-scraper

Extract Salesforce Developer documentation pages into clean Markdown.

27|9|Updated Aug 8, 2025

Search Tags:#documentation #markdown #readability #playwright #scrape #headless-chrome

Ready to run

npx skills add https://github.com/taurgis/sfcc-dev-mcp --skill salesforce-developer-site-scraper

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: salesforce-developer-site-scraper
Source: https://github.com/taurgis/sfcc-dev-mcp/tree/main/.github/skills/salesforce-developer-site-scraper
Command: npx skills add https://github.com/taurgis/sfcc-dev-mcp --skill salesforce-developer-site-scraper

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires playwright, @mozilla/readability, jsdom, turndown, and includes scripts (resource) components.

What problem does it solve?

Salesforce developers and teams often need a readable, portable snapshot of documentation that loads content asynchronously or behind consent banners, enabling offline reading, indexing, or AI-assisted lookup.

Core Features & Use Cases

Extract and convert Salesforce Developer docs into clean Markdown using a headless browser and readability parsing.
Handle dynamic content and consent prompts to produce reliable snapshots for Apex, LWC, and platform docs.
Use case: capture relevant doc pages for quick reference in knowledge bases or training materials.

Quick Start

Install dependencies and run the scrape-to-markdown script with the target URL and an output Markdown path.

Frequently Asked Questions about salesforce-developer-site-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I convert Salesforce developer documentation to Markdown for offline reading?▼

You can convert Salesforce developer documentation to Markdown by using a headless browser to render dynamic pages and readability parsing to extract content. This approach bypasses consent banners and asynchronous loading to produce clean Markdown files for offline access.

Why does scraping Salesforce docs fail with standard HTTP requests?▼

Scraping Salesforce docs fails with standard requests because the content loads asynchronously via dynamic JavaScript and is often gated by consent banners. A headless browser like Playwright is required to render the page fully before applying readability parsing to extract the text.

What's the best way to extract Apex or LWC documentation that renders dynamically?▼

The best way to extract dynamically rendered Apex or LWC documentation is using a headless Chrome browser to execute JavaScript requests, followed by Readability and Turndown libraries to parse the rendered HTML and convert it into clean Markdown.

Do I need Node.js to run a headless browser scraper for Salesforce documentation?▼

Yes, you need Node.js 18 or higher to run this headless browser scraper. The script relies on Playwright for browser automation, along with jsdom and Turndown, which require a compatible Node.js runtime environment to function correctly.

Can I use Playwright and Readability to bypass consent banners when scraping docs?▼

Yes, you can use Playwright with Readability to bypass consent banners when scraping documentation. Playwright renders the full page including the consent prompt, allowing the Readability parser to access and extract the actual documentation content behind it.

What limitations exist when converting dynamically rendered web pages to Markdown?▼

A limitation of converting dynamically rendered pages to Markdown is that interactive elements, complex tables, or embedded media may not translate cleanly into Markdown format. Additionally, the readability parser may strip out content it identifies as non-article, potentially losing technical context.