salesforce-developer-site-scraper

You can convert Salesforce developer documentation to Markdown by using a headless browser to render dynamic pages and readability parsing to extract content. This approach bypasses consent banners and asynchronous loading to produce clean Markdown files for offline access.

Scraping Salesforce docs fails with standard requests because the content loads asynchronously via dynamic JavaScript and is often gated by consent banners. A headless browser like Playwright is required to render the page fully before applying readability parsing to extract the text.

The best way to extract dynamically rendered Apex or LWC documentation is using a headless Chrome browser to execute JavaScript requests, followed by Readability and Turndown libraries to parse the rendered HTML and convert it into clean Markdown.

Yes, you need Node.js 18 or higher to run this headless browser scraper. The script relies on Playwright for browser automation, along with jsdom and Turndown, which require a compatible Node.js runtime environment to function correctly.

Yes, you can use Playwright with Readability to bypass consent banners when scraping documentation. Playwright renders the full page including the consent prompt, allowing the Readability parser to access and extract the actual documentation content behind it.

A limitation of converting dynamically rendered pages to Markdown is that interactive elements, complex tables, or embedded media may not translate cleanly into Markdown format. Additionally, the readability parser may strip out content it identifies as non-article, potentially losing technical context.