scrape-webpage

To scrape a webpage for migration, this tool renders the target URL with Playwright, extracts Open Graph and JSON-LD metadata, downloads images, and outputs a metadata.json file alongside cleaned HTML. It automates the analysis required to prepare page content for import workflows.

To map and download images for an AEM migration, the scrape-webpage script captures all images from the target page, converts formats to PNG using Sharp, and saves them locally. It generates a mapping from original URLs to local paths within the metadata.json output.

Yes, you can use Playwright to generate screenshots and clean HTML during a content audit. The script renders the webpage, captures a full screenshot, cleans the HTML for migrated content, and packages the results for downstream migration-ready analysis.

Yes, you need Node.js and Playwright installed to scrape webpages and extract Open Graph data. The script runs via Node.js and requires Playwright for rendering JavaScript-heavy pages, plus Sharp for processing and converting downloaded images to PNG.

The best way to prepare a migration-ready analysis is running an automated script that extracts HTML, metadata, and screenshots, then localizes images. This generates a structured metadata.json file with document paths and image mappings for seamless downstream import.

When scraping JavaScript-heavy webpages for JSON-LD metadata, limitations depend on Playwright's rendering capabilities and the target site's bot protection. While it executes JavaScript to render dynamic content, strict anti-scraping measures or complex authentication barriers may prevent successful extraction.