skills.rest
pragyaP17
pragyaP17COMMUNITY
Explore all skills by this author →
GitHub

scrape-webpage

Scrape webpage content, extract metadata, download images, and prepare for AEM Edge Delivery Services import.

Updated Sep 18, 2025
Search Tags:#data extraction#migration#Playwright#web scraping#metadata#image download#HTML cleanup
One-click install
npx skills add https://github.com/pragyaP17/universal-editor --skill scrape-webpage-pragyap17
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scrape-webpage
Source: https://github.com/pragyaP17/universal-editor/tree/main/.claude/skills/scrape-webpage
Command: npx skills add https://github.com/pragyaP17/universal-editor --skill scrape-webpage-pragyap17

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires playwright, sharp, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Manually extracting content, images, and metadata from a webpage for migration or analysis is a tedious and error-prone process. This skill automates comprehensive webpage scraping, saving time and ensuring accuracy.

Core Features & Use Cases

  • Full-Page Content Extraction: Loads pages in a headless browser, scrolls to trigger lazy-loaded content, and extracts cleaned HTML.
  • Image Capture & Optimization: Downloads all images, converts various formats (WebP, AVIF, SVG) to PNG, and replaces original URLs with local paths in the HTML.
  • Metadata Extraction: Gathers SEO and social sharing metadata (title, description, Open Graph, JSON-LD, canonical).
  • Path Generation: Automatically generates clean, standardized document paths for the migrated content.

Quick Start

Use the scrape-webpage skill to extract all content, images, and metadata from 'https://www.example.com/about-us' and save the output to the './import-work' directory.

Frequently Asked Questions about scrape-webpage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract webpage content and images for migration to AEM?▼

Web scraping automates extraction of full-page HTML, images, and metadata from live URLs. This skill loads pages in a headless browser, scrolls to capture lazy-loaded content, downloads images, and outputs cleaned HTML with local image references, metadata.json, and a structured directory ready for AEM Edge Delivery Services import.

Can I scrape and convert images in different formats like WebP and AVIF to PNG?▼

Yes. The skill downloads all images encountered on a webpage and uses Sharp to convert various formats—WebP, AVIF, SVG, and others—to PNG, then replaces original URLs with local file paths in the cleaned HTML output.

What metadata does webpage scraping extract for SEO and social sharing?▼

Web scraping captures title, description, Open Graph tags, JSON-LD structured data, and canonical URLs. The skill extracts and organizes this metadata into a metadata.json file alongside cleaned HTML and downloaded images for content migration workflows.

Do I need Node.js and Playwright to scrape webpages with this skill?▼

Yes. This skill requires Node.js, a Chromium-based Playwright runtime for headless browsing, and Sharp for image processing. These dependencies enable full-page rendering, lazy-load triggering, and format conversion in a single automated workflow.

How does lazy-loaded image capture work during webpage scraping?▼

The skill loads pages in a headless browser and scrolls through the entire page to trigger lazy-loaded images before extraction. This ensures all images—including those below the fold—are discovered and downloaded rather than missed during static HTML parsing.

What output does webpage scraping produce for content import?▼

Scraping generates cleaned HTML with local image paths, a metadata.json file containing SEO and social tags, a screenshot, an images/ directory mapping original URLs to local PNG files, and an auto-generated document path for the migrated content.

Related Skills

Explore More Data & Analytics Skills→
Permit-JunctionPermit-Junction

scrape-webpage

Scrape webpage content, extract metadata, and download images for AEM Edge Delivery Services.

Official
Intermediate
jfoxxjfoxx

scrape-webpage

Scrape webpage content, extract metadata, and download images for AEM Edge Delivery Services.

Community
Intermediate
initialyzeinitialyze

scrape-webpage

Scrape webpages, extract metadata, download images, and generate migration-ready outputs.

Official
Advanced
badlogicbadlogic
2.3k

browser-tools

Automate Chrome browser control and content extraction via the Chrome DevTools Protocol.

Community
Advanced