skills.rest
huangrt01
huangrt01COMMUNITY
Explore all skills by this author →
GitHub

playwright-scraper

Automate dynamic web page scraping with Playwright and save HTML or JSON output.

4.0k|479|Updated Apr 16, 2020
Search Tags:#playwright#web-scraping#html-output#browser-automation#dynamic-content#json-output
One-click install
npx skills add https://github.com/huangrt01/CS-Notes --skill playwright-scraper-huangrt01
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: playwright-scraper
Source: https://github.com/huangrt01/CS-Notes/tree/main/.trae/openclaw-skills/playwright-scraper
Command: npx skills add https://github.com/huangrt01/CS-Notes --skill playwright-scraper-huangrt01

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires playwright.

What problem does it solve?

Automates dynamic web page scraping by driving a real browser with Playwright to render JavaScript-heavy sites and extract content.

Core Features & Use Cases

  • Simulates browser execution to fetch pages that rely on client-side rendering.
  • Supports headless and headed modes, configurable wait times, and user-agent customization.
  • Outputs HTML and JSON content for downstream processing.
  • Use Case: Scrape a dynamic site like GitHub or Zhihu to collect rendered HTML for indexing or archival purposes.

Quick Start

Run python3 main.py <url> to fetch the page content and optionally save HTML or JSON output.

Frequently Asked Questions about playwright-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape dynamic web pages that require JavaScript rendering?▼

Scraping dynamic web pages with JavaScript rendering requires a real browser automation tool like Playwright to execute client-side scripts and capture the fully loaded HTML.

Can I extract dynamic content and save the output as HTML or JSON?▼

Yes, you can extract dynamic content and save the output as HTML or JSON formats, allowing you to store the rendered page structure and data for downstream indexing or archival processing.

How do I set up Playwright to wait for specific selectors during browser automation?▼

Setting up Playwright to wait for specific selectors involves configuring wait times in your scraping script, ensuring the browser pauses until targeted dynamic elements fully render before extraction.

Does this web scraping tool support headless mode and custom User-Agent settings?▼

Yes, this web scraping tool supports headless mode and custom User-Agent settings, allowing you to run browser automation without a UI and simulate specific browser identities to access content.

What is the best way to scrape sites like GitHub or Zhihu for rendered HTML?▼

The best way to scrape sites like GitHub or Zhihu for rendered HTML is using browser automation to simulate real execution, capturing JavaScript-rendered content directly as structured HTML or JSON.

Related Skills

Explore More Software Engineering Skills→
mosheshammoshesham

playwright-pro

Automate Playwright-based browser scraping of dynamic web pages.

Community
Intermediate
IgosukiIgosuki
1

web-scraper

Extract clean text from web pages via concurrent scraping with deduplication.

Community
Intermediate
abdullah1854abdullah1854
1

chrome-data-extract

Generate JavaScript snippets to extract structured data from HTML tables, forms, and lists.

Community
Intermediate
masharrattmasharratt
14

CFN Marketing Analytics Data Skill

Retrieve marketing analytics data via authenticated n8n workflow HTTP POST.

Community
Intermediate