web-scraper

Name: web-scraper
Availability: InStock
Author: caoqiubozhangchenqin2

Extract and clean web page content from static sites and SPAs.

Updated Apr 12, 2026

Search Tags:#metadata #content extraction #playwright #scraping #web #scrapy

Ready to run

npx skills add https://github.com/caoqiubozhangchenqin2/qclaw --skill web-scraper-caoqiubozhangchenqin2

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/caoqiubozhangchenqin2/qclaw/tree/main/skills/web-scraper
Command: npx skills add https://github.com/caoqiubozhangchenqin2/qclaw --skill web-scraper-caoqiubozhangchenqin2

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, playwright, scrapy, trafilatura, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables automated, accurate extraction of web page content by employing multi-layered fallback strategies, handling JavaScript-rendered sites, and removing boilerplate noise.

Core Features & Use Cases

Content Extraction: Retrieves meaningful article or page content from static or dynamic websites, including SPA and lazy-loaded pages.
Metadata and Schema: Extracts structured information such as author, publication date, and schema.org data to facilitate data organization.
Use Case: Collects news articles or blog posts from multiple sources for data analysis or digital archiving, ensuring high-quality content regardless of site complexity.

Quick Start

Input a URL to fetch and parse its main content and metadata for downstream analysis.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract web content from JavaScript-rendered or single-page application sites?▼

To extract web content from JavaScript-rendered sites, this tool uses Playwright as a fallback strategy to handle dynamic loading and SPAs. It ensures robust content retrieval by cascading through requests and Scrapy modules when static fetching fails.

Can I extract structured metadata like author and publication date from news articles?▼

Yes, you can extract structured metadata like author and publication date by utilizing the built-in schema.org recognition feature. It automatically identifies and parses structured information to facilitate data organization for news aggregation workflows.

What's the best way to scrape boilerplate-free article text for data aggregation?▼

The best way to scrape boilerplate-free article text is using this multi-layered cascade extraction tool. It automatically removes navigation and ad noise, retrieving meaningful content from static or dynamic websites for high-quality digital archiving.

Does this web scraper work with both static pages and lazy-loaded content?▼

Yes, this web scraper works with both static pages and lazy-loaded content by employing multi-layered fallback strategies. It adapts its fetching mechanism across requests, Playwright, and Scrapy to ensure robust extraction regardless of site complexity.

How do I start scraping a web page URL for downstream analysis?▼

To start scraping a web page URL for downstream analysis, simply input the target URL. The tool will automatically fetch, parse, and clean the main content and metadata for immediate use in your data workflows.