skills.rest
eesha000009-dev
eesha000009-devCOMMUNITY
Explore all skills by this author →
GitHub

web-reader

Extract article content, metadata, and cleaned HTML from web pages.

1|Updated Aug 3, 2025
Search Tags:#html#caching#metadata#web-scraping#content-extraction#rss#web-reader
One-click install
npx skills add https://github.com/eesha000009-dev/Exam-Prep100 --skill web-reader-eesha000009-dev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-reader
Source: https://github.com/eesha000009-dev/Exam-Prep100/tree/main/skills/web-reader
Command: npx skills add https://github.com/eesha000009-dev/Exam-Prep100 --skill web-reader-eesha000009-dev

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires z-ai-web-dev-sdk, and includes scripts (resource) components.

What problem does it solve?

Web pages often contain valuable article content and metadata buried in HTML, which is time-consuming to extract and normalize manually; this Skill automates fetching, extracting, and cleaning page content so applications can consume structured article data reliably.

Core Features & Use Cases

  • Automatic Content Extraction: Fetches a URL and returns title, main HTML, plain text, publication time, and additional metadata.
  • SDK and CLI Options: Provides both z-ai-web-dev-sdk integration examples for backend code and CLI usage for quick one-off extraction tasks.
  • Batch Processing & Pipelines: Demonstrates caching, rate limiting, parallel fetches, RSS/feed reading, and custom processing pipelines for aggregation, monitoring, and analysis.
  • Use Case: Aggregate news articles from multiple sources, extract article text and metadata for indexing, or power a scheduled content fetcher for monitoring changes.

Quick Start

Use the web-reader skill to extract title, html, text, and publish_time from https://example.com/article and return the result as JSON.

Frequently Asked Questions about web-reader

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract clean article text and metadata from arbitrary web pages?▼

To extract article content from web pages, this Skill fetches a URL and parses the HTML to return structured JSON fields including title, cleaned HTML, plain text, publish_time, and url. It automates fetching and normalizing web content for backend consumption.

Can I use web scraping to process RSS feeds and aggregate news articles in batch?▼

Yes, you can process RSS feeds and aggregate news articles in batch by applying parallel fetches, caching, and rate limiting. This approach allows you to extract content from multiple sources simultaneously for aggregation and monitoring workflows.

Does the z-ai-web-dev-sdk support backend-only content extraction for Node.js applications?▼

The z-ai-web-dev-sdk supports backend-only content extraction by providing integration examples for server-side code. The web-reader Skill requires backend SDK usage to fetch URLs and return structured article data while handling error and caching strategies.

What is the best way to get structured JSON output like title and publish_time from scraped HTML?▼

The best way to get structured JSON output from scraped HTML is using a content extraction tool that automatically parses web pages and returns normalized fields. This ensures you receive reliable title, text, and publish_time data without manual HTML cleaning.

What are the limitations of using automated web scraping for content extraction?▼

Limitations of automated web scraping include encountering fetch errors on dynamically rendered pages and hitting rate limits on target servers. The Skill handles basic error and caching strategies to mitigate these issues, but complex JavaScript-heavy pages may still fail extraction.

Related Skills

Explore More Software Engineering Skills→
zakarooozakarooo

web-reader

Extract web page content and metadata via the page_reader function.

Community
Intermediate
InverterNetworkInverterNetwork

blogwatcher

Manage RSS/Atom feed updates and track article read status via blogwatcher-cli.

Official
Intermediate
latestaiagentslatestaiagents
5

llm-rate-limiting

Implement token bucket, sliding window, and concurrency rate limiting for LLM APIs.

Official
Advanced
mazreanmazrean

building-static-site-rss-feeds

Generate RSS 2.0, Atom 1.0, and JSON feeds for static sites.

Community
Intermediate