skills.rest
qodex-ai
qodex-aiOFFICIAL
Explore all skills by this author →
GitHub

content-harvest

Extract and parse article text, metadata, and structure from web URLs.

39|1|Updated Dec 21, 2025
Search Tags:#web scraping#article extraction#url parsing#content harvesting#text retrieval
One-click install
npx skills add https://github.com/qodex-ai/ai-agent-skills --skill content-harvest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: content-harvest
Source: https://github.com/qodex-ai/ai-agent-skills/tree/main/skills/content-harvest
Command: npx skills add https://github.com/qodex-ai/ai-agent-skills --skill content-harvest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires reader-cli, trafilatura, curl, python3, and includes scripts (resource) components.

What problem does it solve?

This Skill automates the extraction of clean, readable article content from web URLs, removing distracting elements like ads and navigation.

Core Features & Use Cases

  • Web Article Extraction: Retrieves text, metadata, and structured information from articles.
  • Content Cleaning: Removes clutter such as ads, navigation, and newsletter signups.
  • Formatting Preservation: Saves clean text while attempting to preserve formatting and context.
  • Use Case: When a user provides a URL to a blog post or news article and asks to "save this article as text," this Skill will fetch the content, clean it up, and save it to a file.

Quick Start

Use the content-harvest skill to extract the article from the provided URL.

Frequently Asked Questions about content-harvest

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract article content from a web URL and save it as clean text?▼

To extract article content from a web URL, the Skill fetches the page, removes ads and navigation clutter, and preserves text and formatting. It uses extraction tools like reader and trafilatura to deliver clean, readable text output.

Does web scraping with trafilatura preserve article formatting and metadata?▼

Yes, trafilatura web scraping preserves article formatting and context during extraction. It retrieves structured information and text while removing distracting elements like newsletter signups and navigation.

Can I use a fallback method for article extraction when standard parsing tools fail?▼

Yes, you can use a fallback curl-based method for article extraction when standard tools fail. This ensures deterministic task completion by retrieving raw web content directly as a reliable fallback mechanism.

What is the best way to parse news articles and remove webpage clutter automatically?▼

The best way to parse news articles and remove webpage clutter is using automated content harvesting tools. These tools extract readable text and metadata while stripping away ads, navigation, and other distracting page elements.

Do I need python3 installed to retrieve text from web sources using content harvesting?▼

Yes, python3 is required to retrieve text from web sources using this content harvesting Skill. The extraction process depends on python3 to run scripts that utilize reader-cli and trafilatura for parsing.

Why does web scraping fail to extract text from heavily dynamic web pages?▼

Web scraping can fail to extract text from dynamic pages when standard tools cannot render JavaScript. In such cases, the Skill relies on its curl-based fallback method to fetch raw content for deterministic task completion.

Related Skills

Explore More Content & Communication Skills→
cooopmaccooopmac

article-extractor

Extract and clean article text from URLs using Bash, trafilatura, or curl.

Community
Intermediate
alibabaalibaba
10.5k

pdf-extractor

Extract text, tables, and metadata from PDF documents into JSON.

Official
Intermediate
marktantongcomarktantongco
2

web-reader

Extract title, HTML, text, and publish time from web pages.

Community
Intermediate
feiskyerfeiskyer
1.6k

youtube-transcribe-skill

Extract YouTube video subtitles to a timestamped text file.

Community
Intermediate