WebMine

Mine web content from URLs or DuckDuckGo searches into a structured workspace.

1|Updated Mar 3, 2026

Search Tags:#research #search #web scraping #duckduckgo #content mining #web archive

Ready to run

npx skills add https://github.com/DaveSteadman/MiniAgentFramework --skill webmine

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: WebMine
Source: https://github.com/DaveSteadman/MiniAgentFramework/tree/main/code/skills/WebMine
Command: npx skills add https://github.com/DaveSteadman/MiniAgentFramework --skill webmine

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the process of fetching and saving web content, whether it's a direct URL or the results of a search query, organizing it into a structured research workspace.

Core Features & Use Cases

Direct URL Mining: Save the content of any web page to a specified domain and date.
Search and Save: Perform DuckDuckGo searches and save the results, optionally fetching and embedding article content.
Deep Mining: Extract individual articles from search results, ideal for in-depth research.
Use Case: You need to gather all articles about a specific topic published by a particular news source in a given month. Use mine_search_deep to find and save these articles into your "TechNews" research domain.

Quick Start

Use the WebMine skill to search for "latest AI advancements" and save the results into the "Technology" research domain.

Frequently Asked Questions about WebMine

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I save web content from a DuckDuckGo search into a structured workspace?▼

To save web content from a DuckDuckGo search, you can use the deep mining feature to fetch results, extract article prose, and organize everything into a structured research workspace sorted by domain and date.

Can I fetch and save article prose directly from a specific URL?▼

Yes, you can fetch and save article prose directly from a specific URL. The direct URL mining feature extracts the content and saves the raw results into your specified research domain.

What is the best way to archive all articles from a specific news source for a given month?▼

The best way to archive articles from a specific source is using deep article extraction. It searches DuckDuckGo, extracts individual articles based on configurable depth and limits, and saves them by domain and date.

Does web scraping with DuckDuckGo support extracting content from index pages?▼

Yes, DuckDuckGo web scraping supports handling index pages. It can navigate index pages and perform deep article extraction to fetch and embed the actual article content into your workspace.

How do I configure the depth and article limits when mining web content?▼

You can configure the depth and article limits during the search and save process. These settings control how many individual articles are extracted and how deeply the tool mines content from search results.

Why organize mined web content by domain and date in a research workspace?▼

Organizing mined web content by domain and date facilitates research archival and retrieval. This structure ensures raw results from direct URLs or searches are systematically categorized for easy access later.