skills.rest
cdeistopened
cdeistopenedCOMMUNITY
Explore all skills by this author →
GitHub

web-scrape

Scrape web pages, convert HTML to Markdown, and generate AI summaries.

24|6|Updated Nov 15, 2025
Search Tags:#markdown#web scraping#content extraction#batch processing#ai summary
One-click install
npx skills add https://github.com/cdeistopened/skill-stack --skill web-scrape-cdeistopened
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scrape
Source: https://github.com/cdeistopened/skill-stack/tree/main/public/skills/web-scrape
Command: npx skills add https://github.com/cdeistopened/skill-stack --skill web-scrape-cdeistopened

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill automates the process of fetching web pages, converting their HTML content into clean, usable Markdown, and optionally generating AI-powered summaries.

Core Features & Use Cases

  • Web to Markdown Conversion: Extracts the main article content from web pages, stripping away unnecessary navigation, ads, and footers.
  • AI Summarization: Generates concise summaries and topic tags for scraped content using AI.
  • Batch Processing: Supports scraping multiple URLs from a text file.
  • Use Case: Quickly save articles from the web for offline reading or research, ensuring the content is in a clean, portable Markdown format.

Quick Start

Use the web-scrape skill to scrape the content from the URL https://example.com/article.

Frequently Asked Questions about web-scrape

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert web page HTML to clean Markdown?▼

You can convert web pages to Markdown by fetching URLs and extracting the main article content, which strips away navigation, ads, and footers. This process yields clean, portable Markdown text suitable for offline reading or research corpora.

Can I batch process multiple URLs for web scraping?▼

Yes, you can batch process web scraping by providing multiple URLs from a text file. The Skill iterates through the list to fetch web pages and convert the HTML content into clean Markdown for each URL automatically.

Do I need a Gemini API key to generate AI summaries of scraped content?▼

Yes, you need a Gemini API key to generate AI summaries of scraped content. While HTML to Markdown conversion works independently, the AI summarization feature uses this key to produce concise summaries and topic tags.

What is the best way to save web articles for offline reading in Markdown?▼

The best way to save web articles for offline reading in Markdown is using a scraping tool that parses HTML and extracts metadata. This Skill fetches web pages, removes unnecessary elements, and converts the main content into a clean Markdown format.

Does web scraping work without AI summarization for building a research corpus?▼

Yes, web scraping works without AI summarization for building a research corpus. You can fetch web pages and convert HTML to clean Markdown independently, bypassing the optional AI summary feature and its Gemini API key requirement.

Related Skills

Explore More Content & Communication Skills→
guyruguyru

ddg-search

Search the web via DuckDuckGo HTML and output titles, URLs, and snippets.

Community
Intermediate
wangjiawei508wangjiawei508
2

docx-generation

Convert Markdown-formatted text into structured Word documents.

Community
Advanced
u9401066u9401066

pdf-reader

Extract text, figures, and citations from PDFs into Markdown.

Community
Intermediate
twjarviszyra-webtwjarviszyra-web
3

scrapling

Extract web content from static, dynamic, and protected pages via Scrapling APIs.

Community
Advanced