skills.rest
Igosuki
IgosukiCOMMUNITY
Explore all skills by this author →
GitHub

web-scraper

Extract clean text from web pages via concurrent scraping with deduplication.

1|Updated Oct 21, 2025
Search Tags:#concurrency#metadata#deduplication#web#scrape#text-extraction#data-archival
One-click install
npx skills add https://github.com/Igosuki/claude-skills --skill web-scraper-igosuki
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/Igosuki/claude-skills/tree/main/web-scraper
Command: npx skills add https://github.com/Igosuki/claude-skills --skill web-scraper-igosuki

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires aiohttp, beautifulsoup4, lxml, aiofiles, and includes scripts (resource) components.

What problem does it solve?

This skill extracts and stores clean text content from web pages through concurrent scraping with deduplication, content filtering, and structured output.

Core Features & Use Cases

  • Concurrent scraping with configurable depth and a cap on pages to prevent runaway crawls.
  • Automatic deduplication, domain controls, and safe URL normalization to build a coherent local archive.
  • Content extraction that removes scripts, navigation, and boilerplate, saving readable text and metadata for downstream analysis.
  • Use cases: building offline documentation archives, collecting training data for NLP, and gathering research content from multiple domains.

Quick Start

Run a small crawl with a single URL: python scripts/scrape.py https://example.com output/ --depth 2

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract clean text from web pages for NLP training data?▼

You can extract clean text from web pages by running concurrent scraping scripts that remove scripts, navigation, and boilerplate, saving readable text and metadata for downstream analysis.

How do I archive documentation from multiple domains concurrently?▼

Archive documentation from multiple domains by running asynchronous web scrapers with configurable depth, concurrency caps, and domain controls to build a coherent local archive.

Does asynchronous scraping with aiohttp support deduplication and URL normalization?▼

Yes, asynchronous scraping with aiohttp supports automatic deduplication and safe URL normalization to ensure your local archive contains only unique, properly formatted pages.

What is the best way to prevent runaway web crawls when gathering research content?▼

Prevent runaway web crawls by configuring a cap on maximum pages and setting a specific crawl depth, which limits how far the scraper traverses from the starting URL.

How do I generate metadata for scraped web content?▼

Generate metadata for scraped web content by using tools that automatically produce a _metadata.json file containing per-page statistics alongside the extracted text output files.

Can I use BeautifulSoup and lxml for offline content archival?▼

Yes, you can use BeautifulSoup and lxml for offline content archival by parsing HTML asynchronously and writing the filtered text results to local storage with aiofiles.

Related Skills

Explore More Data & Analytics Skills→
guia-matthieuguia-matthieu
145

link-checker

Crawls websites and detects broken links and 404 errors, generating CSV reports of inaccessible URLs.

Community
Basic
ericwang915ericwang915
38

pdf_reader

Extract text and metadata from PDFs using PyPDF2.

Community
Intermediate
otrebuotrebu
4

web-to-markdown

Batch-convert JavaScript-rendered web pages to a timestamped Markdown file.

Community
Advanced
jiyangnanjiyangnan
2

kb-fetch

Fetch and extract readable text from URLs for knowledge base ingestion.

Community
Intermediate