web-scraper

Extract clean text from web pages via concurrent scraping with deduplication.

1|Updated Oct 21, 2025

Search Tags:#concurrency #metadata #deduplication #web #scrape #text-extraction #data-archival

Ready to run

npx skills add https://github.com/Igosuki/claude-skills --skill web-scraper-igosuki

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/Igosuki/claude-skills/tree/main/web-scraper
Command: npx skills add https://github.com/Igosuki/claude-skills --skill web-scraper-igosuki

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires aiohttp, beautifulsoup4, lxml, aiofiles, and includes scripts (resource) components.

What problem does it solve?

This skill extracts and stores clean text content from web pages through concurrent scraping with deduplication, content filtering, and structured output.

Core Features & Use Cases

Concurrent scraping with configurable depth and a cap on pages to prevent runaway crawls.
Automatic deduplication, domain controls, and safe URL normalization to build a coherent local archive.
Content extraction that removes scripts, navigation, and boilerplate, saving readable text and metadata for downstream analysis.
Use cases: building offline documentation archives, collecting training data for NLP, and gathering research content from multiple domains.

Quick Start

Run a small crawl with a single URL: python scripts/scrape.py https://example.com output/ --depth 2

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract clean text from web pages for NLP training data?▼

You can extract clean text from web pages by running concurrent scraping scripts that remove scripts, navigation, and boilerplate, saving readable text and metadata for downstream analysis.

How do I archive documentation from multiple domains concurrently?▼

Archive documentation from multiple domains by running asynchronous web scrapers with configurable depth, concurrency caps, and domain controls to build a coherent local archive.

Does asynchronous scraping with aiohttp support deduplication and URL normalization?▼

Yes, asynchronous scraping with aiohttp supports automatic deduplication and safe URL normalization to ensure your local archive contains only unique, properly formatted pages.

What is the best way to prevent runaway web crawls when gathering research content?▼

Prevent runaway web crawls by configuring a cap on maximum pages and setting a specific crawl depth, which limits how far the scraper traverses from the starting URL.

How do I generate metadata for scraped web content?▼

Generate metadata for scraped web content by using tools that automatically produce a _metadata.json file containing per-page statistics alongside the extracted text output files.

Can I use BeautifulSoup and lxml for offline content archival?▼

Yes, you can use BeautifulSoup and lxml for offline content archival by parsing HTML asynchronously and writing the filtered text results to local storage with aiofiles.