skills.rest
ericwang915
ericwang915COMMUNITY
Explore all skills by this author →
GitHub

web_scraper

Extract text, links, and headings from web pages into JSON.

38|14|Updated Mar 1, 2026
Search Tags:#html#links#text#web#scrape#headings#content-extraction
One-click install
npx skills add https://github.com/ericwang915/PythonClaw --skill web-scraper-ericwang915
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web_scraper
Source: https://github.com/ericwang915/PythonClaw/tree/main/pythonclaw/templates/skills/data/scraper
Command: npx skills add https://github.com/ericwang915/PythonClaw --skill web-scraper-ericwang915

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4.

What problem does it solve?

Scraping and extracting content from web pages is often tedious and error-prone. This skill automates retrieving readable text, links, and headings to enable quick analysis and reuse.

Core Features & Use Cases

  • Text extraction: pull clean page text suitable for summaries and indexing
  • Link and heading extraction: collect links and heading structure for navigation and structure-aware processing
  • Structured output: returns JSON with title, text, links, and headings for downstream pipelines
  • Use Case: research assistants collecting data from multiple sites for a literature review

Quick Start

Install dependencies and run the scraper against a URL to retrieve text, links, and headings.

Frequently Asked Questions about web_scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract text and links from HTML web pages for analysis?▼

You can extract readable text, links, and headings from web pages using this scraper, which parses HTML with BeautifulSoup and returns structured JSON containing the title, text, links, and headings for quick analysis and reuse.

What is the best way to scrape headings and page structure for navigation processing?▼

Scraping headings and page structure is best handled by extracting the HTML heading tags directly. This skill collects heading structures alongside links, enabling structure-aware navigation and processing for research assistants collecting data.

Does this web scraper require BeautifulSoup to parse HTML content?▼

Yes, this web scraper requires BeautifulSoup and the requests library to fetch and parse HTML content. These dependencies handle the underlying HTML parsing and HTTP requests to retrieve structured page data.

Can I use this to collect data across multiple sites for a literature review?▼

Yes, this skill supports collecting data across multiple sites for a literature review by automating the retrieval of readable text, links, and headings from web pages, returning structured JSON for downstream research pipelines.

What format does the web scraper output for downstream pipelines?▼

The web scraper outputs structured JSON for downstream pipelines. This JSON includes the extracted page title, clean text content, links, and headings, ensuring the data is ready for indexing, summaries, or further analysis.

Related Skills

Explore More Data & Analytics Skills→
lisniuselisniuse
15

taoguba-crawler

Scrape Taoguba forum articles and generate HTML summaries with embedded images.

Community
Intermediate
vincentchanvincentchan
62

swipe-file-generator

Analyze content from URLs and compile analyses into a swipe file.

Community
Intermediate
KsawyouxKsawyoux

seek-and-analyze-video

Discovers and analyzes YouTube, TikTok, and Instagram videos via AI with Memories.ai integration.

Community
Advanced
appautomatonappautomaton
18

html-to-markdown

Converts HTML or live web pages into structured Markdown with metadata and link inventories.

Official
Intermediate