web-scraper

Name: web-scraper
Availability: InStock
Author: dayuer

Extract web content into Markdown or JSON using Python scripts.

Updated Aug 23, 2026

Search Tags:#markdown #html #links #data-extraction #web-scraping #css-selectors

Ready to run

npx skills add https://github.com/dayuer/survival --skill web-scraper-dayuer

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/dayuer/survival/tree/main/nanobot/workspace/skills/web-scraper
Command: npx skills add https://github.com/dayuer/survival --skill web-scraper-dayuer

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires httpx, beautifulsoup4, readability-lxml, and includes scripts (resource) components.

What problem does it solve?

AI agents often need to digest and structure information from web pages. This skill automates the process of fetching HTML content, cleaning noise, and presenting it as readable Markdown or machine-parsable JSON.

Core Features & Use Cases

Content extraction: fetch page text, metadata, links, and images with optional CSS selectors to target specific sections.
Flexible output: convert to readable Markdown, raw HTML, or structured JSON for downstream processing.
Use Case: quickly extract article content and key metadata for summarization, indexing, or knowledge base ingestion.

Quick Start

Execute the scrape script against a target URL to retrieve the main content and export it in your preferred format.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract article text and metadata from web pages into Markdown?▼

You can extract article text and metadata by fetching HTML content and converting it into clean Markdown or structured JSON. This process removes page noise and isolates specific sections using optional CSS selectors for precise data extraction.

What's the best way to scrape specific HTML sections using CSS selectors?▼

The best way to scrape specific HTML sections is by applying optional CSS selectors to target precise page elements. This method filters out noise and ensures only the desired text, links, or images are extracted and structured for your agent workflow.

Can I use BeautifulSoup and readability-lxml to extract structured JSON from websites?▼

Yes, you can use BeautifulSoup and readability-lxml to extract structured JSON from websites. These libraries parse raw HTML, clean the content, and output structured metadata including text, links, and images suitable for machine ingestion.

Does web scraping with Python scripts work for extracting images and links?▼

Yes, web scraping with Python scripts works for extracting images and links. By processing the fetched HTML, the scripts can isolate and return these specific elements alongside the main article text and metadata in your chosen output format.

How do I convert raw HTML into clean text for AI agent ingestion?▼

To convert raw HTML into clean text for AI agent ingestion, you run a Python script that fetches the page and applies readability algorithms. This strips out page boilerplate and delivers clean Markdown or JSON for immediate summarization or indexing.

What are the limitations of using readability-lxml for web content extraction?▼

A limitation of using readability-lxml for web content extraction is its reliance on standard HTML structures, meaning heavily JavaScript-rendered pages may not parse correctly. It excels at standard article text but may miss dynamically loaded metadata.