Extract web content into Markdown or JSON using Python scripts.
npx skills add https://github.com/dayuer/survival --skill web-scraper-dayuer
Please help me install this Agent Skill. Skill: web-scraper Source: https://github.com/dayuer/survival/tree/main/nanobot/workspace/skills/web-scraper Command: npx skills add https://github.com/dayuer/survival --skill web-scraper-dayuer
AI agents often need to digest and structure information from web pages. This skill automates the process of fetching HTML content, cleaning noise, and presenting it as readable Markdown or machine-parsable JSON.
Execute the scrape script against a target URL to retrieve the main content and export it in your preferred format.
High-intent search queries and answers about installing and using this skill.
You can extract article text and metadata by fetching HTML content and converting it into clean Markdown or structured JSON. This process removes page noise and isolates specific sections using optional CSS selectors for precise data extraction.
The best way to scrape specific HTML sections is by applying optional CSS selectors to target precise page elements. This method filters out noise and ensures only the desired text, links, or images are extracted and structured for your agent workflow.
Yes, you can use BeautifulSoup and readability-lxml to extract structured JSON from websites. These libraries parse raw HTML, clean the content, and output structured metadata including text, links, and images suitable for machine ingestion.
Yes, web scraping with Python scripts works for extracting images and links. By processing the fetched HTML, the scripts can isolate and return these specific elements alongside the main article text and metadata in your chosen output format.
To convert raw HTML into clean text for AI agent ingestion, you run a Python script that fetches the page and applies readability algorithms. This strips out page boilerplate and delivers clean Markdown or JSON for immediate summarization or indexing.
A limitation of using readability-lxml for web content extraction is its reliance on standard HTML structures, meaning heavily JavaScript-rendered pages may not parse correctly. It excels at standard article text but may miss dynamically loaded metadata.
vibewatchFetch URLs and extract readable text from HTML pages and PDFs.
1012LoninIndex large-scale documents and perform semantic search with persistent memory.
BenAIOSAnalyze web page images for SEO and performance optimization.
appautomatonConverts HTML or live web pages into structured Markdown with metadata and link inventories.
