Extract clean article text and structured news metadata from URLs into JSON.
npx skills add https://github.com/droxey/skills --skill web-scraper-droxey
Please help me install this Agent Skill. Skill: web-scraper Source: https://github.com/droxey/skills/tree/main/web-scraper Command: npx skills add https://github.com/droxey/skills --skill web-scraper-droxey
This Skill solves the problem of getting reliable article text and metadata from inconsistent web pages, including JS-heavy sites, without dumping raw HTML or guessing blindly.
Use the web-scraper skill to extract article text and structured metadata from a set of news URLs, detect paywalls, and output JSON with quality and provenance fields.
High-intent search queries and answers about installing and using this skill.
To extract clean article text and structured metadata from multiple news URLs, this skill uses a multi-strategy cascade that fetches static HTML and escalates to Playwright headless Chromium rendering only when needed, outputting structured JSON with titles, authors, dates, and tags.
Yes, web scraping with Playwright handles JavaScript-rendered SPA pages by utilizing a headless Chromium browser to execute client-side scripts, ensuring dynamically loaded content is fully rendered before the text extraction and boilerplate removal stages begin.
The best way to remove boilerplate HTML and get structured JSON is using a staged extraction pipeline that detects news content via Schema.org and Open Graph tags, normalizes the main text, and outputs clean data with provenance and quality assessment fields.
You can extract named entities like people, organizations, locations, and events by running optional LLM entity extraction on the cleaned article text, which ensures the model processes normalized content rather than messy raw HTML.
Yes, this content extraction method enforces safety constraints including robots.txt respect and rate limiting, allowing you to reliably process bulk lists of up to 500 URLs without triggering server blocks or violating crawling protocols.
News detection identifies paywalls and article structures by analyzing URL patterns, parsing Schema.org and Open Graph metadata, and applying content heuristics to validate whether a page is a true article before extracting the main text.
kitaekattConvert local HTML files to PDFs using headless Chromium via Playwright.
ptreezhCollect business ecosystem entities and relationships into structured JSON with quality metrics.
bahayonghangExtract text, entities, and relationships from images, PDFs, and charts.
kevinnftExtract structured data from static HTML, JS-rendered pages, and hybrid sites.
