web-scraper

Scrape web pages into markdown or structured JSON using CSS schemas.

Updated Mar 8, 2026

Search Tags:#data extraction #web scraping #batch processing #json schema #crawling #javascript rendering #llm extraction

Ready to run

npx skills add https://github.com/ankitjh4/skill-builder --skill web-scraper-ankitjh4

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/ankitjh4/skill-builder/tree/main/web-scraper
Command: npx skills add https://github.com/ankitjh4/skill-builder --skill web-scraper-ankitjh4

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires httpx, bs4, crawl4ai, and includes scripts (resource) components.

What problem does it solve?

Web pages are hard to turn into reliable, structured outputs because content formats vary and data often spans multiple pages or dynamic views.

Core Features & Use Cases

Tiered scraping pipeline: Switches between fast HTML fetching, CSS-based extraction, optional JavaScript rendering, deep BFS crawling, and LLM-driven extraction depending on what the target page needs.
Structured extraction: Extracts data via a JSON CSS schema (base selector + fields) or by selecting elements with a CSS selector.
Batch and crawling support: Scrape many URLs from a file and crawl internal links to a target depth/page limit for multi-page collection.
Output flexibility: Produces markdown content, extracted JSON data, or plain text, with an option to save results to a file.

Quick Start

Use the web-scraper skill to scrape a single page into markdown with: python3 /home/workspace/Skills/web-scraper/scripts/scraper.py scrape https://example.com

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract structured data from a web page using a JSON schema?▼

You can extract structured data from a web page by providing a JSON CSS schema that defines a base selector and specific fields. The scraper fetches the HTML and returns the matched elements as structured JSON for downstream analysis.

What's the best way to scrape multiple URLs or crawl internal links in batch?▼

Batch scraping and crawling internal links is supported by processing many URLs from a file or crawling internal links to a target depth and page limit. This multi-page collection outputs markdown, plain text, or extracted JSON.

Does this web scraper support JavaScript rendering for dynamic content?▼

Yes, JavaScript rendering is supported as an optional tier in the scraping pipeline. It switches from fast HTML fetching to rendering dynamic views when the target page requires it to load content.

Can I use LLM-driven extraction on crawled web page content?▼

LLM-driven extraction is supported on crawled content. After fetching pages via httpx and BeautifulSoup, you can route the content through LLM extraction to turn unstructured web pages into structured outputs.

What formats can I save the scraped web data to?▼

Scraped web data can be output as markdown content, structured JSON data, or plain text. You can also save the scraping results directly to a file for later use.

When should I avoid using JavaScript rendering for web scraping?▼

You should avoid JavaScript rendering when scraping static HTML pages to maximize speed. The pipeline defaults to fast HTML fetching with httpx and BeautifulSoup, only enabling rendering for dynamic views that require it.