skills.rest
quazfenton
quazfentonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

web-scraper

Crawl websites and extract structured data using tiered scraping tools.

1|Updated Jun 13, 2025
Search Tags:#markdown#json#data-extraction#web-scraping#batch-processing#crawl4ai#css-selectors
One-click install
npx skills add https://github.com/quazfenton/binG --skill web-scraper-quazfenton
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/quazfenton/binG/tree/main/.agents/skills/web_scraper
Command: npx skills add https://github.com/quazfenton/binG --skill web-scraper-quazfenton

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires httpx, beautifulsoup4, crawl4ai, and includes scripts (resource) components.

What problem does it solve?

This skill eliminates the tedious work of manually browsing pages to collect data by providing a tiered web scraping workflow that can fetch, parse, and extract structured information from websites.

Core Features & Use Cases

  • Tiered approach: perform quick HTTP fetches with httpx/BeautifulSoup, switch to CSS-based extraction with a schema, render JS with Crawl4AI, or run deep crawls and LLM-powered extractions for complex sites.
  • Use cases include product pages data, articles with structured metadata, and batch harvesting across many URLs for analytics.
  • Output formats include markdown, JSON, or CSV-friendly structures, enabling easy integration into data pipelines.

Quick Start

Fetch a webpage and extract data using the simplest fetch first, or apply CSS selectors, a JSON schema, or an LLM extraction strategy for structured results.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured data from web pages using CSS selectors?▼

You can extract structured data from web pages by applying CSS selectors to fetch and parse HTML. The skill uses httpx and BeautifulSoup for quick fetches, allowing precise CSS-based extraction to pull targeted elements into a structured format.

What is the best way to scrape JavaScript-rendered website content?▼

Scraping JavaScript-rendered website content is handled via Crawl4AI, which uses Playwright-based rendering to load dynamic pages. This allows the scraper to extract fully rendered DOM structures that static HTTP requests cannot access.

Can I crawl multiple URLs for batch processing and export JSON?▼

Yes, you can crawl multiple URLs for batch processing and export the results as JSON. The skill supports batch harvesting across many URLs and outputs data in markdown, JSON, or CSV-friendly structures for easy integration.

How do I use LLM-driven extraction for complex web scraping?▼

LLM-driven extraction for complex web scraping works by applying a JSON schema or extraction strategy to the fetched content. This allows the tool to parse intricate pages and return highly structured results without manual element mapping.

Does web scraping with Crawl4AI support deep site crawling?▼

Web scraping with Crawl4AI supports deep site crawling to discover and extract data across entire domains. It combines JS rendering with LLM-powered extraction to navigate complex site architectures and harvest structured data.

Related Skills

Explore More Data & Analytics Skills→
robit-manrobit-man

doc-scraper

Scrape documentation websites into categorized Markdown reference files.

Community
Advanced
89jobrien89jobrien
4

file-converter

Generate Python code to convert files between document, data, and image formats.

Community
Advanced
bruno0henriquebruno0henrique

image-creator

Render self-contained HTML and CSS into PNG images using Playwright.

Community
Advanced
Z43LZ43L

scrapling

Extract web content from static, dynamic, and Cloudflare-protected pages.

Community
Advanced