skills.rest
roach88
roach88COMMUNITY
Explore all skills by this author →
GitHub

scraping-websites

Crawl static and JavaScript-rendered websites and extract structured data.

2|Updated Feb 23, 2026
Search Tags:#firecrawl#web scraping#web data#javascript rendering#site crawling#url mapping#structured data extraction
One-click install
npx skills add https://github.com/roach88/claude-config --skill scraping-websites
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scraping-websites
Source: https://github.com/roach88/claude-config/tree/main/skills/scraping-websites
Command: npx skills add https://github.com/roach88/claude-config --skill scraping-websites

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the manual effort of collecting web content from static and JavaScript-rendered sites, and extracting structured data from web pages without writing custom scraping code.

Core Features & Use Cases

  • Full Site Crawling & Single-Page Scraping: Collect content from entire websites or individual URLs, with support for JavaScript-rendered pages.
  • URL Mapping & Structured Extraction: Discover all pages on a site via sitemap generation, and extract targeted structured data using LLM-powered schemas.
  • Use Case: For example, you can crawl a product documentation site to build a local searchable knowledge base, or extract product pricing and details from an e-commerce store's pages into structured JSON for analysis.

Quick Start

Use the scraping-websites skill to crawl the public API documentation at https://docs.example.com, limit the crawl to 50 pages, and save all main content to a local JSON file.

Frequently Asked Questions about scraping-websites

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape JavaScript-rendered websites without writing custom code?▼

You can scrape JavaScript-rendered websites automatically using this Skill to extract structured content without writing custom scraper code. It handles dynamic web content directly and outputs the scraped data in markdown or JSON formats.

What is the best way to crawl an entire documentation site into a local knowledge base?▼

Site crawling for a local knowledge base is achieved by using the full site crawling feature to collect content from an entire documentation site. You can limit the crawl scope to a specific number of pages and save the output as structured JSON.

Can I extract structured data from web pages into JSON using an LLM schema?▼

Yes, LLM-powered structured data extraction allows you to extract targeted information from web pages into structured JSON. You define schemas to collect specific data like product pricing and details from e-commerce pages.

How do I generate a URL sitemap to discover all pages on a site?▼

URL mapping generates a sitemap to discover all pages on a target site. This feature identifies available crawl paths so you can filter and limit the crawl scope before extracting content from the discovered URLs.

Does web scraping work with static sites and dynamic content filtering?▼

Web scraping works with both static and JavaScript-rendered sites while supporting dynamic content filtering. You can control the process by filtering crawl paths and limiting the crawl scope to specific sections of the website.

Related Skills

Explore More Data & Analytics Skills→
t0labt0lab
2

crawl4ai

Extract structured data and full-text content from URLs, videos, and documents.

Official
Advanced
drillandrillan

crawl

Crawl websites and save pages as local markdown files.

Community
Intermediate
ChimeraFoundationaChimeraFoundationa

nemo-curator

Automate GPU-accelerated curation of multimodal training datasets with RAPIDS integration.

Community
Advanced
abdullah1854abdullah1854
1

crawl-cli

Crawl websites and extract content into Markdown, JSON, or text.

Community
Basic