skills.rest
christopherlouet
christopherlouetCOMMUNITY
Explore all skills by this author →
GitHub

web-scraping

Extract web content and crawl sites into markdown or JSON.

5|Updated Jan 15, 2026
Search Tags:#structured data#firecrawl#content extraction#scraping#web#crawl
One-click install
npx skills add https://github.com/christopherlouet/claude-base --skill web-scraping-christopherlouet
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping
Source: https://github.com/christopherlouet/claude-base/tree/main/.claude/skills/web-scraping
Command: npx skills add https://github.com/christopherlouet/claude-base --skill web-scraping-christopherlouet

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the extraction of structured content from websites, enabling users to gather data efficiently without manual copying or complex coding.

Core Features & Use Cases

  • Web content extraction: Fetch and convert web pages into clean markdown or JSON formats, handling static and JavaScript-heavy sites.
  • Site crawling: Discover and gather multiple pages from a site while respecting robots.txt and avoiding overloading servers.
  • Use Case: Collect product data from e-commerce sites or compile articles for research by crawling and extracting relevant content seamlessly.

Quick Start

Use the web-scraping skill to extract all articles from a specific news site and save them for analysis.

Frequently Asked Questions about web-scraping

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured data from a website with dynamic JavaScript content?▼

You can extract structured data from dynamic JavaScript pages using this Skill, which leverages firecrawl or Playwright fallbacks to fetch and convert content into clean markdown or JSON formats.

Can I crawl multiple pages from a site without overloading the server?▼

Yes, you can crawl multiple pages safely. This Skill discovers and gathers site pages while strictly respecting robots.txt policies and avoiding server overload during the crawling process.

Do I need firecrawl to perform web content extraction, or are there fallback options?▼

You do not strictly need firecrawl. While firecrawl is the primary engine, this Skill supports fallback methods like curl or Playwright to ensure web content extraction still functions.

What is the best way to collect product data from e-commerce sites for competitive analysis?▼

The best way to collect product data for competitive analysis is using this Skill. It automates site crawling and web content extraction, converting e-commerce pages into structured JSON or markdown.

Does web scraping work for compiling research articles from news sites?▼

Yes, web scraping works for compiling research articles. This Skill automates extracting articles from news sites and saves them in clean markdown or JSON formats for analysis.

What are the limitations of crawling sites for structured data collection?▼

The main limitation of crawling sites for structured data collection is that this Skill strictly respects site policies and robots.txt. It avoids overloading servers, which may limit extraction speed.

Related Skills

Explore More Content & Communication Skills→
nathanvalenathanvale
2

firecrawl

Fetch web content and extract structured data via the Firecrawl API.

Community
Advanced
opencmitopencmit
348

deep-research

Conduct web research and generate structured reports with citations and visualizations.

Official
Advanced
bendruckerbendrucker
16

json

Process JSON data with jq filters for parsing, filtering, and transforming.

Community
Basic
korchasakorchasa

browser-automation

Automate web interactions and extract data using Playwright, webfetch, or curl.

Community
Intermediate