skills.rest
nguynbon03
nguynbon03COMMUNITY
Explore all skills by this author →
GitHub

web-scraper

Crawl websites and extract structured data to JSON or CSV.

Updated Aug 23, 2026
Search Tags:#playwright#data-extraction#web-scraping#structured-data#web-crawling#scrapy#beautifulsoup
One-click install
npx skills add https://github.com/nguynbon03/API-XuongMedia-Backup --skill web-scraper-nguynbon03
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/nguynbon03/API-XuongMedia-Backup/tree/main/.claude/skills/web-scraper
Command: npx skills add https://github.com/nguynbon03/API-XuongMedia-Backup --skill web-scraper-nguynbon03

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Thu thập dữ liệu có cấu trúc từ các website khác nhau mà có thể là trang tĩnh hoặc trang động (JS-rendered). Dễ dàng tích hợp vào quy trình phân tích dữ liệu để tự động thu thập và xuất dữ liệu ở định dạng JSON hoặc CSV.

Core Features & Use Cases

  • Hỗ trợ nhiều thư viện: BeautifulSoup4 cho HTML tĩnh, Playwright cho trang động, Scrapy cho crawl quy mô lớn, HAR → requests cho API ẩn.
  • Mẫu triển khai có sẵn cho bs4-scraper.py, playwright-scraper.py và reverse-engineer-api.py để tham khảo.
  • Đầu ra dữ liệu ở JSON hoặc CSV và cơ chế tuân thủ robots.txt, delay và đổi User-Agent.

Quick Start

Chạy web-scraper để thu thập dữ liệu có cấu trúc từ một trang web mục tiêu và lưu kết quả ở JSON hoặc CSV.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape data from JavaScript-rendered pages?▼

To scrape JavaScript-rendered pages, you can use Playwright to execute scripts and extract dynamically loaded content, outputting the results into structured JSON or CSV files.

What is the best way to extract structured data from static HTML?▼

The best way to extract structured data from static HTML is using BeautifulSoup4, which parses the page source and structures the extracted information into JSON or CSV outputs.

How do I reverse engineer hidden APIs for web scraping?▼

You can reverse engineer hidden APIs by analyzing HAR files to discover underlying network requests, then replicating those requests directly to extract structured JSON data.

Can I use Scrapy for large-scale web crawling?▼

Yes, Scrapy is supported specifically for large-scale web crawling, enabling you to extract structured data across many pages efficiently while managing delays and User-Agent rotation.

Does web scraping with Python support CSV and JSON export?▼

Yes, web scraping with Python supports both CSV and JSON export, allowing you to easily integrate the extracted structured data into your downstream analysis workflows.

Related Skills

Explore More Software Engineering Skills→
raphaelchristiraphaelchristi
43

harness:setup

Configure Harness Evolver, LangSmith, and baseline evaluations for Python projects.

Community
Advanced
amanning3390amanning3390
13

data-analyst

Analyze CSV, JSON, Excel, and SQLite datasets to generate structured findings reports.

Community
Intermediate
HouseGarofaloHouseGarofalo
3

web-automation

Automate browser interactions and web scraping with Playwright, Puppeteer, Selenium, and Scrapy.

Official
Intermediate
joyce820309joyce820309

scrape

Automate HTML data extraction using BeautifulSoup and requests.

Community
Intermediate