skills.rest
zsutxz
zsutxzCOMMUNITY
Explore all skills by this author →
GitHub

爬虫数据处理

Clean JSON/CSV crawled data and generate multi-sheet Excel reports with statistics and charts.

Updated Sep 26, 2025
Search Tags:#json#web scraping#data processing#statistics#csv#deduplication#data cleaning#excel reports
One-click install
npx skills add https://github.com/zsutxz/AICodingWithBMad --skill -zsutxz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: 爬虫数据处理
Source: https://github.com/zsutxz/AICodingWithBMad/tree/main/.claude/skills/data-processor
Command: npx skills add https://github.com/zsutxz/AICodingWithBMad --skill -zsutxz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, openpyxl, and includes scripts (resource) components.

What problem does it solve?

Raw web scraped data is often messy, containing duplicates, missing values, and inconsistent formats. This Skill automates the cleaning, analysis, and reporting of such data, transforming it into structured, actionable insights.

Core Features & Use Cases

  • Automated Data Cleaning: Efficiently deduplicate records, fill missing values, and standardize data formats (e.g., converting prices to float) for improved data quality.
  • Statistical Analysis: Generate descriptive statistics to understand data distribution, identify anomalies, and calculate deduplication rates, providing a quick overview of your dataset.
  • Excel Report Generation: Create comprehensive Excel reports including cleaned data, statistical summaries, and visual charts for easy interpretation and sharing.
  • Use Case: Process a large JSON file of product listings scraped from e-commerce sites, remove duplicates, standardize prices, and generate an Excel report with average prices and product counts per category for market analysis.

Quick Start

Clean the attached 'scraped_products.json' file, deduplicating by 'product_id', and generate an Excel report named 'product_report.xlsx'.

Frequently Asked Questions about 爬虫数据处理

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean and deduplicate web scraped JSON data?▼

Web scraping produces messy data with duplicates and inconsistencies. This Skill reads JSON or CSV crawled data, removes duplicate records, fills missing values, standardizes formats (e.g., converting prices to floats), and outputs cleaned data. It handles deduplication by specified fields and applies type conversions automatically.

Can I generate Excel reports from scraped data with statistics and charts?▼

Yes. After cleaning your crawled data, this Skill generates multi-sheet Excel workbooks containing the cleaned dataset, descriptive statistics (counts, averages, distributions), and visual charts. Export formats include data sheets, summary statistics, and chart visualizations in a single .xlsx file.

What statistical analysis does this Skill perform on raw web data?▼

This Skill computes descriptive statistics including deduplication rates, missing-value counts, field distributions, and category-level aggregations (e.g., average prices per category). Statistics are calculated post-cleaning and exported alongside cleaned data in the Excel report for quick dataset overview.

Does this work with both JSON and CSV formats from web scrapers?▼

Yes, this Skill accepts both JSON and CSV as input formats from web scrapers. It standardizes fields across both formats, applies the same cleaning and deduplication logic, and outputs the results consistently in Excel format with all data, statistics, and charts.

How do I handle missing values and inconsistent data types in scraped datasets?▼

This Skill automatically fills missing values (NaNs) using appropriate strategies and converts fields to correct data types—such as parsing price strings to floats or date fields to datetime objects. Type conversions and NaN handling are applied uniformly across all records during the cleaning phase.

What if my scraped data contains many duplicate product records?▼

Specify the deduplication key (e.g., 'product_id') and this Skill removes duplicate records while tracking deduplication rates in the output statistics. The cleaned dataset and deduplication metrics are included in the Excel report for audit and quality assessment purposes.

Related Skills

Explore More Data & Analytics Skills→
jesseotremblayjesseotremblay
1

processing-data

Transform, clean, and extract structured information from CSV and JSON data.

Community
Basic
JonathanMitchell1234JonathanMitchell1234

data-science-expert

Perform data cleaning, feature engineering, and statistical modeling with pandas and scikit-learn.

Community
Advanced
chestnut0698chestnut0698

web-scraping

Extract data from static and dynamic web pages using Python tools.

Community
Advanced
AIDotNetAIDotNet
602

csv-pipeline

Process CSV, TSV, JSON, and JSON Lines files with Python 3 standard library.

Official
Intermediate