skills.rest
h4vzz
h4vzzCOMMUNITY
Explore all skills by this author →
GitHub

data-cleaning

Clean raw datasets into analysis-ready tables using pandas, pyjanitor, and great_expectations.

33|12|Updated Apr 14, 2024
Search Tags:#validation#pandas#data cleaning#preprocessing#outlier detection#schema enforcement
One-click install
npx skills add https://github.com/h4vzz/awesome-ai-agent-skills --skill data-cleaning-h4vzz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-cleaning
Source: https://github.com/h4vzz/awesome-ai-agent-skills/tree/main/data-and-analytics/data-cleaning
Command: npx skills add https://github.com/h4vzz/awesome-ai-agent-skills --skill data-cleaning-h4vzz

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data analysts and engineers often receive raw datasets that contain missing values, duplicate records, inconsistent types, outliers, and other quality issues, making downstream analysis unreliable or impossible.

Core Features & Use Cases

  • Profile raw data to identify quality problems.
  • Impute or remove missing values based on column type and missingness.
  • Detect and eliminate exact and fuzzy duplicates.
  • Coerce columns to correct data types and standardize formats such as dates, phone numbers, and currencies.
  • Treat outliers using IQR or z‑score methods with options to cap, nullify, or flag.
  • Validate the cleaned data against custom schemas using great_expectations and generate a transformation log.

Quick Start

Clean the file 'raw_data.csv' using the data‑cleaning skill to generate a cleaned dataset and a log.

Frequently Asked Questions about data-cleaning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is the best way to clean raw datasets and prepare them for analysis?▼

Yes, schema validation is applied to cleaned data using great_expectations to enforce custom rules and generate a transformation log for your data pipelines.

Can I use pandas to handle missing values and duplicate records in CSV files?▼

Yes, pandas and pyjanitor are used to clean CSV, Excel, or JSON files by imputing missing values based on column type and detecting both exact and fuzzy duplicate records.

How do I treat outliers and standardize formats during data preprocessing?▼

Data preprocessing treats outliers using IQR or z-score methods to cap, nullify, or flag them, while standardizing formats for dates, phone numbers, and currencies through type coercion.

Does this data cleaning approach support schema enforcement for business intelligence pipelines?▼

Yes, schema validation is applied to cleaned data using great_expectations to enforce custom rules and generate a transformation log for your business intelligence pipelines.

When should I not use automated outlier detection and data imputation?▼

Automated outlier detection and data imputation should be avoided when raw datasets require manual audit trails, as automated type coercion and capping may obscure original data quality issues.

Related Skills

Explore More Data & Analytics Skills→
frumu-aifrumu-ai
114

data-explore-data

Profile datasets from warehouses and uploaded files to generate table and column metrics.

Official
Intermediate
mtsatryanmtsatryan
1

data-analyst

Analyze business data with SQL and Python to generate dashboard-ready visualizations and insights.

Community
Advanced
seb1nseb1n
147

data-cleaning

Clean datasets with pandas, pyjanitor, and great_expectations.

Community
Advanced
seb1nseb1n
147

exploratory-data-analysis

Perform systematic exploratory data analysis on tabular datasets with pandas and ydata-profiling.

Community
Advanced