data-cleaning-pipeline-generator

Data cleaning pipelines automate missing value handling and deduplication across pandas or polars DataFrames. This Skill generates a complete pipeline with built-in methods to remove duplicates and fill or drop missing values, delivering validated, deduplicated data ready for analysis.

Yes, the pipeline includes outlier detection and removal functionality for numerical columns. It applies statistical methods to identify and filter anomalies, returning a cleaned dataset with outliers removed and data quality validated.

Data cleaning pipelines fix inconsistent data types, normalize text formatting, and encode categorical variables in one configurable workflow. This ensures type consistency and text uniformity across your pandas or polars dataset without manual intervention.

The generated pipeline includes built-in validation steps and dataset analysis to assess data quality post-cleaning. It provides reports on completeness, type correctness, and statistical properties, confirming your data meets quality standards.

Yes, the pipeline generator supports pandas, polars, and PySpark. Choose your framework based on dataset size and infrastructure; polars and PySpark handle larger-scale data cleaning operations efficiently.

The generated pipeline is fully configurable with individual methods for removing duplicates, handling missing values, fixing types, removing outliers, normalizing text, and encoding categories. Combine or skip steps based on your dataset's unique cleaning needs.