data-processing

Build ETL pipelines for large CSV datasets by loading data with optimized dtypes, performing vectorized transformations, and saving cleaned results to Parquet. This approach uses pandas, polars, and Dask to extract, transform, and load data efficiently.

Reduce memory usage in large tabular datasets by downcasting numeric dtypes, reading data in chunks, and utilizing Parquet storage formats. These strategies optimize memory management during data processing with pandas and polars.

Polars provides fast, vectorized operations for data cleaning and feature engineering, while pandas offers broad functionality for standardizing formats and deriving features. Both clean missing values and aggregate datasets, with Polars excelling in performance optimization.

Yes, Dask handles time-series resampling and aggregation on large datasets. It works alongside pandas and NumPy to build end-to-end pipelines that process, transform, and aggregate time-series data using vectorized operations and grouping patterns.

Save cleaned and transformed data in Parquet format to optimize storage and memory efficiency. Parquet reduces file size compared to CSV, supports optimized dtypes for vectorized computations, and enables fast chunked reads for downstream ETL workflows.