pandas-pro

Data cleaning in pandas uses dropna() to remove missing values, drop_duplicates() to eliminate duplicate rows, and fillna() to impute values. Vectorized operations handle these transformations efficiently across large datasets without explicit loops.

Use groupby() to partition data by column values, then apply aggregation functions like sum(), mean(), or count(). Pandas groupby enables complex multi-level aggregations and pivot-like results while maintaining memory efficiency through vectorized operations.

Merge DataFrames using merge() or join() with specified keys and join types (inner, outer, left, right). Pandas handles memory-aware joins on indexed columns, and choosing appropriate dtypes like categorical reduces memory overhead for large-scale merging operations.

Pandas supports time series analysis with datetime indexing, resampling, and rolling operations. For large datasets, use dtype optimization (categorical, int64 vs float64), chunking strategies, and vectorized operations to maintain performance while preserving memory.

Choose categorical dtype for low-cardinality strings, use smaller integer types (int32 vs int64), and avoid object dtype when possible. Explicit dtype selection reduces memory footprint significantly, especially for production-grade DataFrames with millions of rows.

Preserve or reset index explicitly using reset_index() or set_index() to maintain data alignment during transformations. Careful index management ensures merge operations align rows correctly and prevents silent data misalignment in aggregations.