feature-engineering

Use one-hot encoding for nominal categories via pd.get_dummies or OneHotEncoder, label encoding for ordinal data, and target or frequency encoding for high-cardinality columns. Choose based on whether categories have inherent order and how many unique values exist.

Split data into train and test sets before any feature engineering, then fit scalers and encoders only on training data. For time series, use TimeSeriesSplit and never compute features with future information such as shift(-1).

StandardScaler centers data to mean 0 and std 1, suiting normally distributed features. MinMaxScaler scales to [0, 1] for bounded ranges, while RobustScaler uses median and IQR, making it the choice when outliers are present.

Train a RandomForestClassifier and rank features by feature_importances_, or use SelectKBest with f_classif or mutual_info_classif for univariate selection. Also drop features with correlation above 0.95 to reduce multicollinearity.

Pipelines combine imputation, scaling, and encoding into one reproducible object via ColumnTransformer, ensuring identical transformations in training and production. This prevents inconsistencies and makes the workflow easy to deploy and validate.

Compare training and production feature distributions using the Kolmogorov-Smirnov test (ks_2samp from scipy). A p-value below 0.05 indicates the distribution has shifted, signaling potential model degradation that needs investigation.