datanalysis-credit-risk

Run the example.py script, which executes an 11-step pipeline covering missing rate filtering, IV-based selection via toad binning, PSI stability checks, Null Importance denoising with LightGBM, and high-correlation removal. Each step accepts interactive threshold parameters with defaults.

Null Importance trains LightGBM models on both real and label-permuted data, then compares feature gain values. Features whose real gain is not sufficiently larger than their permuted gain (difference below the gain threshold, default 50) are flagged as noise and dropped.

The get_dataset function attempts to read parquet, csv, xlsx, and pkl formats in order. Parquet is recommended for best performance. The data must contain a date column, a binary label column (0/1), an organization column, and primary key columns for deduplication.

PSI is calculated per organization across consecutive months using 10-bin quantile binning with NaN as a separate bin. A feature is dropped when the number of unstable organizations (those exceeding the PSI threshold in too many months) reaches the max_orgs limit, default 6.

IV calculation uses toad.transform.Combiner with decision-tree binning (5 bins, empty_separate=True). It can fail when features have too few unique values, insufficient samples, or constant values after cleaning. Check that features remain after earlier steps and that the label column contains both 0 and 1.

The pipeline assumes features are prefixed with 'i_' and requires at least 1000 rows for Null Importance. It is designed for binary classification with monthly organization-level data, so it does not handle regression targets, non-tabular data, or real-time scoring scenarios.