model-comparison

Run XGBoost, LR+WoE, and DNN on a shared split and aggregate OOT metrics.

580|66|Updated Apr 21, 2025
One-click install
npx skills add https://github.com/aliyun/qwen-dianjin --skill model-comparison
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: model-comparison
Source: https://github.com/aliyun/qwen-dianjin/tree/main/DianJin-SKILLS/financial-engineering-expert/model-comparison
Command: npx skills add https://github.com/aliyun/qwen-dianjin --skill model-comparison

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires json, logging, numpy, pandas, scipy, sklearn, torch, xgboost, optbinning, joblib, and includes scripts (resource) components.

What problem does it solve?

This Skill removes guesswork from “which algorithm is better” by running multiple models under the same data split and producing an objective, multi-metric comparison report for financial ML use.

Core Features & Use Cases

  • Fair multi-model evaluation: runs XGBoost, LR (WoE), and DNN side-by-side on the same train/val/OOT split.
  • Multi-dimensional, non-subjective reporting: computes OOT AUC/KS/BCR/Brier, plus KS gap and PSI drift, and performs DeLong significance tests (AUC).
  • Pareto frontier “candidate set”: identifies algorithms on the Pareto front (no主观加权, no gatekeeping), then leaves final selection to LLM/business reasoning.
  • Supports common financial scoring scenarios: general, scorecard (LR-friendly), fraud (capture-focused), stability-first (drift-focused).

Quick Start

Ask the AI to run the Skill with your dataset by providing the data path, target column, OOT time column rules, and output directory so it generates the multi-algorithm comparison report and artifacts.

Frequently Asked Questions about model-comparison

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compare XGBoost, DNN, and logistic regression models fairly using the same data split?

To compare models fairly, this Skill runs XGBoost, LR+WoE, and DNN side-by-side on a shared train/val/OOT data split, aggregating OOT AUC, KS, BCR, and Brier scores to generate an objective multi-model evaluation report.

What is a Pareto frontier candidate set in multi-model evaluation?

A Pareto frontier candidate set identifies algorithms on the Pareto front without subjective weighting or gatekeeping, leaving the final model selection to LLM or business reasoning based on objective metrics.

How do I measure model stability and drift for financial ML scoring?

You measure model stability by calculating KS gap and PSI drift alongside OOT metrics, and performing DeLong AUC significance tests to evaluate binary scoring model stability across time periods.

Can I run multi-algorithm comparisons for fraud and scorecard scenarios?

Yes, you can run multi-algorithm comparisons for general, scorecard, fraud, and stability-first scenarios by applying consistent data slicing and scenario context to evaluate which model is better.

Do I need tuned hyperparameters to run a binary scoring model comparison?

No, you do not need tuned hyperparameters to run the comparison, but the Skill optionally loads them to refine the XGBoost, DNN, and LR+WoE models during the multi-dimensional evaluation process.