ml-pipeline

MLflow tracking logs parameters, metrics, and model artifacts within a run context using mlflow.start_run(). The skill provides wrapper classes for logging hyperparameters, registering models to the MLflow Model Registry, and transitioning versions between Staging and Production stages.

Kubeflow pipelines are defined with the kfp.dsl component decorator, where each step (load, preprocess, train, evaluate) is a containerized component passing Dataset and Model artifacts. The skill includes a complete template with conditional deployment using dsl.If based on evaluation thresholds.

Kubeflow runs containerized components on Kubernetes with native artifact passing, Airflow uses DAGs with XCom for task communication and scheduling, and Prefect offers Python-native flows with caching and retries. The skill provides working templates for all three orchestrators.

Data drift detection compares current feature distributions against a reference using the Kolmogorov-Smirnov test or Population Stability Index. The skill's FeatureDriftDetector flags drift when p-values fall below significance levels or PSI exceeds 0.2.

Shadow deployment runs a candidate model alongside production, logging predictions without affecting responses. The skill's ShadowDeployment class compares prediction agreement and latency between models, letting you validate performance on real traffic before promotion.

Full pipeline orchestration is unnecessary for quick one-off experiments, simple scripts without hyperparameters, or interactive prototyping. The references explicitly recommend against these tools for ad-hoc analysis without reproducibility or deployment requirements.