skills.rest
pymc-labs
pymc-labsOFFICIAL
Explore all skills by this author →
Website𝕏@pymc_labsGitHub

model-evaluation

Compare Bayesian models with ArviZ 1.1 LOO, ELPD, and Bayes factors.

76|10|Updated Feb 5, 2026
Search Tags:#model comparison#model evaluation#arviz#bayesian modeling#loo-cv#model averaging
One-click install
npx skills add https://github.com/pymc-labs/python-analytics-skills --skill model-evaluation-pymc-labs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-evaluation
Source: https://github.com/pymc-labs/python-analytics-skills/tree/main/skills/model-evaluation
Command: npx skills add https://github.com/pymc-labs/python-analytics-skills --skill model-evaluation-pymc-labs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires arviz, arviz.stats, arviz.plots, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps users compare and evaluate Bayesian models using ArviZ 1.1 metrics and diagnostics, providing insights into model performance and reliability.

Core Features & Use Cases

  • Model Comparison: Compare multiple models on predictive accuracy using LOO, ELPD, and Bayes factors.
  • LOO-CV: Perform leave-one-out cross-validation with PSIS for efficient computation.
  • Pareto k Diagnostics: Assess the reliability of PSIS approximation for each observation.
  • Moment Matching: Refit problematic observations using moment matching for improved PSIS estimates.
  • K-Fold CV: Use exact k-fold cross-validation when PSIS fails.
  • Model Averaging: Combine predictions from multiple models for better predictive performance.
  • Use Case: Suppose you have three different models for predicting house prices. Use this Skill to compare their predictive accuracy and select the best model.

Quick Start

Run the model-evaluation skill to compare the predictive performance of two models using LOO-CV and ELPD metrics.

Frequently Asked Questions about model-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compare Bayesian models using leave-one-out cross-validation?▼

Compare Bayesian models by computing expected log predictive density (ELPD) and LOO metrics with Pareto smoothed importance sampling (PSIS) to evaluate and rank predictive performance accurately.

What does a high Pareto k diagnostic value mean in LOO-CV?▼

A high Pareto k diagnostic value indicates the PSIS approximation is unreliable for that observation. Use moment matching to refit problematic observations or switch to exact k-fold cross-validation for reliable Bayesian model evaluation.

Do I need ArviZ 1.1 to compute Bayes factors for model selection?▼

Yes, ArviZ 1.1 is required to compute Bayes factors and execute model diagnostics. The skill relies on ArviZ 1.1 stats to calculate ELPD, perform LOO-CV, and compare Bayesian predictive performance.

Can I combine predictions from multiple Bayesian models?▼

Combine predictions from multiple Bayesian models through model averaging. This merges predictive performance from different models to yield more robust and accurate overall predictions.

When should I use k-fold cross-validation instead of PSIS for model evaluation?▼

Use exact k-fold cross-validation instead of PSIS when the Pareto k diagnostics indicate that PSIS approximation fails. K-fold CV provides a robust alternative for computing model predictive performance in Bayesian statistics.

Related Skills

Explore More Data & Analytics Skills→
Hung-3008Hung-3008
1

pymc

Build and analyze Bayesian models with PyMC using MCMC sampling and diagnostics.

Community
Advanced
nferrer-devnferrer-dev

pymc

Perform Bayesian modeling with PyMC and ArviZ for MCMC sampling and diagnostics.

Community
Advanced
xjtulycxjtulyc
33

bayesian-stats

Build Bayesian models with PyMC 5.x and ArviZ using NUTS sampling.

Community
Advanced
ovachieverovachiever
52

pymc-bayesian-modeling

Build, fit, and evaluate Bayesian models with PyMC using MCMC sampling.

Community
Advanced