model-validation-and-uncertainty

Compute performance metrics and diagnostic plots for ecological predictive models.

2|Updated Mar 2, 2026

Search Tags:#python #calibration #uncertainty #model-validation #occupancy #sdm #roc-curve

Ready to run

npx skills add https://github.com/baratadiego/ecological-agent-skills --skill model-validation-and-uncertainty

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: model-validation-and-uncertainty
Source: https://github.com/baratadiego/ecological-agent-skills/tree/main/skills/model-validation-and-uncertainty
Command: npx skills add https://github.com/baratadiego/ecological-agent-skills --skill model-validation-and-uncertainty

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, pandas, matplotlib, scikit-learn, and includes scripts (resource) components.

What problem does it solve?

Evaluates and quantifies the performance and reliability of predictive ecological models by computing standard metrics, evaluating calibration, and mapping uncertainty across predictions.

Core Features & Use Cases

Comprehensive performance metrics: AUC-ROC, TSS, calibration curves, and threshold analysis for SDMs, occupancy models, and ensemble projections.
External validation & transferability: supports application to independent datasets or different time periods to assess generalization.
Uncertainty quantification & visualization: provides bootstrap/ensemble uncertainty maps and a narrative validation report.

Quick Start

Provide a fitted model object and an independent validation dataset to run the evaluation and generate the diagnostic outputs.

Frequently Asked Questions about model-validation-and-uncertainty

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I validate species distribution models using an independent dataset?▼

To validate species distribution models, supply a fitted model object and an independent dataset to generate AUC-ROC, TSS, calibration curves, and a narrative validation report.

What metrics are used for occupancy model validation and uncertainty mapping?▼

Occupancy model validation uses AUC-ROC, TSS, calibration curves, and threshold analysis to quantify performance, while bootstrap and ensemble approaches generate uncertainty maps for the predictions.

Can I assess ecological model transferability across different time periods?▼

Yes, ecological model transferability is assessed by applying the fitted model to independent datasets or different time periods to evaluate generalization through cross-validation and external validation metrics.

Does this model validation approach work with scikit-learn and pandas?▼

Yes, the model validation workflow leverages scikit-learn and pandas, alongside numpy and matplotlib, to compute metrics and generate diagnostic ROC and calibration plots.

What's the best way to generate ROC curves and calibration plots for ensemble forecasts?▼

To generate ROC curves and calibration plots for ensemble forecasts, input the ensemble model and validation data to automatically produce diagnostic plots alongside uncertainty maps.