skills.rest
ehadziabdic
ehadziabdicCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

model-observability

Implement ML observability with SHAP/LIME explainability, prediction logging, tracing, and fairness metrics.

Updated Sep 23, 2026
Search Tags:#opentelemetry#shap#lime#model explainability#fairness metrics#prediction logging#ml observability
One-click install
npx skills add https://github.com/ehadziabdic/WAgents --skill model-observability-ehadziabdic
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-observability
Source: https://github.com/ehadziabdic/WAgents/tree/main/opencode/skills/model-observability
Command: npx skills add https://github.com/ehadziabdic/WAgents --skill model-observability-ehadziabdic

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires shap, lime, numpy, pandas, scikit-learn, matplotlib, opentelemetry-sdk, opentelemetry-exporter-otlp-proto-grpc, joblib, xgboost, lightgbm, torch, and includes scripts (resource) and references (resource) components.

What problem does it solve? Production ML models often behave as black boxes, making it hard to debug mispredictions, detect bias, satisfy regulatory audits, or trace requests across inference pipelines. This Skill provides the code patterns and scripts to explain predictions, log every inference, and measure fairness across segments. ## Core Features & Use Cases - Model Explainability: Generate SHAP (Tree, Kernel, Deep, Gradient explainers) and LIME explanations for sklearn, XGBoost, LightGBM, and PyTorch models, with global feature importance and per-instance attributions. - Prediction Logging & Audit Trails: Log structured prediction records to SQLite or JSONL with unique request IDs, then query, compute statistics, detect anomalies with Isolation Forest, and export to CSV/Parquet. - Fairness & Slice Analysis: Compute disparate impact, equal opportunity, and equalized odds metrics across demographic groups, plus slice-based performance analysis by cohort. - Use Case: A credit-scoring model's accuracy drops for one customer segment. Use the explainability script to generate SHAP values for failing predictions, run slice analysis to isolate the underperforming cohort, and produce fairness metrics for an ECOA compliance review. ## Quick Start Ask the AI to explain why the model in model.pkl mispredicted row 42 of data.csv using SHAP and save the plots to an explanations folder.

Frequently Asked Questions about model-observability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I explain model predictions with SHAP in Python?▼

Use shap.TreeExplainer for tree models like XGBoost and LightGBM, DeepExplainer for PyTorch, and KernelExplainer as a model-agnostic fallback. The explain_predictions.py script auto-selects the right explainer and outputs global feature importance plus per-instance SHAP values.

SHAP vs LIME for model explainability, which should I use?▼

SHAP provides mathematically consistent Shapley values that aggregate cleanly to global importance, making it preferred for tree models and regulatory audits. LIME fits a local surrogate per instance, so use it for human-readable rule-style explanations or black-box models behind APIs.

How do I log ML predictions for audit trails?▼

Use the prediction_logger.py script to record each prediction with a unique request ID, timestamp, input features, probability, model version, and latency into SQLite or JSONL. It supports querying by date range, computing confidence statistics, and exporting to CSV or Parquet.

Does SHAP work with PyTorch deep learning models?▼

Yes, SHAP supports PyTorch through DeepExplainer and GradientExplainer, which use gradient-based attribution on background data. The script loads full pickled modules with torch.load, so only explain model artifacts you trust since unpickling executes code.

Why do my SHAP feature importances look wrong for classifiers?▼

SHAP 0.45 and later returns a 3-D array with a trailing class axis instead of a list of per-class arrays, so naive averaging produces misaligned importances. Normalize the output by selecting the positive class axis before computing mean absolute values.

How do I measure fairness metrics like disparate impact?▼

Compute per-group accuracy, positive rate, TPR, and FPR across a sensitive attribute, then derive disparate impact as the ratio of positive rates and equalized odds as the max of TPR and FPR gaps. Note that calibration and equalized odds cannot both hold when base rates differ.

Related Skills

Explore More Data & Analytics Skills→
RamanEbrahimiRamanEbrahimi
3

shap

Compute SHAP values and generate SHAP plots for machine learning model predictions.

Community
Advanced
DTMC-marketplaceDTMC-marketplace

evidently-ai

Monitor ML models in production for data drift and performance degradation.

Official
Intermediate
DuqueOMDuqueOM
5

drift-detection

Detect data and concept drift in ML services using PSI and sliced AUC metrics.

Community
Advanced
davila7davila7
30.5k

phoenix-observability

Trace, evaluate, and monitor LLM applications with OpenTelemetry-based observability.

Community
Advanced