model-interpretability

Export Integrated Gradients, Grad-CAM, and attention weights from trained PyTorch models with QA validation.

89|5|Updated Mar 25, 2026
One-click install
npx skills add https://github.com/CUHK-AIM-Group/NeuroDiscovery --skill model-interpretability-cuhk-aim-group
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: model-interpretability
Source: https://github.com/CUHK-AIM-Group/NeuroDiscovery/tree/main/skills/model-interpretability
Command: npx skills add https://github.com/CUHK-AIM-Group/NeuroDiscovery --skill model-interpretability-cuhk-aim-group

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, numpy, pandas, nibabel, matplotlib, and includes scripts (resource) and references (resource) components.

What problem does it solve? After training a neuroimaging or tabular model, researchers need reproducible explanation artifacts—attribution maps, Grad-CAM localizations, or attention weights—without inventing explainers or weakening checkpoint security. This Skill produces target-specific explanations with numerical QA and a provenance manifest. ## Core Features & Use Cases - Integrated Gradients: Computes signed per-feature or per-voxel attributions against an explicit baseline, with trapezoidal integration and completeness checks. - Grad-CAM and attention export: Generates ReLU Grad-CAM maps from a selected 2D/3D layer, or exports real normalized attention weights (including BNT's get_attention_weights()). - Spatial NIfTI output: Writes per-subject attribution maps aligned to reference model-grid NIfTIs with validated affines, plus diagnostics.csv, qc.json, a preview figure, and run_manifest.json. - Use Case: Given a trained 3D CNN checkpoint classifying MRI scans, run the gradcam method with a chosen encoder layer to produce per-patient spatial heatmaps and a manifest binding checkpoint, preprocessing, and output hashes. ## Quick Start Run the explain script with a reviewed adapter file, checkpoint, preprocessed NPZ inputs, and metadata JSON, choosing method ig, gradcam, or attention plus a target class and output directory.

Frequently Asked Questions about model-interpretability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compute Integrated Gradients for a PyTorch model?

Integrated Gradients integrates input gradients along a path from an explicit baseline to the actual input using a trapezoidal rule. Provide a baseline NPZ matching input shapes, select a target logit, and the script checks signed attribution sums against the score difference for completeness.

How to generate Grad-CAM maps for 3D CNN models?

Grad-CAM requires selecting a layer with [1,channels,spatial...] activations and a spatial input of shape [N,C,X,Y,Z]. The script computes spatially averaged target gradients, applies a weighted activation sum with ReLU, and upsamples the map to the input grid.

Can I export attention weights from any transformer model?

Attention export only works when the selected module actually returns (output, weights) tuples or exposes get_attention_weights(), such as BNT. Weights must be finite, nonnegative, and row-normalized; the script never fabricates attention from gradients or synthetic values.

Why does the IG completeness check fail?

Completeness fails when the summed attributions deviate from the input-minus-baseline score difference beyond tolerance, usually due to too few integration steps or a poorly chosen baseline. Retry with more --steps in a new directory rather than loosening tolerances.

Is loading arbitrary pickled model checkpoints supported?

No, checkpoints load with weights_only=True and strict state-dict matching to prevent untrusted pickle execution. The adapter must be a reviewed local Python file explicitly opted in with --trust-adapter.

Do explanation maps prove causal or clinical validity?

No, attribution and Grad-CAM maps are numerical explanations of model behavior, not causal effects or clinical evidence. Biomarker claims require separate perturbation, randomization, stability, and external validation experiments.