Multi-Modal Interpretability

Use CLIP-Dissect: collect each neuron's activations over a probing image set, compute activation-weighted CLIP image embeddings, and rank concept-set text embeddings by cosine similarity. The top concept becomes the neuron's human-readable description, saved to a CSV file.

Concept Relevance Propagation (CRP) is an LRP-based method from the zennit-crp library that produces heatmaps conditioned on a chosen latent concept such as a channel or neuron. It also supports Relevance Maximization reference images and attribution graphs tracing relevance flows through layers.

They solve different sub-problems and are often composed. Use CLIP-Dissect alone for pure neuron labeling, Zennit-CRP alone for explaining single predictions with heatmaps, and both together when building per-component concept vectors, where CRP crops reference images before CLIP embedding.

Yes, CLIP-Dissect is model-agnostic and works with any PyTorch model including ResNets and ViTs by implementing a loader function in data_utils.py. For ViTs, zennit-crp lacks CRP through self-attention, so use upsampled spatial-token activation maps as a fallback heatmap.

Labels inherit the biases and blind spots of the alignment model, so concepts CLIP cannot embed cleanly yield noisy rankings. Probing-image and concept-set choices also change labels, and embedding full reference images instead of CRP-cropped regions adds background noise.

A high similarity score only localizes a concept correlationally; it does not prove the unit is causally responsible for the model using that concept. Causal claims require interventional follow-up such as masking, swapping, or steering the labeled unit and observing prediction changes.