clip-dissect

Describe individual neuron functions in vision networks using CLIP-based concept similarity.

75|7|Updated May 2, 2026
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill clip-dissect-zjunlp
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: clip-dissect
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/mechanism-skills/multi-modal/clip-dissect
Command: npx skills add https://github.com/zjunlp/Mechanist --skill clip-dissect-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, torchvision, clip, and includes scripts (resource) components.

What problem does it solve? Understanding what individual neurons inside deep vision networks respond to normally requires manual inspection of thousands of activation maps. This Skill automates neuron interpretation by matching neuron activation patterns against natural-language concept sets using CLIP embeddings, producing human-readable descriptions for every unit in a layer. ## Core Features & Use Cases - Automatic Neuron Labeling: Computes cosine similarity between activation-weighted CLIP image embeddings and concept text embeddings to assign each neuron its best-matching concept. - Multi-Layer and Model-Agnostic Dissection: Works with ResNet-50, ResNet-18 (Places-365), Vision Transformers, and custom PyTorch models across multiple layers in one run. - Flexible Probing Setup: Supports Broden and ImageNet validation probing datasets plus bundled 3k, 10k, and 20k English concept sets, with custom datasets and concept files supported. - Use Case: A researcher dissecting ResNet-50 layer4 runs the pipeline with the Broden dataset and 20k concept set, then receives a CSV mapping every neuron to a description like "dog" or "striped" with its similarity score. ## Quick Start Ask the agent to dissect the layers of a pretrained ResNet-50 using the Broden probing dataset and the 20k concept set, saving per-neuron descriptions to a CSV file.

Frequently Asked Questions about clip-dissect

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automatically describe neurons in a neural network?

Use CLIP-Dissect to compute neuron activations over a probing dataset, weight CLIP image embeddings by those activations, and match each neuron against concept text embeddings via cosine similarity. The best-matching concept becomes the neuron's description, saved to a CSV file.

How to run CLIP-Dissect on a custom PyTorch model?

Implement a loader for your model in data_utils.py under get_target_model, returning the model and its preprocessing transform. Then run describe_neurons.py with --target_model set to your model name; any PyTorch architecture including ViT variants is supported.

What probing datasets does CLIP-Dissect support?

CLIP-Dissect supports the Broden dataset, downloaded via dlbroden.sh, and the ImageNet validation set when you configure its path in data_utils.py. Custom torchvision-compatible datasets can be added through the get_data function.

Can CLIP-Dissect run on CPU instead of GPU?

Yes, CLIP-Dissect supports CPU inference by passing --device cpu to describe_neurons.py. A CUDA-compatible GPU is recommended for speed, and you can select a specific GPU with --device cuda:1.

How does CLIP-Dissect compare to NetDissect or MILAN?

CLIP-Dissect labels neurons using CLIP vision-language similarity rather than segmentation masks or captioning models, and requires no per-concept training. The repository includes pre-computed NetDissect and MILAN results plus notebooks for direct quantitative and qualitative comparison.