clip

Load a model with clip.load("ViT-B/32"), preprocess the image, tokenize candidate text labels, then compute softmax over the model's logits. The highest-probability label is the predicted class, with no training data required.

Encode all images with model.encode_image, normalize the embeddings, and store them. Encode a text query the same way, then rank images by cosine similarity. Embeddings can be stored in vector databases like Chroma or FAISS for fast retrieval.

CLIP is best for zero-shot classification and image-text retrieval. BLIP-2 produces better image captions, while LLaVA supports vision-language chat. For segmentation tasks, use Segment Anything instead.

ViT-B/32 is the recommended default, balancing speed and quality with 151M parameters. RN50 is fastest, while ViT-L/14 offers the best quality at 428M parameters but runs slowest.

CLIP runs on CPU but a GPU is 10-50x faster. Image encoding takes roughly 200ms on CPU versus 20ms on a V100 GPU, so GPU is recommended for batch processing or large image indexes.

CLIP performs poorly on fine-grained distinctions, counting, and spatial reasoning, and it produces whole-image predictions without bounding boxes. It also requires descriptive text labels and may reflect biases from its web training data.