clip

Load a model with clip.load("ViT-B/32"), preprocess the image, tokenize your text labels, then compute softmax over the logits from model(image, text). Descriptive labels like "a photo of a dog" outperform single words.

Encode all images with model.encode_image, normalize the embeddings, then encode a text query and compute cosine similarity against the image index. Store embeddings in Chroma or FAISS for scalable retrieval.

CLIP excels at zero-shot classification and image-text matching. BLIP-2 is better for image captioning, LLaVA for vision-language chat, and Segment Anything for segmentation tasks.

ViT-B/32 is the recommended default, balancing speed and quality at 151M parameters. RN50 is fastest, while ViT-L/14 at 428M parameters gives the best quality but runs slowest.

CLIP runs on CPU with image encoding around 200ms, but a GPU is recommended since it is 10-50x faster, roughly 20ms per image on a V100. Use device = "cuda" if torch.cuda.is_available().

CLIP handles whole images only with no bounding boxes, performs weakly on fine-grained tasks, counting, and spatial reasoning, and may carry biases from its web training data. Vague text labels also degrade performance.