segment-anything-model

Load a SAM checkpoint with sam_model_registry, create a SamPredictor, call set_image on your RGB image, then predict with point coordinates and labels or a bounding box. The predictor returns masks with quality scores; select the highest-scoring mask.

ViT-B is the smallest (375MB) and fastest with good accuracy, ViT-L is medium (1.2GB), and ViT-H is the largest (2.4GB) and slowest but most accurate. Use ViT-B when GPU memory or speed is limited.

SAM alone does not accept text prompts; it only takes points, boxes, or masks. Combine GroundingDINO, which converts text descriptions into bounding boxes, with SAM to generate masks from those boxes for text-prompted segmentation.

SAM can run on CPU by loading the model without calling .to("cuda"), but inference is significantly slower, especially for ViT-H. For limited GPU memory, use the ViT-B checkpoint, half precision, or export to ONNX for lighter deployment.

CUDA out of memory occurs because ViT-H requires substantial VRAM, especially with large images or batch processing. Switch to the ViT-B model, resize images, call torch.cuda.empty_cache() between images, or process on CPU.

Avoid SAM when you need class labels, since it produces class-agnostic masks. Use YOLO or Detectron2 for real-time object detection with categories, Mask2Former for semantic segmentation, and SAM 2 for video segmentation tasks.