segment-anything-model

Load a checkpoint with sam_model_registry, create a SamPredictor, call set_image once, then predict with point coordinates and labels (1 for foreground, 0 for background). The predictor returns masks with IoU scores; select the highest-scoring mask.

ViT-H (2.4 GB) is the slowest but most accurate, ViT-L (1.2 GB) is a middle option, and ViT-B (375 MB) is the fastest with good accuracy. Use ViT-B when GPU memory or inference speed is limited.

SAM only accepts geometric prompts (points, boxes, masks) and cannot interpret text. Combine GroundingDINO, which converts text descriptions into bounding boxes, with SAM to generate masks from those boxes for text-prompted segmentation.

SAM runs on CPU but is significantly slower than on CUDA GPUs. For browser and edge deployment, export the mask decoder to ONNX with the provided export script and run it with ONNX Runtime, computing image embeddings separately.

The ViT-H checkpoint and large images consume substantial GPU memory. Switch to the ViT-B model, downscale images, call torch.cuda.empty_cache() between batches, or load the model in half precision to reduce memory usage.

SAM does not assign class labels, so use YOLO or Detectron2 for real-time detection with categories, and Mask2Former for semantic or panoptic segmentation. For video segmentation with object tracking across frames, use SAM 2 instead.