segment-anything-model

Load a checkpoint with sam_model_registry, create a SamPredictor, call set_image once, then predict with point_coords and point_labels or a bounding box. SAM returns multiple masks with scores; select the highest-scoring one.

ViT-B is the smallest and fastest at 375MB, ViT-L is medium at 1.2GB, and ViT-H is the largest and most accurate at 2.4GB. Choose ViT-B for limited GPU memory or speed-critical applications, and ViT-H for best mask quality.

SAM alone does not accept text prompts; it only takes points, boxes, or masks. Combine GroundingDINO, which converts text descriptions into bounding boxes, with SAM to generate masks from those boxes for text-driven segmentation.

SAM can run on CPU but inference is slow, especially for the image encoder. For browser and edge deployment, export the mask decoder to ONNX with the export script and compute image embeddings separately.

The ViT-H model requires significant GPU memory, especially with large images. Switch to the ViT-B checkpoint, resize images to a maximum dimension, use half precision, and call torch.cuda.empty_cache() between batches.

SAM produces class-agnostic masks, so use YOLO or Detectron2 for real-time detection with class labels, Mask2Former for semantic or panoptic segmentation with categories, and SAM 2 for video segmentation with object tracking.