segment-anything-model

Load a checkpoint with sam_model_registry, create a SamPredictor, call set_image on your RGB image, then call predict with point coordinates and labels. The model returns masks with quality scores, and you select the highest-scoring one.

ViT-H is the largest at 2.4GB with the best accuracy but slowest inference. ViT-B is 375MB and fastest with good accuracy, while ViT-L sits in between at 1.2GB. Choose based on your GPU memory and speed requirements.

SAM alone only accepts point, box, or mask prompts, not text. Combine it with GroundingDINO, which converts text prompts into bounding boxes that SAM then turns into masks.

The original SAM processes single images only. For video, use SAM 2, which adds a streaming memory architecture to track and propagate masks across frames from an initial prompt.

The ViT-H model requires significant VRAM, especially with large images. Switch to the ViT-B checkpoint, resize images to a maximum dimension, call torch.cuda.empty_cache() between images, or use half precision.

Use YOLO or Detectron2 for real-time object detection with class labels, and Mask2Former for semantic or panoptic segmentation with categories. SAM produces class-agnostic masks and is best for prompt-based or automatic segmentation without predefined classes.