segment-anything-model

Load a SAM checkpoint with sam_model_registry, create a SamPredictor, call set_image on your RGB image, then predict with point coordinates or a bounding box. The predictor returns masks with quality scores; select the highest-scoring mask.

ViT-B is the smallest at 375MB and fastest, ViT-H is the largest at 2.4GB with the best accuracy, and ViT-L sits in between at 1.2GB. Use ViT-B for limited GPU memory or speed-critical applications.

SAM alone only accepts point, box, and mask prompts, not text. Combine it with GroundingDINO, which converts text descriptions into bounding boxes that SAM then turns into masks.

Yes, SAM works zero-shot on domain-specific images, but grayscale inputs must be converted to three-channel RGB first. For better medical results, consider MedSAM, a SAM variant fine-tuned on medical imaging data.

The ViT-H model requires significant GPU memory, especially with large images. Switch to the ViT-B checkpoint, resize images to a maximum dimension, call torch.cuda.empty_cache() between images, or use half precision.

Use SAM when you need class-agnostic masks without training, such as annotation tools or novel domains. Choose YOLO or Detectron2 for real-time detection with class labels, and Mask2Former for semantic or panoptic segmentation with categories.