segment-anything-model

Load a checkpoint with sam_model_registry, create a SamPredictor, call set_image once, then predict with point coordinates and labels or a bounding box. The predict method returns masks, quality scores, and logits; select the mask with the highest score.

Use SamAutomaticMaskGenerator with your loaded SAM model and call generate on the image. Tune points_per_side, pred_iou_thresh, and stability_score_thresh to control mask density and quality, then filter results by predicted_iou or area.

SAM 2 is the right choice for video because it adds a streaming memory architecture that tracks objects across frames. SAM only processes individual images and has no temporal tracking, so use it for still-image segmentation tasks.

SAM alone does not accept text prompts. Combine it with GroundingDINO in a Grounded SAM pipeline: GroundingDINO converts text into bounding boxes, which SAM then uses as box prompts to produce masks.

The ViT-H checkpoint requires significant VRAM, especially with large images. Switch to the smaller ViT-B model, resize images to a maximum dimension, call torch.cuda.empty_cache between images, or convert the model to half precision.

Avoid SAM when you need class labels, since it produces class-agnostic masks. Use YOLO or Detectron2 for real-time detection with categories, Mask2Former for semantic or panoptic segmentation, and SAM 2 for video tasks.