segment-anything-model

Load a checkpoint with sam_model_registry, create a SamPredictor, call set_image once, then predict with point_coords and point_labels. Set multimask_output=True to get three mask candidates and select the one with the highest score.

ViT-B is the smallest at 375MB and fastest with good accuracy, ViT-L is 1.2GB with medium speed, and ViT-H is 2.4GB, slowest but most accurate. Choose based on your GPU memory and latency requirements.

SAM provides zero-shot class-agnostic segmentation from geometric prompts without training. Use YOLO or Detectron2 for real-time detection with class labels, Mask2Former for semantic segmentation with categories, and GroundingDINO plus SAM for text-prompted segmentation.

SAM runs on CPU but inference is significantly slower, especially for the image encoder. For CPU or limited VRAM, use the ViT-B model, reduce image resolution, or export to ONNX for optimized runtime performance.

Incorrect masks usually result from wrong coordinate order (points must be x,y not row,col), BGR instead of RGB images, or ambiguous prompts. Add background points with label 0, combine box and point prompts, or use iterative refinement with previous mask logits.

The original SAM processes images only and has no tracking across frames. For video segmentation, use SAM 2, which adds a streaming memory architecture to propagate masks through video frames from initial prompts.