segment-anything-model

Load a checkpoint with sam_model_registry, create a SamPredictor, call set_image once, then predict with point coordinates and labels (1 for foreground, 0 for background). The predictor returns masks with scores; select the highest-scoring mask.

ViT-B is the smallest (375MB) and fastest with good accuracy, ViT-L is medium (1.2GB), and ViT-H is the largest (2.4GB) with the best accuracy but slowest inference. Choose based on your GPU memory and speed requirements.

SAM alone does not accept text prompts; it only takes points, boxes, or masks. Combine GroundingDINO, which converts text descriptions into bounding boxes, with SAM to generate masks from those boxes for text-prompted segmentation.

SAM can run on CPU by not calling .to("cuda"), but inference is significantly slower, especially for the image encoder. For limited GPU memory, use the ViT-B model, half precision, or export to ONNX for optimized runtime.

Incorrect masks usually result from wrong coordinate order (points must be x,y not row,col), BGR instead of RGB images, or ambiguous prompts. Add background points, use a bounding box, or combine box and point prompts to improve results.

Avoid SAM when you need class labels with real-time detection (use YOLO or Detectron2), semantic or panoptic segmentation with categories (use Mask2Former), or video tracking (use SAM 2, which adds streaming memory across frames).