segment-anything-model

Load a SAM checkpoint with sam_model_registry, create a SamPredictor, call set_image on your RGB image, then call predict with point coordinates and labels. Set multimask_output to True to get three candidate masks and pick the one with the highest score.

ViT-B is the smallest at 375MB and fastest, ViT-L is medium at 1.2GB, and ViT-H is the largest at 2.4GB with the best accuracy. Use ViT-B when GPU memory or speed is limited, and ViT-H when mask quality matters most.

SAM alone does not accept text prompts; it only takes points, boxes, or masks. Combine GroundingDINO, which converts text descriptions into bounding boxes, with SAM to turn those boxes into segmentation masks.

The ViT-H model requires significant GPU memory, especially with large images. Switch to the ViT-B checkpoint, resize images to a maximum dimension, call torch.cuda.empty_cache() between images, or run inference in half precision.

The original SAM only processes individual images and has no tracking capability. For video segmentation, use SAM 2, which adds a streaming memory architecture to propagate masks across frames from an initial prompt.

The default grid of points may be too sparse for small objects. Increase points_per_side, lower pred_iou_thresh and stability_score_thresh, enable crop_n_layers for multi-scale processing, and reduce min_mask_region_area.