segment-anything-model

Load a checkpoint with sam_model_registry, create a SamPredictor, call set_image once, then predict with point coordinates and labels (1 for foreground, 0 for background). With multimask_output enabled, SAM returns three mask candidates with scores so you can pick the best.

ViT-H is the largest at 2.4 GB with the best accuracy but slowest speed, ViT-L is a medium 1.2 GB option, and ViT-B is the smallest at 375 MB and fastest. Use ViT-B when GPU memory or latency is constrained.

SAM alone only accepts geometric prompts like points and boxes, not text. Combine GroundingDINO to convert a text caption into bounding boxes, then feed those boxes into SAM to produce masks for the described objects.

SAM processes individual images only and has no tracking across frames. For video segmentation, use SAM 2, which adds a streaming memory architecture to propagate masks through video after prompting on an initial frame.

The ViT-H checkpoint is 2.4 GB and large images increase memory pressure. Switch to the ViT-B model, resize images to a maximum dimension around 1024 pixels, call torch.cuda.empty_cache() between images, or run inference in half precision.

Increase points_per_side for a denser prompt grid, enable crop_n_layers for multi-scale processing, and lower min_mask_region_area. Alternatively, split the image into overlapping patches and run the mask generator on each patch.