segment-anything-model

Load a checkpoint with sam_model_registry, create a SamPredictor, call set_image once, then predict with point coordinates and labels or a bounding box. With multimask_output enabled, select the mask with the highest score.

ViT-H is the largest at 2.4 GB with the best accuracy but slowest speed, ViT-L is a medium 1.2 GB option, and ViT-B is the smallest at 375 MB and fastest. Use ViT-B when GPU memory or latency is constrained.

SAM alone does not accept text prompts; it only takes points, boxes, or masks. Combine GroundingDINO, which converts text descriptions into bounding boxes, with SAM to generate masks from those boxes.

The original SAM processes images only and has no tracking. Use SAM 2, which adds a streaming memory architecture to propagate masks across video frames after prompting on an initial frame.

The ViT-H checkpoint and large images can exceed GPU memory. Switch to the ViT-B model, resize images to a maximum dimension, clear the CUDA cache between images, or load the model in half precision.

The default grid density and minimum region area filter out small regions. Increase points_per_side, enable crop_n_layers for multi-scale detection, and lower min_mask_region_area to retain tiny masks.