Image Segmentation

Image segmentation is the computer vision task of assigning a label to every pixel in an image, carving it into meaningful regions: which pixels belong to the pedestrian, which to the road, and which to the sky. Unlike classification, which labels the whole image, or object detection, which draws boxes around things, segmentation works at pixel precision.

Deep learning turned it into one of the most mature problems in computer vision; since 2023, foundation models have pushed it toward general-purpose use without task-specific training.

Semantic, instance, and panoptic

The field distinguishes three variants. Semantic segmentation assigns a class to each pixel without telling individuals apart: every car in the scene merges into a single car-class region. Instance segmentation does separate them, producing an individual mask for each car or each person; its reference model is Mask R-CNN (He et al., ICCV 2017), which extended the Faster R-CNN detector to output a mask per detected object. Panoptic segmentation, formulated by Kirillov et al. at CVPR 2019, unifies both: every pixel receives a class and, if it belongs to a countable object, an instance identity.

From U-Net to transformers and SAM

U-Net (Ronneberger, Fischer and Brox, MICCAI 2015) defined the convolutional era: its U-shaped encoder-decoder architecture, designed for biomedical images with few training examples, remains a standard in medicine. The next wave came with vision transformers: SegFormer (Xie et al., NeurIPS 2021) paired a hierarchical transformer encoder with a lightweight decoder and reached 84% mIoU on the Cityscapes validation set, while Mask2Former (Cheng et al., CVPR 2022) showed that a single masked-attention architecture can solve all three variants of the task.

Meta AI made the leap to foundation models with SAM (Segment Anything Model, April 2023): a promptable model — a click, a box — trained on SA-1B, a dataset of more than 1.1 billion masks over some 11 million images, able to segment objects it has never seen. SAM 2 (July 2024) extended it to video with a streaming memory, trained on roughly 51,000 videos from the SA-V dataset, and according to Meta runs six times faster than the original on images.

How it is measured

The basic metric is intersection over union (IoU): the ratio between the area where the prediction and the human annotation overlap and the total area they cover together; its per-class average (mIoU) is the standard on benchmarks such as COCO, Cityscapes, and ADE20K. Panoptic segmentation is evaluated with Panoptic Quality (PQ), proposed in the same Kirillov et al. paper, which combines mask quality with how reliably each instance is recognized. Medical imaging favors the Dice coefficient, used by work such as MedSAM (Ma et al., Nature Communications, 2024).

Uses and limits

Segmentation underpins the perception stack of autonomous cars, the delineation of organs and tumors in medical imaging, and automatic object cutouts in photo and video editing. Its limits have been measured too: Mazurowski et al. (Medical Image Analysis, 2023) evaluated SAM without fine-tuning on 19 medical datasets and found IoU ranging from 0.11 on spine MRI to 0.87 on hip X-rays; generality does not guarantee uniform results across domains. On bias, the SAM paper itself measured similar performance across perceived skin tones and gender presentations, but acknowledges that large segmentation datasets, SA-1B included, underrepresent Africa and low-income countries. Whether foundation models will replace specialized ones in critical domains such as diagnosis remains, for now, an open question.