Semantic Segmentation

Semantic segmentation is the computer-vision task of assigning a class label to every pixel in an image. Instead of stating what a photo contains, it draws a dense map in which each point is tagged as sky, road, building, person or vegetation. The output is scene understanding at pixel resolution: not a single global answer but millions of local decisions that together trace the exact outline of each region.

That density is what sets it apart from other recognition tasks. Semantic segmentation does not count objects or tell them apart; it groups by meaning. Every pixel belonging to the class «car» shares the same label, whether the scene holds one car or twenty. Separating those instances is already a different task.

Versus classification, detection and instances

It helps to place it within the family of vision tasks. Image classification emits a single label for the whole picture («there is a cat»). Object detection goes further and locates each object with a bounding box, but a box is a coarse rectangle that includes background pixels. Semantic segmentation is finer: it labels the exact interior of each shape, with no boxes.

Its well-known limit is that it does not separate individuals of the same class: two people hugging form a single «person» blob. That is where instance segmentation comes in, telling object 1 from object 2 even when they share a class. Panoptic segmentation combines both: it assigns a class to every pixel (the semantic part) while also giving each countable object its own identity (the instance part), unifying background and foreground in one map.

Architectures: from FCN and U-Net to DeepLab and SAM

The decisive leap came in 2015 with the fully convolutional networks (FCN) of Jonathan Long, Evan Shelhamer and Trevor Darrell. Their idea was to replace the final layers of a classification network with convolutions, so the network could accept images of any size and produce a same-sized map of labels, trained end to end, pixel to pixel. That scheme popularized the modern approach.

On that foundation came the encoder-decoder design: an encoder compresses the image into features and a decoder rebuilds it at full resolution. Its exponent in biomedical imaging is U-Net (Ronneberger, Fischer and Brox, 2015), which has its own entry in this glossary. The DeepLab family (Liang-Chieh Chen and colleagues) introduced dilated, or atrous, convolutions, which widen the field of view without losing resolution, together with the ASPP module for capturing objects at several scales. In 2023, Meta AI released Segment Anything (SAM), a general, promptable segmentation model trained on more than a billion masks that can segment almost any object from a point or a box, though it does not by itself assign the class label that semantic segmentation requires.

Uses

The dense map is valuable whenever the exact «where» matters. In medical image diagnosis, it delineates tumors, organs or lesions in MRI and CT scans, helping to measure volumes precisely. In autonomous driving, it separates road, sidewalk, pedestrians and vehicles so the system grasps the drivable space. In satellite remote sensing, it classifies each pixel of terrain as cropland, water, forest or urban area, the basis of precision agriculture and environmental monitoring. In every case, the per-pixel label turns an image into an operational description of the scene.