Computer Vision: Five Tasks That Must Not Be Measured Alike

As of July 30, 2026, saying that an AI system “analyzes images” is as uninformative as saying that a hospital “works with patients.” Classifying a picture, locating objects, painting every pixel, tracking a trajectory, and generating a scene produce different outputs and require different annotations and metrics. The useful skill is to match question, label, model, and test. Without that fit, an “accuracy” figure does not reveal what the system does.

Before the model: define the visual unit

An image is a matrix of values; video adds time. But labels determine the learning object. In classification, the complete image receives one category. In detection, each object receives a box and class. In semantic segmentation, every pixel receives a class; instance segmentation also separates two objects of the same class. In tracking, an identity must persist across frames. In generation, the system produces pixels conditioned on text, an image, or another signal.

This ontology precedes training. Does a partially occluded bicycle count? Is a reflection a person? Where does an ambiguously bordered lesion end? Two reasonable annotators may disagree. Microsoft COCO explicitly combined detection, segmentation, and captions over complex scenes; its labels are not interchangeable with a collection of centered single-object images.

Classification: one label for the whole

A convolutional network shares filters across positions and builds hierarchical representations. AlexNet showed on ImageNet how a deep CNN trained with GPUs, data augmentation, and regularization reduced classification error compared with previous systems. The result belonged to that task, dataset, and protocol; it did not prove that the network “saw” like a person.

Accuracy works when classes are balanced and errors have similar costs. If one class is rare, inspect precision, recall, F1, the confusion matrix, and calibration. ImageNet top-5 counts whether the correct label appears among five candidates; it does not mean that five diagnoses would be acceptable. A metric retains the meaning of the procedure that produced it.

Detection: class, location, and threshold

A detector proposes boxes, classes, and scores. A prediction matches a reference only when their overlap exceeds a threshold, usually expressed as intersection over union (IoU). Average precision combines confidence ordering and matches; mAP aggregates across classes and, depending on the benchmark, multiple IoU thresholds.

“90% precision” is therefore insufficient. The system may miss small objects, duplicate a person, or draw poorly aligned boxes. Report confidence threshold, matching criterion, duplicate handling, object size, and recall. PASCAL VOC rules and COCO rules do not produce directly interchangeable numbers.

Segmentation: the question lives in every pixel

Semantic segmentation marks regions by class. Its IoU compares the intersection and union of predicted and true pixels; averaging across classes prevents the background from dominating. The Dice coefficient provides another overlap measure. Both can hide boundary errors or the loss of small objects, so results should be broken down by class and size.

DeepLabv3 used dilated convolutions to enlarge the receptive field and spatial pyramid modules to capture context at multiple scales. That mechanism is relevant to segmentation, not a universal improvement in “vision.” The right choice depends on whether the product needs the shape of a road, the presence of a component, or merely a global label.

Tracking: time adds identity

In video, detection in each frame is not enough. A tracker associates the same entity through occlusion, exit, and reentry. Failures include losing a trajectory, swapping two identities, or fragmenting one into several. Evaluation should separate detection quality from association quality.

Frame rate, resolution, camera motion, and latency must also be fixed. A system that succeeds while processing recorded footage slowly may fail in real time. For alerts, false alarms per hour and time to detection matter more than an average per frame.

Vision Transformers: changing the representation

Vision Transformer split images into patches, projected them as a sequence, and applied a Transformer encoder. With large-scale pretraining, it achieved competitive classification results. The comparison demonstrated that convolution is not the only way to introduce spatial relationships, but also that data, pretraining, and budget belong to the result.

A CNN includes inductive biases for locality and translation; a basic ViT relates patches through attention and needs positional encoding. Neither “understands scenes” by name. Selection requires comparing quality, data, memory, latency, input size, and adaptation under the same budget.

Generation: realism is not fidelity

A GAN trains a generator against a discriminator. It can produce plausible images without matching one reference. Fréchet Inception Distance, introduced in the work on TTUR, compares representation statistics between real and generated sets. A lower value can indicate closer distributions under that feature extractor, but it does not guarantee prompt alignment, freedom from artifacts, sufficient diversity, or rights over training data.

Visual production should separate at least perceptual quality, condition fidelity, diversity, memorization, safety, and provenance. Editing also requires measuring what changed and what should have remained. Video adds temporal consistency, flicker, and identity continuity. Asking “does it look real?” covers only one fraction.

The medical case requires a narrow claim

The CheXNet paper evaluated pneumonia and other label detection in chest X-rays under a particular dataset and comparison. It does not follow that “AI surpasses specialists” in clinical radiology: the study is not equivalent to the full diagnostic process, which includes history, multiple images, prevalence, decisions, and consequences.

A medical application must state cohort, hospital, equipment, reference standard, readers, threshold, and intended use. High sensitivity can come with many false positives. A retrospective test may degrade after a population change. The model may assist with one bounded task; evidence must preserve that perimeter.

Bias: inspect error by subgroup and condition

Global averages conceal differences by skin tone, age, lighting, camera, or geography. Gender Shades audited commercial gender-classification systems and reported intersectional disparities. The lesson is not to add “diversity” as a slogan, but to disaggregate performance across the groups and conditions where the system will operate.

Labels can also contain bias: imposed categories, annotators without context, or cultural disagreement. Document who labelled, which instructions they received, and how uncertain cases were resolved. In surveillance or moderation, a false positive is not just a wrong pixel; it may trigger an intervention against a person.

Robustness: small variations, large effects

Adversarial examples showed that small perturbations could change network predictions. Everyday robustness also includes blur, compression, rain, framing, new devices, and temporal drift. Tests should cover plausible transformations and attacks relevant to the actual adversary.

A visual explanation such as a heat map may help inspection, but it does not prove causation or safety. Validate it by perturbing regions, checking stability, and comparing with known evidence. An attractive visualization does not replace an error rate.

The matrix that prevents false comparisons

For each system, fill six columns: question; output—label, box, mask, trajectory, or image; annotation; metric; conditions; and error cost. Add the architecture last. A CNN, ViT, or GAN is then tied to its function instead of becoming a synonym for vision.

The durable skill is matching an output to its test. When someone boasts about “accuracy” without stating which unit was annotated and how a match was decided, half the experiment is missing. Seeing is not one task, and measuring cannot be one number.