Image Recognition

Image recognition is the computer-vision task that consists of identifying and classifying the content of an image: the objects, people or scenes that appear in it. It helps to distinguish three levels of detail: classification puts a label on the whole image, object detection locates each object with a box, and segmentation classifies pixel by pixel to draw its exact shape.

The turning point

For decades, recognition relied on hand-designed features crafted by engineers (descriptors such as SIFT or HOG), which worked in controlled conditions but failed in the face of real variation in light, scale and occlusion. Everything changed in 2012, when a neural network called AlexNet—by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton—won the great ImageNet visual recognition challenge by a crushing margin. That result unleashed the deep learning revolution in computer vision.

How a convolutional network works

The engine of that advance is the convolutional neural network (CNN). In broad strokes, they apply filters that sweep across the image detecting local patterns, and organize them into a hierarchy: the first layers recognize edges, the next textures, then parts and finally whole objects. Pooling layers summarize the information along the way, and a final part handles the classification.

What it is for

Its applications are enormous: medical diagnosis by image, autonomous driving, facial recognition, industrial quality control or the visual search for products.

Its limits

But it is not infallible. Systems depend on their training data, and when these are not representative, biases appear: a 2018 study showed that several commercial systems were far less accurate with dark-skinned faces, especially women's. Moreover, they are vulnerable to adversarial examples: small alterations of the pixels, imperceptible to a person, that can lead the network to be completely wrong.