Computer Vision and AGI: Telling a Machine That Sees From One That Scores

Every time a vendor announces that its system «recognizes objects with superhuman accuracy», it is leaning on a chain of results that began at an academic competition in 2012. It is worth knowing what was actually measured there, because decisions that affect everyone now hang from that measurement: which airport camera flags a suitcase, which system in a car decides the thing ahead is a shadow rather than a pedestrian, which medical application marks a spot on an X-ray.

The short answer is that 2012 did not measure «seeing». It measured something far narrower, and far more useful to understand.

What happened in 2012, with the exact numbers

The contest was ILSVRC, the ImageNet Large Scale Visual Recognition Challenge. Its main task was to assign each photograph one of 1,000 categories, training on roughly 1.28 million images, as documented by Olga Russakovsky, Jia Deng, Li Fei-Fei and the rest of the team in the challenge's reference paper.

The 2012 edition was won by a convolutional network built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton. In their own words in the NeurIPS 2012 proceedings, it achieved «a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry». Ten percentage points in one go, in a field that fought over tenths. The network had 60 million parameters and 650,000 artificial neurons.

Now the part that almost never gets told: «top-5 error» means the system offers five candidate labels and is only charged with a mistake if the correct one is absent from all five. It is not «it was right 84.7% of the time». It is «in 84.7% of cases, the right answer was somewhere in its list of five». That is a legitimate, comparable metric, and it did exactly the job it was designed for. But it is not a measure of visual understanding, and anyone who translates it into «the machine sees better than a human» has swapped the meaning along the way.

The experiment that showed where the limit was

The demonstration arrived a year later, from the field's own protagonists. In December 2013, Christian Szegedy, Ilya Sutskever, Ian Goodfellow, Rob Fergus and colleagues published «Intriguing properties of neural networks», where they write that «we can cause the network to misclassify an image by applying a certain imperceptible perturbation».

Imperceptible in the literal sense: the human eye sees the same photograph before and after. The network, meanwhile, switches categories with complete confidence. And the paper's most uncomfortable finding is that the same perturbation also fooled a different network trained on a different subset of the data. This was not one badly tuned model misbehaving; it was a property of the method.

The durable lesson is not «networks fail». Everything fails. The lesson is that a system can post an excellent number on the task it was measured against while relying on regularities in the image that have nothing to do with what the object is. The number was real. The interpretation was not.

Ten years on: borrowed generality

The field moved, and moved a long way. In 2021 a team at OpenAI introduced CLIP, trained on 400 million image-text pairs gathered from the internet. Its most-quoted result, in their words: «we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples». That is, without having seen that particular test's training set at all.

This is a genuine qualitative jump, and it explains why a single model today can describe a photo, read a chart and answer questions about a video. The Gemini technical report, published by Google in December 2023, states that its largest version «advances the state of the art in 30 of 32» benchmarks evaluated.

But look at the shape of the claim, because it is 2012's claim in new clothes: it is still a tally of wins on tests. Better designed, more varied, harder to memorize. Tests all the same.

Why vision is the best thermometer of generality

This is where the subject meets the argument about artificial general intelligence. In «Mind Children» (Harvard University Press, 1988), Hans Moravec set out the observation that now carries his name: it is comparatively easy to make a machine perform at adult level on intelligence tests or at games, and comparatively very hard to give it the perceptual and motor skills of a one-year-old.

The reason is evolutionary, which is why it does not expire. Abstract reasoning is recent in our species and still costs us conscious effort, while seeing and interpreting the world has had hundreds of millions of years of refinement and happens without our noticing. What we do effortlessly is the hardest thing to reproduce, precisely because we cannot explain how we do it.

That is what makes perception such a good thermometer. A system reasoning over symbols someone else has already digested for it starts halfway home. A system that has to work out what is out there, in fog and backlight and half-occluded, cannot hide behind the way the problem was posed.

The capability: three questions for any «this AI can see»

Here is what to take away, and it will still work a year from now under whatever product name:

1. Measured on which task, and which set? «99% accuracy» means nothing without the test. Ask for the dataset, the number of categories, and whether the metric allows several attempts, as top-5 does. A percentage without its task is advertising.

2. Does it survive a small perturbation? A robust system does not change its mind because the lighting shifted, the framing moved, or invisible noise was added. If the vendor has published nothing about robustness, that does not mean robustness is good: it means it was not measured, or is not being shown.

3. Was it tested away from where it was trained? The interesting result is not the test the team chose, but the situation it did not anticipate: another hospital, another camera, another country, another light. If every number comes from the same environment, what is being measured is memory of that environment.

None of these three questions requires mathematics. They require knowing that a number always answers one specific question, and that the question travels worse than the number.

The deep end: the sources, undiluted

All of the above is checkable in open primary documents, and they reward opening:

• Krizhevsky, Sutskever and Hinton (2012), the AlexNet paper in the NeurIPS proceedings. Useful warning: the abstract page shows the submitted version's figures (37.5% and 17.0% error on the 2010 edition); the 15.3% against 26.2% appears in the final proceedings PDF.
• Russakovsky et al. (2014), the full description of the ImageNet challenge: how it was built, what was annotated and under which criteria.
• Szegedy et al. (2013), the adversarial examples paper.
• Radford et al. (2021), CLIP, to understand what «zero-shot» does and does not mean.
• Gemini technical report (Google, 2023), as a current example of how a lab presents its own results.

One is missing, and its absence teaches something too. The founding paper of this whole discussion, «Computing Machinery and Intelligence», which Alan Turing published in Mind in October 1950, sits behind a paywall at Oxford Academic: its PDF is listed as available to subscribers only. It can be cited by its permanent identifier (DOI 10.1093/mind/LIX.236.433), but anyone wanting to read it in full will have to pay or go through a library. Worth remembering when someone quotes it from memory: the text that founded the debate over whether machines think is not one click away for the average reader.

That is the capability you leave with: faced with any system that boasts of seeing, ask which task was measured, whether it survives a perturbation, and whether it was tested away from home. Those three questions read a 2012 announcement, a today announcement, and one five years from now.