OpenAI seeks to explain neuron behavior in natural language models

In May 2023, OpenAI published work titled "Language models can explain neurons in language models," whose own description sums the method up in a sentence: "We use GPT-4 to automatically explain neurons in GPT-2, and to score those explanations."

The second half of that sentence is the half that matters, and it is precisely the half that fell out of nearly every summary that circulated. Getting a machine to produce an explanation is easy. Subjecting that explanation to a test it could actually fail is something else entirely, and it is what separates a scientific result from a well-told story.

Why explaining a single neuron is hard

Inside a language model sit hundreds of thousands of units that fire more or less strongly at each fragment of text. Picking one and asking "what does this one handle?" seems reasonable, and for years it was done by hand: a researcher would watch which texts set a particular unit off and propose a label — "fires on country names," "responds to closing quotation marks."

That procedure has two problems. It is desperately slow, because there are far too many units to inspect one at a time. And it cheats without meaning to: if you look at the ten texts where a neuron fires hardest and hunt for what they share, you will almost always find something. The human mind is an excellent machine for manufacturing patterns out of ten examples. The resulting label may describe the neuron, or it may describe the researcher.

The three steps, and why the second changes everything

OpenAI's method is organised, per the document itself, into three stages: explanation, simulation and scoring.

In the first, a capable model is shown text examples together with what the neuron did on each, and writes a natural-language hypothesis about what activates it. So far this is the old manual practice, automated.

The second stage is the real idea. Instead of asking "does this explanation sound convincing?", you take the explanation, hand it to a model without showing it the real activations, and ask it to predict how strongly that neuron would fire on fresh text. In other words: if the explanation is right, it ought to be good for predicting.

The third stage compares those predictions against what the neuron actually did and produces a number. An explanation that fails to predict scores badly, however elegant it sounds.

Here is the capability you take away, and it reaches far beyond interpretability: an explanation that lets you predict nothing is not an explanation, it is a description. It applies to why a server fell over, why a stock rose, or why a treatment works for a given patient. The question that separates them is always the same: if what you are telling me were true, what should happen next time? And then: shall we check?

And who checks the checker?

One obvious objection remains, and it is the one always worth raising about an automated system that evaluates itself: if the explainer and the scorer are both language models, how do we know they are not simply agreeing with each other?

The repository shows how this is approached. Among the published usage examples is one devoted to "generating explanations for human-written neuron puzzles": cases built by hand where the right answer is known in advance. If the automated system solves puzzles whose solutions we already know, it earns part of the trust it needs for the cases where we do not.

That idea — deliberately manufacturing problems with known answers in order to calibrate an instrument — is as old as metrology and remains the best defence against a system that marks its own homework. It applies equally to a model, a medical test or a corporate dashboard: before I believe you about what I don't know, I want to watch you get right what I do.

There is a second safeguard inside the method itself. Explanations are generated along two distinct routes — from the activations, and from the tokens with the highest average activations — and having two paths lets you see whether they agree. Two instruments measuring the same thing by different procedures are far harder to fool than one.

What was released — which is what makes checking possible

The work did not stop at the paper. OpenAI also published the code and the datasets: neuron activations for GPT-2 XL, model-generated explanations with their simulation results, lists of related neurons and of tokens with high average activations. Plus a viewer for walking through neuron by neuron.

This matters more than it looks. An interpretability result that exists only as an assertion in a paper cannot be audited. Publishing the activations and the scores lets a third party look at the same neurons and disagree with data in hand. It is the difference between "trust us" and "check it."

The small print almost nobody reported

And here is the most instructive detail in the whole story, which sits not in the paper but in the repository's later updates.

On 5 July 2023, explanations for GPT-2 Small were added, with an explicit warning: "The methodology is slightly different from the methodology used for GPT-2 XL so the results aren't directly comparable." Three lines that head off exactly the mistake a hurried reader would make.

Then on 30 August 2023 came something rather less common: the authors announced they had found a bug of their own. They had used an optimised GELU implementation rather than the original one associated with GPT-2. The model's behaviour is very similar under both, but the activation values used to generate and simulate explanations depart from the correct ones — for GPT-2 Small — by a median of 0.0090, with 0.0252 at the 90th percentile, 0.0839 at the 99th and 0.1736 at the 99.9th.

These are small deviations, and that is exactly the point. An implementation detail that barely moves the model's output does move the numbers being explained, because here the object of study is not what the model answers: it is its innards. Interpretability is sensitive to the exact substrate in a way that performance benchmarking is not.

The second lesson is about who publishes. A lab that comes back three months later to say "the numbers we gave you carry this bias, and here is its size" is doing science. When you read a correction like that, do not file it as a slip: file it as evidence that somebody is genuinely looking.

Where to go next, with no middlemen

The material is public and free. The full paper details the method and — this one is worth reading first-hand — its limitations section. The repository holds the code, the dataset descriptions and both updates cited above, at the end of the README. The neuron viewer is the fastest way to build your own intuition.

The honest conclusion of this work is not that we now understand language models. It is that a way has been built to measure how much we understand of each piece, which is the indispensable prior step. And the capability you take with you is the question to put to any explanation, whether it comes from a model or a person: what does it predict, and what would happen if it were wrong?