What is OpenAI Codex?

When OpenAI introduced Codex, it did so with a number that is almost never quoted in full. In the paper "Evaluating Large Language Models Trained on Code," published on 7 July 2021 by Mark Chen and fifty-seven co-authors, the model solves 28.8% of the programming problems in its benchmark.

And a few lines later comes the other number: allowed one hundred attempts per problem, it solves 70.2%.

Both are true, both sit in the same abstract, and the distance between them is the most useful thing this system can teach you. Because one of the two only works if you have something you may not have.

What Codex is, from the record

It is an OpenAI language model fine-tuned on public source code, able to turn a natural-language instruction into code and to complete what you are writing. The paper ties it to the product almost everyone knows: "a distinct production version of Codex powers GitHub Copilot."

That sentence deserves attention on its own: a distinct production version. What was measured in the paper is not exactly what runs in anybody's editor. It is an honest precision from the vendor itself, and the kind always worth hunting for: published figures describe the laboratory model, not necessarily the product you pay for.

How "can it program" gets measured

To measure it they built a new benchmark, HumanEval, whose purpose is to "measure functional correctness for synthesizing programs from docstrings": you hand it a description of a function and check whether the code it writes passes the tests.

That definition is stricter than it sounds and worth holding on to. It does not assess whether the code "looks like" what a human would write, or whether it compiles: it runs it and checks that it does what was asked. It is an excellent methodological choice, because superficial resemblance is exactly what a language model can manufacture while understanding nothing.

To place the 28.8%, the same paper gives the comparators: GPT-3, without code fine-tuning, solved 0%. GPT-J solved 11.4%. In other words, code-specific fine-tuning did not improve the result a little: it created it from nothing.

The two numbers, and why only one is yours

Here is the capability. The 28.8% is what it gets right on the first attempt. The 70.2% is what it gets right given a hundred and picking the one that works.

And there is the small print: to pick the one that works you must know which one works. In the experiment that is trivial, because automated tests decide. In your job, perhaps not.

If you have a test suite that verifies the result, the 70.2% is within reach: generate several solutions, run them, keep the one that passes. If you do not, that number does not exist for you; yours is the first attempt, or worse, because you will be reviewing a hundred candidates by hand to decide which is right — which is precisely the work you were trying to avoid.

Generalise, because this shape of figure is everywhere: any number of the form "best of N attempts" silently assumes somebody able to decide which is best. That somebody is called a verifier, and it is the real thing to ask about. When you read "it gets it right 90% of the time," the question is not where the 90% came from but who decided it was right — and whether that judge will be available on the day you use it.

GPT-3's zero says more than the 28.8

Go back for a moment to that 0%. GPT-3 was, at the time, among the most powerful language models in existence, trained on vast quantities of internet text — which includes an enormous amount of code. And it solved zero problems.

It was not that it programmed badly: it did not program. The capability was not latent, waiting for someone to unlock it with the right prompt. It had to be built, by training specifically on code.

That refutes a comfortable and widespread intuition: that a large general model "knows a bit of everything" and you merely have to ask nicely. Specialisation is not a matter of presentation: it is the difference between zero and something. And it carries a cost — specific data, specific training, specific evaluation — that somebody pays.

Keep it for the next time a general model is offered as the answer to a very specific problem. The right question is not whether the model is powerful, but whether anyone has done the work of specialising it and can show you the measurement.

How to measure it on your own code

The paper's methodology can be copied in an afternoon and is worth far more than any comparison chart.

Take twenty real tasks from your own history — ones that already have tests written, and they exist. Hand the tool the description exactly as it was written, without polishing it to help the tool succeed. Run whatever comes back against the tests that already existed. Count how many pass on the first attempt.

That percentage is your 28.8%, and it is the only one useful for deciding. If it comes out well above the published figure, check whether the tasks you picked were the easy ones; if well below, you now know your domain does not resemble the benchmark — which is exactly what you needed to find out before buying anything.

What the model does NOT do, however often it is repeated

One widespread idea needs dismantling, because it changes decisions: Codex does not learn from you while you use it.

A model's weights are fixed when training ends and are not modified by conversing with it. That it appears to adapt within a session is a different thing: your code is being passed to it as context, and when the session closes that context is gone. Nothing you wrote has changed the model.

The distinction matters for two very practical reasons. One: do not expect it to improve at your habits through use alone; if you want it to know your codebase, you must supply it each time or build a system that retrieves it. Two, and more serious: if the system does not learn from your data, neither is it absorbing them merely because you use it — which does not mean they are not stored somewhere else. That last point is not decided by the architecture but by the service contract, and that is where to look.

Where it fails, according to those who built it

The paper states its own limits, and they are the ones you notice in use: the model shows "difficulty with docstrings describing long chains of operations and with binding operations to variables."

Translated into practice: it does well with single-step requests — "sort this list," "validate this email" — and degrades when the instruction chains several conditions that depend on one another. That is exactly the border between autocompleting and programming, and the authors declared it from day one.

Where to go next, with no middlemen

The original paper is public and free, and its limitations and broader-impacts sections are more interesting than its results. HumanEval, the benchmark, was released alongside the work, so anyone can reproduce the measurement instead of taking it on faith.

The capability you take from this is always asking who acts as judge: a "70% success rate" with no verifier is not a result you can use, it is a result somebody was able to check in their own lab.