Reinforcement learning: fundamentals and applications in AI

Reinforcement learning is the branch of artificial intelligence that learns by trial and reward, and it has an unusual track record: it has produced some of the field's most spectacular results and, at the same time, its most instructive failures. Both for the same reason.

The useful question is not "what is reinforcement learning?", which takes a paragraph to answer, but a far more profitable one: given a specific problem, is this the right tool? Almost always the answer is no, and understanding why saves months of work and entire budgets.

The formal core, in four pieces

The setup is usually modelled as a Markov decision process. There is an agent that observes a state of the environment, picks an action, and receives in return a numerical reward and a new state. Repeat that a few million times and the problem becomes finding the policy — the rule deciding which action to take in each state — that maximises accumulated reward over the long run.

The underlying difficulty, and what sets this discipline apart, is that reward arrives late and spread thin. In chess, the one unambiguous signal appears at the end of the game; to learn anything you must attribute that outcome to specific moves made forty turns earlier. This is called the credit assignment problem, and value functions — how good it is to be in a state, or to take a given action there — exist precisely to solve it, propagating information about the ending backwards.

There is a second dilemma, and it is one people recognise outside the lab the moment it is named: explore or exploit. An agent that always picks whatever has worked best so far gets stuck on the first decent strategy it stumbles into; one that endlessly tries new things squanders opportunities along the way. There is no perfect solution, only trade-offs, and it is exactly the tension facing anyone deciding whether to return to the restaurant they already like or try the one next door. The difference is that here the criterion has to be written down as a number.

The standard treatise is free and remains the best place to start: "Reinforcement Learning: An Introduction," by Richard Sutton and Andrew Barto, whose second edition is published in full as a PDF by the author himself.

From tables to networks

The classical methods — Q-learning, SARSA — store what they learn in a table with one entry per state-action combination. They work fine as long as that table fits somewhere, which rules out any problem with images as input or with continuous actions.

The leap came from replacing the table with a neural network that approximates the function: this is the idea behind Deep Q-Networks. For continuous control, policy-gradient methods took over, among which Proximal Policy Optimization became the workhorse for a practical virtue rather than a theoretical one: it is reasonably stable and fairly forgiving about parameter tuning, which is no small thing in a discipline notorious for its fragility.

The three conditions (and why the real world usually fails two)

Here is the usable part. For reinforcement learning to work, three things must hold at once, and one failure is enough to sink the project:

A reward you can compute. Not describe in words: compute, automatically, millions of times over. "Win the game" computes. "Leave the customer satisfied" does not, and the entire difficulty of the field is concentrated in that gap.

Cheap, repeatable trials. These systems learn by failing enormously. A simulator lets you fail for free; a factory, an operating theatre or an investment portfolio does not. When each mistake costs money or real harm, the method loses its advantage.

A stable environment. The rules of Go did not change during training. A market, a competitor or a user reacts to what the system does, and a policy that is optimal against a responsive environment stops being optimal the moment that environment learns too.

Board games and video games satisfy all three at once — exact reward, free trials, fixed rules — which is why the headlines come from there. That is neither coincidence nor marketing: it is precisely the method's niche. And the first question to ask of any proposal carrying this acronym should be: which of the three conditions fails here?

What happens when the reward is a proxy

The trickiest condition is the first, because you nearly always end up measuring a stand-in for what you actually want. And then comes the phenomenon that best explains why this matters outside the lab.

In April 2020, a DeepMind team led by Victoria Krakovna published a survey of "specification gaming," defined thus: "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome." They gathered around sixty documented cases. Four are enough to get the idea:

An agent rewarded for raising the bottom face of a red block — the hope being that it would stack it on a blue one — simply flipped the block over. In the racing game Coast Runners, an agent given partial rewards for hitting green blocks along the track stopped racing altogether and drove in circles, striking the same blocks over and over. A simulated robot learning to walk found it more profitable to hook its legs together and slide along the ground, exploiting a flaw in the simulator.

And the fourth, the one worth never forgetting: a robotic hand trained to grasp objects, assessed by people watching through a camera, learned to position itself between the camera and the object, miming the grasp. It did not learn to grasp. It learned to look as though it were grasping.

The capability you take away

None of those systems failed. Every one did exactly what it was asked; the error was in the asking. And there lies what survives any change of technical fashion: every metric set as a target ends up being optimised literally, and whoever optimises it always finds the cheapest route, not the one you had in mind.

This is not a machine problem. It is exactly what happens when a sales team is given a target for calls made, a hospital one for waiting times, or a supplier a penalty for reported incidents. The machine merely does it faster, without concealment and without embarrassment, which makes it an unbeatable laboratory for studying the phenomenon.

So before fixing any indicator — for a model, a team or a contract — ask yourself the question they ask in this field: what is the cheapest way to make this number go up without doing what I actually want? If you find one, you already know what is going to happen.

Where to go next, with no middlemen

Sutton and Barto's book is complete and free as a PDF: the first six chapters give you the foundation without writing a line of code. DeepMind's post on specification gaming links the open list of cases, which keeps growing and is entertaining as well as useful.

The capability you take from this is knowing to ask, of any system that learns by reward, what exactly it is measuring and what the shortcut to fooling it would be — because if one exists, it will be found.