Reinforcement Learning

Reinforcement learning (RL) is the paradigm in which an agent learns by doing. Placed in an environment, the agent observes its state, takes an action and receives a numerical reward in return; by repeating that loop through trial and error, it shapes its behaviour to maximise the cumulative reward over time rather than any single immediate payoff.

The contrast with supervised learning is fundamental. There, a model learns from labelled examples that spell out the correct answer; here, no one reveals which action was best. The agent has only a reward signal —often sparse and delayed— and must discover for itself which sequence of decisions leads to the best outcome. It learns from the consequences of its actions, not from a teacher.

The building blocks and the Markov decision process

RL is formalised as a Markov decision process (MDP): a set of states, a set of actions, a dynamic that governs how one state leads to the next, and a reward function. The Markov property requires that the future depend only on the present state, not on the entire past history.

Two central pieces operate within that frame. The policy is the agent's strategy: the rule that, given a state, decides which action to take. The value function estimates how much reward can be expected in the long run from a state (or from a state-action pair), acting as a compass for improving the policy. The goal of learning is to find a policy that maximises expected value. Richard Sutton and Andrew Barto set out this vocabulary in the field's standard text, “Reinforcement Learning: An Introduction”.

Explore or exploit: the dilemma and its methods

Because the agent learns while it acts, it faces a constant tension: exploration versus exploitation. To exploit is to pick the action that looks best right now; to explore is to try alternatives whose value is still unknown, in case they turn out better. Too much exploitation traps the agent in a mediocre optimum; too much exploration squanders reward. Balancing the two is one of the field's founding problems.

Three families stand out among the classical methods. Temporal-difference (TD) learning updates its estimates from other estimates, without waiting for the episode to end. Q-learning, formulated by Christopher Watkins, learns the value of each action directly and converges to the optimal policy even when the agent explores sub-optimally. And policy-gradient methods adjust the policy's parameters directly in the direction that increases expected reward, bypassing an explicit value function.

From the game board to RLHF

For years RL shone in closed domains —board games, video games, robot control— where the reward is clear and games can be played by the million. When it is paired with deep neural networks to handle high-dimensional environments, we speak of deep reinforcement learning, which this encyclopedia covers in its own entry.

Open fronts remain. Reward design is still delicate: a poorly framed signal pushes the agent to exploit shortcuts instead of solving the task. And sample efficiency —how many interactions an agent needs in order to learn— limits its use in the physical world, where every trial costs time and resources.