Deep Reinforcement Learning

Deep Reinforcement Learning (Deep RL) combines reinforcement learning —the paradigm in which an agent learns to make decisions through trial, error and reward— with deep neural networks that act as function approximators. Instead of storing the value of every possible situation in a table, the network learns to estimate the policy (which action to take) or the value function (how good a state is), working directly from raw data.

This pairing breaks a classic bottleneck: traditional reinforcement learning collapses when the number of states is vast. The underlying machinery —Markov decision process, policy, value, reward— is covered in the «Reinforcement Learning» entry; here we focus on what the deep networks add.

Why deep networks: high-dimensional states

A board, a video-game screen or a robot camera produce states of enormous dimension: millions of shifting pixels. Enumerating them in a table is impossible. A deep neural network instead compresses that input into useful representations and generalises to unseen situations. As a result, an agent can learn to act by looking straight at the pixels, without a human specifying in advance which features matter.

Milestones: from DQN to AlphaGo

The first major milestone was DQN (Deep Q-Network). In «Human-level control through deep reinforcement learning» (Mnih and colleagues, DeepMind, Nature, 2015), a single agent learned to play dozens of Atari games from nothing but the pixels and the score, matching or beating human level on many of them. In 2016, AlphaGo (DeepMind, Nature) defeated Go world champion Lee Sedol 4 to 1 by combining deep networks with tree search. From that lineage descend AlphaZero, which mastered chess, shogi and Go through self-play alone, and the wider family of DeepMind systems that carried these ideas into scientific research.

Methods: value versus policy

There are two broad families. Value-based methods, such as DQN, learn to estimate how good each action is and pick the best one. Policy-gradient methods adjust the policy directly; among them, actor-critic approaches such as A3C blend both ideas. The most widely used today is PPO (Proximal Policy Optimization, OpenAI, 2017), which caps how much the policy can change at each step to keep training stable and simple.

Honest limits

Deep RL remains expensive and brittle. It suffers severe sample inefficiency: it needs millions of interactions to learn what a human grasps in a handful of tries. Training is unstable and sensitive to tuning. Transferring what an agent learns to a different environment is hard, and reward design is an art —a poorly framed reward yields absurd behaviour. Sample efficiency, stability, transfer and interpretability all remain open problems.