Deep Learning
Deep learning uses many-layered networks to learn hierarchical representations of data, without hand-designed features. We review its fundamentals (backpropagation, representation learning), its architectures and scaling, and its limits: data, compute, robustness, interpretability, hallucinations and bias.
Deep learning is the branch of machine learning that uses neural networks with multiple layers to learn hierarchical representations of data directly from examples, rather than relying on hand-designed features. What sets it apart is not “thinking like a brain,” but that the useful features are learned on their own, layer by layer.
Fundamentals
The central idea is representation learning: where classical machine learning required an expert to design the relevant variables, a deep network discovers them by itself. The algorithm that makes this practical is backpropagation, which propagates the error from the output backward to adjust the weights. Thus the early layers detect simple features—edges in an image—the middle ones combine them into parts, and the deep ones into abstract concepts. In the words of LeCun, Bengio and Hinton (Nature, 2015), these models “learn representations of data with multiple levels of abstraction.”
Architectures and scaling
The main architectures are convolutional networks (vision), recurrent networks (sequences) and, since 2017, Transformers, today dominant in language and multimodality. A central driver of recent years has been scaling: performance improves fairly predictably when data, compute and parameters are increased together. The work known as “Chinchilla” (2022) refined the optimal recipe, showing that model size and the amount of training data should be scaled in parallel. From this come the foundation models: models trained on broad data that are then adapted to many different tasks.
Efficiency and limits
Against the race for size there is an equally active line of efficiency: distillation, quantization and small models meant for resource-limited devices. And the limits deserve to be named plainly: deep learning needs large volumes of data and compute; it is not very robust to perturbations or distribution shifts; it remains largely a “black box”; generative models can “hallucinate,” that is, produce plausible but false statements; and it tends to inherit the biases present in the data it is trained on.
