Deep Neural Networks

A deep neural network is an artificial neural network with several hidden layers between input and output. The adjective “deep” refers to that depth—the number of layers—and to no other quality. Each layer transforms the output of the previous one, so the network composes successive representations of the data.

A clarification up front: a deep network is not a “subdiscipline” but a class of models defined by its depth. The subdiscipline is deep learning as a field; deep networks are the models that field works with.

Deeper does not mean better

It is a common mistake to claim that deep networks generally outperform shallower ones. Performance depends on the architecture, the available data, regularization and the task; adding layers blindly may not help, or may even hurt. In fact, the universal approximation theorem shows that a network with a single hidden layer and enough neurons can already approximate any continuous function: in theory, depth is not essential for expressiveness. What depth contributes in practice is efficiency: a deep network can reach the same accuracy with far fewer parameters than an equivalent wide network, and it builds a hierarchy of representations. But it has a price—higher compute cost, risk of overfitting, and training difficulties such as the vanishing gradient—mitigated, though not eliminated, by techniques such as ReLU activations, normalization or residual connections.

Architectures

Different architectures are “deep” when they stack several layers, and they differ in how they connect them: the multilayer perceptron stacks dense layers; convolutional networks, convolution blocks, dominant in vision; recurrent networks process sequences; and Transformers stack attention blocks and are the basis of today's language and multimodal models.

Scale and efficiency

Scaling—more parameters, more layers and more data—has driven the foundation models of recent years. But exaggeration should be avoided: in parallel, more efficient alternatives are being explored, such as state-space models or attention-free architectures, and competitive small language models. The real trend is not “unlimited depth,” but obtaining more capability with less compute, which reinforces the idea that the advantage depends on the problem, not on depth in the abstract.