Deep Neural Networks

A deep neural network is an artificial neural network with several hidden layers between input and output. The adjective “deep” refers to that depth—the number of layers—and to no other quality. Each layer transforms the output of the previous one, so the network composes successive representations of the data.

A clarification up front: a deep network is not a “subdiscipline” but a class of models defined by its depth. The subdiscipline is deep learning as a field; deep networks are the models that field works with.

Deeper does not mean better

It is a common mistake to claim that deep networks generally outperform shallower ones. Performance depends on the architecture, the available data, regularization and the task; adding layers blindly may not help, or may even hurt. In fact, the universal approximation theorem shows that a network with a single hidden layer and enough neurons can already approximate any continuous function on a bounded domain: in theory, depth is not essential for expressiveness. Depth can provide representational efficiency for particular function families. Telgarsky constructs functions represented by small deep networks that certain shallower networks can approximate only with exponentially more nodes. This is a separation result under defined functions and architectures, not a promise that every deep network reaches the same accuracy with fewer parameters. In practice, depth also lengthens the computation path and can complicate optimization; ReLU, normalization, and residual connections mitigate specific problems but do not remove the need to validate architecture, data, and regularization.

Architectures

Architectures count as “deep” when they compose many blocks, and differ in their operation and connection pattern. A multilayer perceptron stacks dense layers; a recurrent network reuses state across a sequence. AlexNet is a historical case of deep convolutional blocks applied to vision. The original Transformer replaced recurrence and convolution with attention for translation. Those papers document concrete architectures and tasks; “dominant” or “the basis of every current model” would require a defined date, corpus, and market share, so it is not treated here as a technical property.

Scale and efficiency

Scaling does not mean merely adding layers. A study of language-model scaling laws measured how loss varied with parameters, data, and compute within its experimental regimes. The report that defined foundation models emphasizes that a model trained at scale is later adapted to many tasks, alongside risks and limits. Designs can also change the dominant operation: Mamba uses selective states and linear time in sequence length. An efficient benchmark result does not establish a universal trend; quality, memory, latency, data, and hardware must be compared for the actual task.