Neural networks and deep learning

On 30 July 2026, calling a neural network “intelligent” still conceals more than it explains. A network does not understand a task as a person does, nor does it improve through intent: it chains mathematical operations whose parameters are adjusted to reduce an error measured on data. Deep learning is the form of this idea that uses several layers of representation. Understanding a demonstration, product or failure requires more than recognising labels such as CNN, RNN or GAN. It requires following the cycle that turns an input into a prediction and a wrong prediction into a parameter update.

That cycle provides a practical skill: for any network, identify its input, architecture, output, loss, validation data and cost of error. If one is missing, an evaluable system has not yet been described.

A network is a parameterised function

A unit receives numbers, computes a weighted combination, adds a bias and applies a nonlinear function. A layer contains many units; several layers compose a function. Weights and biases are the parameters changed by training. The biological-neuron metaphor gave the field its name, but it is not an equivalence: these components do not reproduce the organisation, chemistry or plasticity of a brain.

Depth matters because layers can build successive representations. In an image, some operations may first respond to local contrasts; later operations combine those patterns into structures useful for the task. In text, a representation can integrate relationships among elements in a sequence. The review by LeCun, Bengio and Hinton defines deep learning through multiple levels of representation, not through any particular application.

More layers do not guarantee a better model. They can also increase the capacity to fit accidental regularities, make optimisation harder and raise cost. “Deep” describes computational structure; it does not certify conceptual depth, generality or reliability.

The forward pass: from input to output

The first step in both training and inference is the forward pass. An input is transformed layer by layer until the network produces an output. In classification, that output may become scores or probabilities for classes. In regression it may be a number. In a generative model it may be a distribution over the next element or a sample built from a latent signal.

Input form is not a minor detail. An image may be represented as a matrix of pixels; a sequence as ordered vectors; a table as normalised variables. Preprocessing determines which information the model retains and which assumptions it introduces. If the training set contains a mark that accidentally coincides with the label, the network may exploit that shortcut. Its calculation can be correct while its solution becomes useless outside those data.

Loss turns the objective into a quantity

Adjusting parameters requires a measure of the discrepancy between output and target. This is the loss function. It is not synonymous with the metric that matters to the user. A model may optimise cross-entropy and later be judged by accuracy, recall or calibration; a mean error can conceal severe failures in one group. Choosing the loss determines which errors exert greater force during training.

The actual objective often combines loss, regularisation and constraints. “The model learned from data” is therefore incomplete. It learned under an architecture, a sample, a loss function, an optimisation algorithm and a budget. Changing any of them can change what is learned.

Backpropagation and optimisation are different jobs

Backpropagation calculates how the loss would vary under small changes to every parameter. It traverses the chain of operations in reverse, applies the chain rule and obtains gradients. The 1986 paper by Rumelhart, Hinton and Williams showed how such adjustments enable hidden units to form useful representations. It was not the origin of all automatic differentiation, but it was pivotal in training distributed representations in multilayer networks.

The optimiser uses those gradients to choose an update. Stochastic gradient descent estimates a direction from batches of examples. Adaptive methods such as Adam maintain moment estimates of the gradient. Backpropagation answers “in which direction does the loss change?”; the optimiser answers “what step should we take?” Confusing them prevents a diagnosis of whether a failure comes from gradient calculation, learning rate, sampling or the objective itself.

The cycle repeats: batch, forward pass, loss, backpropagation, update. An epoch goes through the training set once, although order and batch composition may vary. A falling training loss demonstrates only that the procedure is fitting that sample under that objective.

Network families encode different assumptions

A fully connected network allows every unit in one layer to receive all outputs from the previous layer. It is flexible, but it ignores spatial or sequential structure unless the data and training reveal it.

Convolutional networks share filters across positions. This assumption favours local pattern detection and reduces parameters relative to an equivalent dense layer. The ImageNet work by Krizhevsky, Sutskever and Hinton combined deep convolutions, GPU computation and regularisation for visual classification. Its result supported that configuration on that benchmark; it did not make every CNN a good vision system.

Recurrent networks reuse a state while traversing sequences. Variants such as LSTM introduced mechanisms for preserving and controlling information over long intervals. Transformers reorganised sequential processing around attention. These are different architectural decisions, not degrees of “intelligence”.

Generative adversarial networks train a generator and a discriminator with opposing objectives. The original work by Goodfellow and colleagues framed this as a minimax game. A GAN is not a layer attached to any network: it is a training regime with two models and its own stability and evaluation problems.

Generalisation is the test, not memorisation

A model overfits when it performs better on known examples than on relevant new data. Separating training, validation and test sets prevents the final exam from being used to make decisions. Validation guides architecture and hyperparameters; the test estimates performance once the procedure is fixed. If the test is consulted repeatedly, it is no longer an independent measurement.

Regularisation modifies training to constrain solutions that fit too narrowly. L1 and L2 penalties act on weights. Early stopping uses the validation trajectory. Dropout randomly removes units and connections during training. No such technique repairs bad labels, information leakage or a sample that fails to represent deployment. Nor should one be added by habit: each introduces a bias that must be tested.

A workflow that supports diagnosis

1. Define the prediction unit. State what enters, what comes out and when the decision is made.
2. Build a simple baseline. A rule or small model shows whether the network adds value beyond a cheap solution.
3. Split data before tuning. Avoid placing the same person, device or time period on both sides when that would leak information.
4. Record the curves. Training and validation losses and metrics help distinguish underfitting, overfitting and instability.
5. Run ablations. Remove components to test which part causes an improvement.
6. Evaluate error cost. Break results down by classes, groups and use conditions; an average is insufficient.
7. Document inference. Latency, memory, consumption and behaviour on out-of-distribution inputs are part of the real model too.

The transferable skill is reading a network as a circuit of testable decisions: forward representation, loss, gradient, update and independent evaluation. Architecture families help encode suitable assumptions. Only evaluation shows whether those assumptions captured a useful regularity rather than memorising the setting.