How a Deep Network Learns: From Forward Pass to Gradient

As of July 30, 2026, calling a network “deep” only means that it chains several learned transformations. It does not mean the system thinks more deeply or simulates a brain. Its learning can be followed through a concrete cycle: compute an output, measure an error, assign responsibility backward, and update parameters. If a reader can narrate one turn of that cycle, architecture, objective, optimizer, and regularization become separable rather than a list of names.

The model is a function with parameters

A layer receives numbers, multiplies by weights, adds biases, and applies a nonlinear function. Without nonlinearity, stacking linear layers would remain one linear transformation. ReLU keeps positive values and maps negative ones to zero; Deep Sparse Rectifier Neural Networks studied rectified units and their training behavior.

Depth allows representations to be composed. In an image, early layers may respond to local patterns and later layers combine them; in text, successive states integrate context. Concepts are not manually assigned to each unit. Parameters move to improve an objective, and interpreting an activation requires tests.

First: the forward pass

Imagine a network receiving three measurements from a component and estimating the probability of a defect. Each neuron computes a weighted sum; ReLU adds nonlinearity; the last layer produces a score and a sigmoid maps it between zero and one. Traversing input→layers→output with the current weights is the forward pass.

Intermediate values are retained because derivatives will need them. Training mode may include operations such as dropout; evaluation mode must disable them or use the appropriate stored statistics. Mixing the modes changes output without changing weights.

Second: a loss turns the objective into a number

A label alone does not say how to correct the model. A loss function compares output and reference. In binary classification, cross-entropy penalizes the probability assigned to the wrong class. In regression, squared error emphasizes large deviations. Ranking and generation use other objectives.

Choosing a loss means choosing which errors push parameters harder. A rare class may be weighted or sampling may change when labels are imbalanced. But optimizing a loss does not guarantee the product metric: precision, recall, calibration, and cost by error type are validated separately.

Third: backpropagation applies the chain rule

The output depends on the final layer, which depends on the previous one, all the way to every weight. The chain rule expresses how the loss changes after a small change in each parameter. Backpropagation reuses results from output to input to calculate those gradients efficiently. The classic paper Learning representations by back-propagating errors explained how internal layers could be adjusted through this propagation.

A gradient contains no verbal explanation and does not find a global minimum in one step. It is a local direction. It may become tiny through many saturated operations or grow unstably. Initialization, normalization, residual connections, and activations affect that path.

Fourth: the optimizer takes a step

Gradient descent subtracts a fraction of the gradient from each weight. With minibatches, the direction is estimated from a sample and contains noise. The learning rate controls step length: too high may diverge; too low may consume an enormous budget.

Adam keeps moment estimates of gradients and adapts steps by parameter. Fast training convergence does not guarantee better generalization. Comparing optimizers requires a fixed budget, tuned hyperparameters, and training and validation curves rather than only the best run.

Batch, epoch, and update are not synonyms

A batch is the set used for one gradient. An update applies that gradient. An epoch passes approximately once over the training examples. Under augmentation, sampling, or streams, “one epoch” may be a convention. Reports should include examples, steps, batch size, and operations.

A large batch uses hardware efficiently but changes optimization noise and memory. Gradient accumulation simulates a larger batch across steps. Reproducibility requires seeds, order, numerical precision, data version, and checkpoints; nominally identical runs can diverge.

Generalization means performing beyond tuning material

Training, validation, and test sets have distinct roles. Training changes weights; validation selects hyperparameters and stopping time; test estimates final performance. Repeatedly consulting the test effectively converts it into validation. Related examples should also be grouped so near-duplicates cannot cross splits.

Overfitting appears when training loss improves while relevant outside performance worsens. A smaller model is not the only remedy: the gap may expose unrepresentative data, noisy labels, or a wrong objective. Temporal and external tests are often harder than a random split.

Regularization changes data, function, or capacity

Dropout randomly disables units during training and approximates an ensemble of subnetworks; inference uses the complete network with corresponding scaling. Data augmentation applies transformations that should preserve labels. Weight penalties discourage large parameters. Early stopping limits how far fitting proceeds.

Batch normalization, described in its 2015 paper, normalizes activations through minibatch statistics in training and accumulated statistics during inference. It can have regularizing effects, but its main role is not to be a universal vaccine against overfitting. Small batches and domain shifts require care.

Architecture determines what is shared and connected

A CNN shares filters across positions; an RNN reuses a transition through time; a Transformer mixes positions through attention. A ResNet adds residual connections that make deep networks easier to optimize by learning corrections over an identity path. These are assumptions about data structure, not levels of intelligence.

The Transformer showed that attention and feed-forward layers could replace recurrence in translation and train in parallel. A new task still requires input, output, loss, data, and metric to be specified. The architecture name does not complete the experiment.

AutoML automates search, not judgment

AutoML may select transformations, hyperparameters, or architectures. auto-sklearn combined Bayesian optimization, meta-learning, and ensembles. It searches inside a space and against a metric defined by people.

If the dataset is filtered or the metric rewards a wrong shortcut, automation multiplies the problem. Search budget, trials, and validation data belong to the result. “The model designed itself” erases the human decisions that bounded the search.

Federated means distributed data, not automatic privacy

In Federated Averaging, clients train locally and a server aggregates updates. Raw data need not be centralized, reducing one exposure route and enabling learning from distributed information. But updates can leak signals, malicious clients can poison the model, and participation reveals metadata.

Deep Leakage from Gradients demonstrated data reconstruction from gradients in certain settings. Privacy therefore needs a threat model and controls such as secure aggregation, differential privacy, authentication, and contribution limits. Performance disparities across devices and users must also be measured: averaging does not make data representative.

Auditing one turn of learning

Select one batch and record input, weight version, activations, output, loss, gradients, optimizer step, and new weights. Confirm that loss falls on a tiny example the model should be able to memorize; then confirm generalization on separate data. Failure to overfit the tiny sample often exposes an implementation bug. Ability only to memorize points to data, objective, or generalization.

That turn organizes the field. Dropout changes the forward pass; backprop assigns responsibility; Adam changes the update; ResNet changes the path; AutoML repeats the experiment; federated learning distributes where computation happens. Understanding the cycle prevents a chain of derivatives from being mistaken for thought and makes every claimed advance answer one question: which part changed, and what evidence improved?