Batch Gradient Descent

Batch Gradient Descent is an optimization algorithm that adjusts a model's parameters to minimize a loss function. Its defining trait is a matter of timing: it computes the gradient of that function using the entire training set before every parameter update.

How it works: an exact gradient over the whole set

On each iteration, the algorithm passes over every training example, sums the error each one produces, and computes the average gradient of the loss function. Only then does it take a step, subtracting that gradient —scaled by the learning rate— from the parameters (θ = θ − η·∇J(θ)). As Sebastian Ruder describes in his overview of gradient descent algorithms, this amounts to a single update per full pass through the data. The resulting gradient is exact —the true direction of steepest descent across the whole set— so the path toward the minimum is smooth and almost noise-free.

Strengths and limits

The strength is stability. On convex error surfaces the method is guaranteed to converge to the global minimum, and to a local minimum on non-convex ones, with updates that do not oscillate. The cost is scale. Computing the gradient over the entire set just to take one step is slow, and it demands holding all the data in memory: when the set does not fit, the method becomes intractable, as Ruder himself warns. It also rules out online learning —feeding in new examples on the fly— because each step needs the full batch.

Batch versus stochastic and mini-batch

The contrast with its two relatives frames the choice. Stochastic gradient descent (SGD) updates the parameters from a single example per step: many updates, fast but noisy, that make the objective function fluctuate. Mini-batch uses a small subset at each step, a middle ground that, as Ruder notes, reduces the variance of the updates and yields more stable convergence. So batch offers exact but infrequent updates; SGD and mini-batch, approximate but numerous and scalable ones. On convex problems batch converges cleanly, but in the practice of deep learning, with millions of data points, mini-batch prevails: the textbook Deep Learning by Goodfellow, Bengio and Courville and Dive into Deep Learning agree it is the default choice for training neural networks, thanks to its balance of computational efficiency and stability.