Artificial Neural Networks

An artificial neural network (ANN) is a computational model built from processing units called neurons, or nodes, connected to one another, that turn inputs into outputs. Each connection carries a weight, and each neuron combines its inputs through a weighted sum and passes them through a non-linear activation function. By chaining many of these operations, the network approximates complex functions from examples.

It helps to clear up a common misconception: ANNs are inspired by biological neurons, but they do not simulate the brain. The analogy is loose and implies no functional equivalence. An artificial neuron reduces its decision to a simple arithmetic operation, whereas a biological neuron is far more complex —so much so that several artificial layers are needed to approximate the behavior of a single real cell. Talking about ANNs means talking about mathematics, not biology.

The neuron: weights, sum and activation

The basic unit takes several inputs, multiplies them by their weights, adds them together with a bias and passes the result through an activation function. Without that non-linearity, stacking neurons would collapse into a single linear operation. The most common activations are the sigmoid, which squeezes the output between 0 and 1, and the ReLU, which lets positive values through and zeroes out negative ones; ReLU became popular because it eases the vanishing-gradient problem that hampers the sigmoid. The first formal neuron model, McCulloch and Pitts (1943), was a binary threshold with fixed weights; Rosenblatt's perceptron (1958) added adjustable weights that the network could learn.

Layers and architecture

Neurons are arranged in layers: an input layer, one or more hidden layers and an output layer. A single-layer perceptron could not solve problems that were not linearly separable, a limit overcome by stacking layers. Specialized families are built on that idea: convolutional networks for images, recurrent networks for sequences and Transformers for language. The depth and the way layers connect define the architecture and shape what the network can learn.

How they are trained: loss, gradient and backpropagation

Training a network means adjusting its weights so that its outputs match the desired ones. A loss function measures the error, and gradient descent reduces it by working out in which direction to nudge each weight so the loss falls. The algorithm that makes this calculation efficient in many-layered networks is backpropagation, popularized by Rumelhart, Hinton and Williams in Nature (1986), which propagates the error backward from the output. Repeated over many examples, this cycle gradually refines the weights.

Limits and uses

ANNs are not magic. They need large volumes of data and heavy computation, and they tend to overfit —memorizing examples instead of generalizing— unless regularization techniques such as dropout are applied. They are also black boxes: explaining why they produce a given output is hard, and interpretability remains an open problem. With those caveats, they underpin very real applications: computer vision, speech recognition, translation, recommendation systems and medical imaging. Understanding what they do —and what they don't— is the best place to start.