Loss Function

A loss function measures how far a model's prediction is from the desired value for one example. Training is framed as an optimization problem: adjusting the model's parameters to minimize that loss, usually by gradient descent. Sometimes the loss, referring to a single example or batch, is distinguished from the cost function, which is its average over the whole training set.

Loss, metric and objective are not the same

Three concepts that are often confused are worth separating. The loss function is what is optimized; it guides learning and must therefore be differentiable, so that gradients can be computed. The evaluation metric is what is measured to judge the model—accuracy, F1, AUC—and need not be differentiable. And the objective is the most general term: the function the model optimizes, which may include the loss plus other terms. That loss and metric do not coincide is common and deliberate: cross-entropy, which gives good gradients, is optimized, but accuracy, which is what really matters, is reported.

Common losses

The choice depends on the task. For regression, the mean squared error (MSE) heavily penalizes large errors and is sensitive to outliers, while the mean absolute error (MAE) is more robust to them; the Huber loss is a hybrid of the two. For classification, binary cross-entropy serves two classes and categorical cross-entropy several; the hinge loss is proper to support vector machines.

Conditions

Two conditions matter. The loss must align with the task: a poorly chosen loss optimizes something other than what is sought. And it must be (almost always) differentiable, which is precisely why one cannot train directly with a metric such as accuracy. A regularization term is also usually added to the objective, penalizing the model's complexity to reduce overfitting.