Regression

In machine learning, regression is the supervised task of predicting a continuous value —a number— from one or more input variables. That sets it apart from classification, its sibling task, which predicts a category instead of a number: whether an email is spam, whether a tumor is benign or malignant. Forecasting a house price, tomorrow's temperature or a region's electricity demand are regression problems; labelling an image is classification.

The name has a curious history. Francis Galton coined it in his 1886 paper «Regression towards mediocrity in hereditary stature», after noticing that the children of very tall parents tended to be tall but less so: their heights «regressed» toward the population average. From that phenomenon —regression to the mean— the term passed to a whole family of methods.

Linear regression as the base case

The founding model is linear regression: fitting a straight line (or, with several inputs, a hyperplane) that runs through the cloud of points as well as possible. «As well as possible» is defined by minimizing the squared error —the sum of the squared vertical distances between each data point and the line. This criterion, the method of least squares, was published by Adrien-Marie Legendre in 1805; Carl Friedrich Gauss claimed to have used it since 1795 and was the one who tied it to the normal distribution, giving it a probabilistic footing. Its appeal is a closed-form solution and readable results: each coefficient tells you how much the output changes when an input rises by one unit.

How a regression model is measured

Because the output is a number, being right is not «yes or no» but how far the prediction lands from the true value. The usual metrics are the mean squared error (MSE), which averages the squared errors and punishes large misses heavily; its root, the RMSE, which returns the error to the variable's original units; the mean absolute error (MAE), which averages the errors in absolute value and is more robust to outliers; and the coefficient of determination (R²), which states what share of the output's variability the model explains, with 1 being a perfect fit.

Variants and the logistic-regression caveat

The family is broad: polynomial regression bends the line to fit non-linear relationships, and models such as decision trees, random forests or neural networks also tackle regression tasks. One common confusion is worth clearing up: logistic regression, despite its name, does not predict a free number but the probability of belonging to a class, and applying a threshold turns it into a category. In the machine-learning taxonomy it is therefore a classification method; it keeps the word «regression» because, internally, it fits a continuous function.

Overfitting and regularization

Like any model, regression can overfit: memorizing the noise in the training data rather than the signal, and failing on new data. This is the visible face of the bias-variance tradeoff. To hold it in check we use regularization, which penalizes large coefficients: Ridge regression (Hoerl and Kennard, 1970) shrinks them without zeroing them out, whereas Lasso (Tibshirani, 1996) can drive them exactly to zero and thus select variables. Both trade a little bias for less variance —the core idea gathered in texts such as «The Elements of Statistical Learning».