Hyperparameters

Hyperparameters are the settings fixed before training a model that control how it learns or what shape it has, unlike the parameters, which the model learns from the data during training. The weights of a neural network or the coefficients of a regression are parameters; the learning rate, the number of layers, a tree's depth, the k in k-nearest neighbors or the regularization strength are hyperparameters.

Parameters versus hyperparameters

The rule is simple: if the training algorithm adjusts it on its own, it is a parameter; if you have to set it beforehand, it is a hyperparameter. Parameters are the model's learned core; hyperparameters are the external instructions that shape that core and are not learned by gradient descent.

How they are tuned

Searching for good hyperparameters is called tuning, and there are three main methods. Grid search exhaustively tries all combinations of a grid, but its cost explodes as the number of hyperparameters grows. Random search samples combinations at random and tends to perform better on the same budget when only a few hyperparameters truly matter. And Bayesian optimization builds a probabilistic model of the relationship between hyperparameters and performance to choose more intelligently what to try next, obtaining good results in fewer evaluations.

Validation and the risk of overfitting

An essential point: hyperparameters are chosen with a validation set, or by cross-validation, never with the test set, which is kept intact for a single, unbiased final estimate. If they are tuned by repeatedly looking at the same set, there is a risk of overfitting to it: knowledge of that set “leaks” into the model and the metrics no longer reflect the ability to generalize. That is why data are split into training, validation and test, and in delicate cases nested cross-validation is used.