Statistical Inference

Statistical inference is the body of methods used to draw conclusions about the properties of a population from a finite sample of data. Because the whole population is rarely observable, inference reasons in reverse, from the observed toward the unknown, always quantifying the uncertainty that comes with that leap.

Its two central tasks are the estimation of unknown parameters and hypothesis testing. Both rest on probability theory and on a statistical model that describes how the data might have been generated.

Point and interval estimation

Point estimation summarizes a population parameter, a mean or a proportion, as a single number computed from the sample. Since that number almost never equals the true value, interval estimation offers a range of plausible values: the confidence interval. A 95% interval describes a procedure that, repeated across many samples, would capture the true value 95% of the time; it does not mean there is a 95% probability that the parameter lies in that particular interval. A common error is worth flagging: a model's performance metrics (accuracy, precision, recall, F1 or mean squared error) do not estimate the population or any parameter of the data-generating process; they measure the model's predictive performance on a test set and are themselves sample estimates carrying their own uncertainty.

Hypothesis testing and p-values

A test pits a null hypothesis, often the absence of an effect, against an alternative and gauges how compatible the data are with the null. Its most common tool is the p-value. According to the American Statistical Association statement signed by Ronald Wasserstein and Nicole Lazar in 2016, a p-value is «the probability under a specified statistical model that a statistical summary of the data would be equal to or more extreme than its observed value». The same statement spells out what a p-value is NOT: it does not measure the probability that the studied hypothesis is true, nor the probability that the data arose by chance alone, and a threshold such as 0.05 should not decide a scientific conclusion on its own.

Frequentist versus Bayesian

Two frameworks coexist. The frequentist approach treats parameters as fixed but unknown quantities and grounds inference in the long-run frequency of the data. The Bayesian approach treats parameters as random variables with a distribution: it starts from a prior belief (the prior distribution) and, through Bayes' theorem, updates it with the data to obtain the posterior distribution, proportional to the product of the likelihood and the prior. It is especially useful when data are scarce or valuable prior knowledge exists.

Inference and machine learning

Learning from data is, in essence, inference. Fitting a model amounts to estimating its parameters, and the goal is not to memorize the sample but to generalize to new data. That challenge is captured by the bias-variance trade-off described by Hastie, Tibshirani and Friedman in The Elements of Statistical Learning: too simple a model biases predictions, while too flexible a model overfits the noise in the sample. Another frequent confusion is worth undoing: convolution is not an inference method but a mathematical operation inside convolutional neural networks; inference enters when their parameters are estimated. Deep problems remain open, such as causal inference, telling correlation from cause, and the reliable quantification of uncertainty in large-scale models.