Information Theory

Information theory is the mathematical framework that quantifies information: it measures how much uncertainty a source of messages carries and how far those messages can be compressed or transmitted without error. It grew almost entirely out of a single work, «A Mathematical Theory of Communication», which Claude Shannon published in the Bell System Technical Journal in 1948. There Shannon introduced the bit as the unit of information and showed that communication is, at bottom, a statistical problem with exact limits.

Its central idea is counterintuitive: the information in an event depends not on its content but on its probability. A certain event tells us nothing; a rare one tells us a great deal. The less likely an outcome is, the more «surprise» —more bits— we gain by observing it.

Entropy: measuring uncertainty

Entropy is the average information produced by a source, measured in bits. It formalizes the intuition above: it weighs the surprise of each possible outcome —the negative logarithm of its probability— by how often that outcome occurs. A predictable source, like a biased coin that almost always lands heads, has low entropy; a fair coin reaches the maximum, one bit per flip, because every result is equally unpredictable. Entropy therefore sets the theoretical limit on compression: no code can represent a source's messages using fewer bits, on average, than its entropy.

Mutual information and Kullback-Leibler divergence

Two tools extend the framework. Mutual information measures how much one variable reveals about another: how much of the uncertainty about one dissolves once the other is observed. It is zero when the two are independent and grows with their dependence, without requiring the relationship to be linear. The Kullback-Leibler divergence, or relative entropy, instead measures the «distance» between two probability distributions: how many bits are wasted when data that follow one distribution are encoded using the wrong model. It is directed —not symmetric— and never negative, which is why it works as a measure of the gap between what a model predicts and what actually happens.

Channel capacity and Shannon's theorems

Shannon split the problem of communication into two halves. The source coding theorem answers how much a message can be compressed: the limit is its entropy. The channel coding theorem answers how much can be sent over a noisy channel: there exists a number, the channel capacity, below which reliable communication is possible with an arbitrarily small probability of error, and above which it is impossible. This was a startling result: noise does not force us to accept errors, it only caps the speed. The version for bandwidth-limited channels with Gaussian noise is the Shannon-Hartley formula, which links capacity, bandwidth and the signal-to-noise ratio.

Information theory in machine learning

These ideas are now routine in machine learning. Cross-entropy is the standard loss function in classification: minimizing it amounts to pulling the model's predicted distribution toward the true one, that is, to shrinking the KL divergence between them. That same divergence regularizes variational autoencoders, where it pushes the learned latent space toward a simple distribution, and it guides model distillation, in which a small model mimics the probabilistic outputs of a larger one. Mutual information, in turn, helps select the most informative features and learn useful representations without labels. What Shannon devised for the telephone has become the language in which we measure what a machine learns.