Anomaly Detection

Anomaly detection —also known as outlier detection— is the task of identifying observations that deviate significantly from the expected pattern, that is, from what a model treats as «normal». The field's landmark reference, the survey by Chandola, Banerjee and Kumar (2009), defines it as the problem of finding patterns in data that do not conform to expected behaviour, and distinguishes three kinds: point anomalies (a single, unusually rare observation), contextual anomalies (odd only in a given context, such as a high temperature in winter) and collective anomalies (a group of points that is anomalous as a whole even when each point looks ordinary on its own).

It helps to view anomalies through density estimation: what is anomalous is simply what is improbable. A model that learns how normal data are distributed can flag anything that falls in low-density regions. This idea ties anomaly detection to unsupervised learning and runs through nearly every method described below.

Approaches by label availability

Methods are grouped above all by how many labelled examples are available. The supervised approach relies on examples tagged as normal and anomalous and trains a classifier; it is the least common case, because anomalies are rare and labelling them is expensive. The semi-supervised approach learns a model from data known to be normal alone and treats as anomalous anything that departs from that profile; it fits well when «normal» is plentiful and anomalies have barely been seen. The unsupervised approach, by far the most common in practice, uses no labels: it assumes anomalies are few and different from the bulk of the data and surfaces them by their statistical rarity. This three-way split comes from Chandola and colleagues' survey and organises much of the literature.

Main methods

Statistical methods are the oldest: they model the distribution of the data and set a threshold, so that whatever lands in the tails —say, beyond a certain number of standard deviations— is flagged as anomalous. Distance- and density-based methods start from each point's neighbourhood: k-nearest neighbours and, above all, the Local Outlier Factor (LOF, from Breunig, Kriegel, Ng and Sander, 2000) compare a point's local density with that of its neighbours and single out those living in far sparser regions.

The Isolation Forest (Liu, Ting and Zhou, 2008) flips the angle: instead of modelling what is normal, it isolates what is anomalous. It builds random partitioning trees and observes that anomalies, being «few and different», are isolated with very few splits —that is, at a shorter average depth. The One-Class SVM (Schölkopf and colleagues, 2001) draws a boundary that wraps the normal data in a feature space and treats whatever falls outside as anomalous. Finally, autoencoders —neural networks that learn to compress and reconstruct normal data— flag as anomalous whatever they reconstruct poorly: a high reconstruction error betrays a pattern the network never learned.

Uses and challenges

Anomaly detection underpins very tangible applications: fraud detection in cards and payments, cybersecurity and network intrusion detection, industrial and systems monitoring (to anticipate breakdowns or outages) and clinical uses in healthcare. In all of them, the interesting thing is precisely the infrequent.

That rarity is also its greatest difficulty. Class imbalance is extreme —anomalies may be one in a million—, which complicates both training and evaluation. Defining what is «normal» is no simpler when it changes over time: data drift means that behaviour that is legitimate today would have looked suspicious yesterday, and it forces models to be readjusted. And because the boundary between normal and anomalous is rarely sharp, the false-positive rate tends to be high: too many alarms erode the trust of whoever reviews them. There is no universally best method; the choice depends on the type of data, how many labels are available and the relative cost of erring by excess or by omission.