Unsupervised learning: methods and techniques

Unsupervised learning covers the techniques that look for structure in data nobody has labelled: segmenting customers without knowing in advance how many types exist, spotting odd transactions with no list of known frauds, squeezing a thousand variables down to two so they can be plotted.

And it carries a difficulty with no technical fix, only discipline: there is no answer key here. In supervised learning you can be wrong and find out, because a correct label exists to measure against. In unsupervised learning you can be wrong indefinitely, and with beautiful charts. Almost every "insight" that emerges from a data segmentation is born this way, and very few are ever validated.

The families, one sentence each

Clustering divides data into groups of items similar to one another. Dimensionality reduction compresses many variables into a few while keeping what matters, whether for plotting or for feeding another model. Anomaly detection hunts for whatever departs from the dominant pattern. And density estimation models how the data spread through their space, which is the formal basis of nearly all the above.

Four distinct problems, sharing one Achilles heel: all of them return a result every time. There is never a message saying "there is nothing here."

k-means and the trap of the round number

The most widely used clustering algorithm is k-means, and its first characteristic says a great deal: you must tell it in advance how many groups you want. Ask for five, get five. Ask for twelve, get twelve. At no point does it reply that the data do not cluster.

The scikit-learn documentation, the reference implementation, warns precisely: the measure k-means minimises "makes the assumption that clusters are convex and isotropic, which is not always the case" and "responds poorly to elongated clusters, or manifolds with irregular shapes." Translated: the method looks for blobs, and if your data arrange themselves in filaments or crescents, it will hand you blobs anyway, cutting where there is no cut.

The same page flags a second problem that surprises many people: "in very high-dimensional spaces, Euclidean distances tend to become inflated (this is an instance of the so-called curse of dimensionality)." When everything is far from everything, the very notion of "similar" degrades, and grouping by distance stops meaning what you think. Hence the advice to reduce dimensions before clustering.

The chart that convinces everybody

Here is the costliest trap, because it is visual and deeply persuasive. The techniques that project high-dimensional data onto a plane — t-SNE, UMAP — produce those point clouds with cleanly separated islands that turn up in presentations everywhere, and that almost nobody knows how to read.

In October 2016, Martin Wattenberg, Fernanda Viégas and Ian Johnson published on Distill an interactive article on how these charts get misread. Their warnings are concrete and worth keeping to hand:

First: "You cannot see relative sizes of clusters in a t-SNE plot," because the algorithm "naturally expands dense clusters, and contracts sparse ones, evening out cluster sizes." That blob apparently dominating the chart may contain four points.

Second: "Distances between well-separated clusters in a t-SNE plot may mean nothing." Two islands being far apart does not imply they are very different.

And the third, which ought to be taught before the rest: "Random noise doesn't always look random." The authors demonstrate it with genuinely random data that, at certain settings of the perplexity parameter, "seem to show dramatic clusters." Structure that is crisp, compelling and entirely false.

How not to fool yourself: four tests

This is what you take away, and it applies to any finding out of a label-free analysis, whether produced by a model or a consultancy.

One, the noise test. Run the exact same procedure on random data of the same shape — same number of rows, columns and ranges — and look at the result. If pretty groups appear there too, the structure comes from the method, not your data. It is the cheapest test in existence and almost nobody runs it.

Two, the stability test. Repeat the analysis on a random half of the data, several times over. If the groups change composition on each repetition, they are not groups: they are arbitrary cuts through a continuous cloud.

Three, the second-method test. If k-means and a density-based algorithm, which work on different principles, roughly agree, there is something there. If each tells a different story, the story belongs to the algorithm.

Four, the external test. Do the groups predict anything that was not used to build them? If "segment 3" also buys differently, churns sooner or responds to a different campaign, then it exists. If it is distinguishable only by the variables used to manufacture it, it is a tautology with a nice name.

Anomalies and their awkward arithmetic

Anomaly detection deserves its own section because it fails in a very specific way that has nothing to do with the algorithm and everything to do with the numbers.

Suppose a genuinely good fraud detector: right 99 % of the time, both when flagging fraud and when clearing a transaction. And suppose fraud affects one operation in a thousand. Across a million transactions there will be 1,000 fraudulent ones, of which the system catches 990. But of the 999,000 legitimate ones it wrongly flags 1 %: nearly 10,000 false alarms.

Which means: of every eleven alerts reaching the analyst, ten are noise. With a 99 % detector. And no threshold tweak fixes that without letting fraud through, because the problem is not the model: it is that the thing being hunted is extremely rare.

That arithmetic — the same as screening for a rare disease — accounts for most detection projects abandoned after six months "because the system throws too many false positives." It was not throwing too many: it was throwing the number the base rate dictates. The question to ask before starting is not what accuracy the model achieves, but how many alerts a human team can genuinely process per day, and what each one costs.

And when there are no labels to evaluate against

The scikit-learn documentation puts it plainly: "If the ground truth labels are not known, evaluation must be performed using the model itself," and it offers the Silhouette Coefficient, where a higher score indicates better-defined clusters.

It is a useful tool and worth using, but notice what that sentence says: the model is scoring itself against its own idea of what a good cluster is. Silhouette rewards compact, well-separated groups, so it will favour the methods that look for exactly that. It is not independent verification. It is a second opinion from the same doctor.

Where to go next, with no middlemen

The scikit-learn clustering documentation is free, well written, and contains a comparison table of algorithms with their assumptions and use cases: half an hour there saves weeks. The Distill article on t-SNE is interactive and is one of the few reads that genuinely changes how you look at a chart.

The capability you take away is a reflex: whenever someone shows you groups found in data, ask what came out when the same method was applied to noise — because if nobody has tried, nobody knows whether the finding belongs to the world or to the algorithm.