Support Vector Machines

A support vector machine (SVM) is a supervised learning algorithm that separates two classes by finding the maximum-margin hyperplane: the decision boundary that sits as far as possible from the closest examples of each class. That distance, the margin, is the heart of the method; the wider it is, the more robust the classification of new data tends to be.

The method traces back to the «generalized portrait» algorithm that Vladimir Vapnik and Alexey Chervonenkis developed in the Soviet Union around 1963-1964, but the modern SVM took shape in the 1990s. It is used mainly for classification, though it also handles regression.

The maximum-margin hyperplane and support vectors

Among the infinitely many boundaries that could separate two classes, an SVM picks just one: the boundary that maximizes the margin. When that optimization problem is solved, most of the training examples turn out to be irrelevant. The boundary is determined entirely by a handful of points sitting right at the edge of the margin: the support vectors. Move any distant point and the solution stays put; move a support vector and the boundary shifts. This economy —deciding with the borderline cases rather than the bulk of the data— is what gives the method its name and its elegance.

Soft margin and the C parameter

Real data rarely separates without errors. In 1995, Corinna Cortes and Vapnik published «Support-Vector Networks» in the journal Machine Learning, introducing the soft margin: the boundary may tolerate some examples falling on the wrong side or inside the margin, at the cost of a penalty. The trade-off is governed by the regularization parameter C. A high C punishes every error harshly and narrows the margin, risking overfitting; a low C tolerates more errors and widens the margin, favoring generalization. Tuning C is one of the most important practical decisions when training an SVM.

The kernel trick and non-linearity

Many problems cannot be split by a straight line or a flat plane. The solution, proposed by Bernhard Boser, Isabelle Guyon and Vapnik in 1992, is the kernel trick: implicitly project the data into a higher-dimensional space where a separating hyperplane does exist, without ever computing that projection. It suffices to replace the dot products with a kernel function. The most common kernels are the linear, the polynomial and the RBF or Gaussian kernel; the choice of kernel and its parameters sets the shape of the boundary in the original space, which can be sharply curved even though it remains a flat plane in the projected space.

Where they still shine

The same idea can predict continuous values: that is support vector regression (SVR). SVMs no longer dominate as they did in the 2000s: deep learning overtook them on tasks with massive, unstructured data such as images, speech or text at scale. Yet they remain a first-line tool on small and medium datasets and on high-dimensional problems, such as text classification or bioinformatics, where they work well with few examples, rest on solid theory and resist overfitting. It is worth staying honest: they do not always beat simpler models like logistic regression, and their training scales poorly to millions of examples.