Data Mining

Data mining is the process of discovering useful, non-obvious patterns, relationships and knowledge in large volumes of data, combining methods from statistics, machine learning and database systems. The name is somewhat misleading: it does not “extract data,” but knowledge from it.

A piece of the KDD process

Data mining is usually placed as the analysis step within a broader process, knowledge discovery in databases (KDD). Usama Fayyad, Gregory Piatetsky-Shapiro and Padhraic Smyth defined it in 1996 as “the non-trivial process of identifying valid, novel, potentially useful and understandable patterns in data.” In that scheme, mining is the moment when the algorithms are applied; before it come selection and cleaning, and after it, interpretation.

Its tasks

Data mining groups together several tasks. Classification assigns each case to a category; regression predicts a numerical value; clustering discovers natural groups without prior labels; association rules find what things go together—the classic example is “market basket analysis”—; and anomaly detection locates cases that deviate from the norm, useful for spotting fraud or failures.

How a project is organized: CRISP-DM

The reference methodology for structuring a project is CRISP-DM (Cross-Industry Standard Process for Data Mining), published around 1999-2000. It divides the work into six phases—business understanding, data understanding, preparation, modeling, evaluation and deployment—and is cyclical: it is not traveled in a straight line, but by returning to earlier steps.

Neighbors and limits

It is worth not confusing terms that overlap. Data mining focuses on discovering patterns; machine learning, on predicting; data science is a broader field that encompasses both; and big data refers to the size of the data, not a technique. Finally, because it works with often personal data, data mining raises serious privacy and ethics demands, with legal frameworks such as the European GDPR that regulate its use.