Data Mining

Data mining is the process of discovering useful, non-obvious patterns, relationships and knowledge in large volumes of data, combining methods from statistics, machine learning and database systems. The name itself is somewhat misleading: it is not the data that are extracted, but useful knowledge discovered from them.

A piece of the KDD process

Data mining is usually placed as the analysis step within a broader process, knowledge discovery in databases (KDD). Usama Fayyad, Gregory Piatetsky-Shapiro and Padhraic Smyth defined it in 1996 as “the non-trivial process of identifying valid, novel, potentially useful and understandable patterns in data.” In that scheme, mining is the moment when the algorithms are applied; before it come selection and cleaning, and after it, interpretation. Primary source.

Its tasks

Data mining groups together several tasks. Classification assigns each case to a category; regression predicts a numerical value; clustering discovers natural groups without prior labels; association rules find what things go together—the classic example is “market basket analysis”—; and anomaly detection locates cases that deviate from the norm, useful for spotting fraud or failures. Primary source.

How a project is organized: CRISP-DM

The reference methodology for structuring a project is CRISP-DM (Cross-Industry Standard Process for Data Mining). It divides the work into six phases—business understanding, data understanding, preparation, modeling, evaluation and deployment—and is cyclical: it is not traveled in a straight line, but by returning to earlier steps. Primary source.

Neighbors and limits

It is worth not confusing terms that overlap. Data mining focuses on discovering patterns; machine learning, on predicting; data science is a broader field that encompasses both; and big data refers to the size of the data, not a technique. Finally, because it works with often personal data, data mining raises serious privacy and ethics demands, with legal frameworks such as the European GDPR that regulate its use. Primary source.