Data Integration

Data integration is the process of combining information from heterogeneous sources —relational databases, APIs, files, spreadsheets, sensors or cloud services— into a single, coherent view, as if all of it lived in one place. It is not about copying data from one system to another; it is about reconciling meaning: aligning different schemas, resolving duplicate identities and making sure the numbers add up.

It is worth distinguishing integration from mere ingestion. Ingesting moves data; integrating reconciles its semantics. In his influential paper «Data Integration: A Theoretical Perspective» (2002), Maurizio Lenzerini formalized an integration system as the relationship between a global schema, the source schemas and the mapping that connects them. That tension —between what the user wants to see and how the data is actually stored— sits at the heart of the problem.

The core problems: schemas, entities and quality

The first obstacle is schema heterogeneity: two sources may call the same concept «customer» and «cliente», or split an address across different fields. Reconciling them is schema matching and schema mapping, central themes in «Principles of Data Integration» by Doan, Halevy and Ives. The second is entity resolution (also called deduplication or record linkage): deciding that «J. Smith, London» and «John Smith, UK» are the same person even though they share no key. Elmagarmid, Ipeirotis and Verykios surveyed these techniques in «Duplicate Record Detection: A Survey» (2007). The third is data quality and consistency: missing values, incompatible formats or contradictions between sources that must be reconciled before the result can be trusted.

Architectures: from ETL to the data lake

The classic approach is ETL (extract, transform, load): data is extracted, cleaned and transformed, then loaded into a data warehouse, whose best practices were popularized by Bill Inmon and Ralph Kimball in the 1990s. As storage grew cheaper, ELT emerged —loading first and transforming later— along with the data lake, a term coined by James Dixon in 2011, which keeps data in its native format. As an alternative to these materialized stores, virtual or federated integration leaves data at the source and combines it on the fly at query time.

Why it matters for artificial intelligence

A machine learning model is only as good as the data that feeds it. Before training, scattered signals must be assembled into a coherent dataset: integration is a precondition, not an afterthought. Poor entity resolution duplicates examples and biases the model; misaligned schemas inject silent noise. This is why data integration connects directly to feature engineering and data governance —frameworks such as DAMA-DMBOK stress the point— disciplines that ensure what reaches the model is reliable, traceable and semantically consistent.