Data Integration

Data integration is the process of combining information from heterogeneous sources —relational databases, APIs, files, spreadsheets, sensors or cloud services— into a single, coherent view, as if all of it lived in one place. It is not about copying data from one system to another; it is about reconciling meaning: aligning different schemas, resolving duplicate identities and making sure the numbers add up.

It is worth distinguishing integration from mere ingestion. Ingesting moves data; integrating reconciles its semantics. In his influential paper “Data Integration: A Theoretical Perspective” (2002), Maurizio Lenzerini formalized an integration system as the relationship between a global schema, the source schemas and the mapping that connects them. That tension —between what the user wants to see and how the data is actually stored— sits at the heart of the problem.

The core problems: schemas, entities and quality

The first obstacle is schema heterogeneity: two sources may call the same concept “customer” and “cliente,” or split an address across different fields. Reconciling them is schema matching and schema mapping, problems that Doan, Halevy, and Ives cover alongside mappings and transformations. The second is entity resolution (deduplication or record linkage): deciding whether “J. Smith, London” and “John Smith, UK” represent the same person despite sharing no key. The 2007 survey by Elmagarmid, Ipeirotis, and Verykios explains why errors, incomplete fields, and inconsistent formats defeat exact comparison. The third is data quality and consistency: missing values, incompatible formats, or contradictions between sources that must be resolved before trusting the integrated view. A Google Research review places these problems in the field's development.

Architectures: from ETL to the data lake

The classic approach is ETL (extract, transform, load): data is extracted, cleaned, and transformed, then loaded into a data warehouse. Bill Inmon published Building the Data Warehouse in 1992, and Ralph Kimball systematized dimensional modeling in 1996. ELT reverses the order: load first, transform later. In October 2010, James Dixon described the “data lake” as data in a more natural state, contrasted with a data mart's “bottled water”; the 2011 date previously given here was wrong. As an alternative to materialized stores, virtual or federated integration leaves data at the source and combines it at query time. The NIST reference architecture helps compare these pieces without conflating storage, processing, and integration.

Why it matters for artificial intelligence

Before training a model, scattered signals must be assembled into a coherent dataset: integration is a precondition, not an afterthought. If two records for one entity are treated as different people, that entity may be weighted twice; if two schemas assign different meanings to one field, the feature is no longer comparable. These are risks to test, not an automatic guarantee of bias. A Google Research review connects integration with mappings, queries, and provenance; DAMA-DMBOK places integration and interoperability within a broader management and governance framework. Traceability also requires preserving origin, transformations, and resolution rules: citing the framework does not prove that a particular pipeline follows them.