Big Data

Big Data refers to datasets so large, so fast-moving, or so varied that conventional storage and analysis tools cannot handle them, together with the techniques built to capture, store, and make use of them. It is not a single technology or a fixed threshold of gigabytes, but an engineering problem and a field of practice: how to draw useful information from data that grows faster than any single machine can process it.

The term spread in the 2000s, as the web, sensors, and mobile devices began producing volumes that traditional relational databases could not comfortably absorb.

The three V's (and the ones that came later)

The most cited framework for describing Big Data is the «three V's», set out by analyst Doug Laney in a META Group research note published in 2001 and titled «3D Data Management». Laney named three axes: volume (how much data), velocity (how fast it is generated and must be processed), and variety (the mix of formats, from tidy tables to text, images, or sensor logs). Later, others in the industry extended the list: IBM added veracity (the reliability and quality of the data) and Oracle added value (the actual usefulness extracted). There is no consensus on how many V's there are, nor on a fixed threshold that separates Big Data from ordinary data: they are best treated as a useful mnemonic, not a mathematical definition.

From Hadoop and Spark to the cloud and the lakehouse

The first generation of tools was built to spread work across many cheap computers. Apache Hadoop, with its HDFS file system and MapReduce programming model, and later Apache Spark, which sped up in-memory processing with its RDDs (resilient distributed datasets), defined the 2010s. Today they belong more to the field's history than to its everyday practice. The current state rests on cloud data warehouses such as Snowflake and Google BigQuery, which separate storage from compute and scale on demand; on the lakehouse pattern, which blends the flexibility of the data lake with the guarantees of the warehouse through open table formats like Delta Lake (Databricks) and Apache Iceberg (born at Netflix); and on stream processing with Apache Kafka —created at LinkedIn— and Apache Flink for data in motion.

Data governance

Hoarding data is no longer treated as a virtue. Governance gathers the practices that keep a body of data trustworthy and usable: quality (that the data is correct and complete), lineage (being able to trace where each figure came from and how it was transformed), cataloguing (knowing what you hold and what it means), and privacy and regulatory compliance, with rules such as Europe's GDPR requiring organisations to know where personal data sits and for what purpose it is used.

Big Data and AI

The link between Big Data and artificial intelligence is real, but it is worth stating soberly. Large machine-learning models are trained on enormous volumes of data, and without that raw material they would not exist in their present form. Yet massive data does not «give» intelligence on its own: without quality data, clear goals, and sound methods, more data only yields more noise. Big Data supplies the infrastructure and the raw material; usefulness depends on what you ask and how you process it.