OpenAI buys Rockset to bolster enterprise data search

On June 21, 2024, OpenAI announced its acquisition of Rockset to strengthen retrieval infrastructure. The acquisition release says it will integrate the indexing and querying technology across products and bring the team into OpenAI, but discloses neither a price nor a specific customer feature.

The financial terms of the acquisition were not disclosed. OpenAI said Rockset’s technology would strengthen the information retrieval and indexing infrastructure across its products, and that the company’s team would join OpenAI.

The challenge is not just generating text, but finding the right data

Models like those that power ChatGPT can draft, summarize, and answer questions, but they do not inherently know a company’s internal documents or changes made just minutes ago in a database. To work with that information without retraining the model, companies rely on retrieval systems.

The basic idea is relatively simple. Before answering a question, the application searches authorized sources—manuals, contracts, support records, inventories, or sales data—and provides the model with the relevant excerpts as context. The model then generates an answer based on those materials.

This approach is commonly known as retrieval-augmented generation, or RAG. Its quality depends on two things: whether the system finds the right data and whether it can do so quickly. If it retrieves outdated, incomplete, or irrelevant information, a well-written answer can still be wrong.

That is where Rockset fits in. Its product is designed to ingest continuously changing data, index it, and support queries through SQL, the standard language used to work with databases. Rather than waiting for periodic update processes, a real-time analytics database aims to make newly arrived information available for querying almost immediately.

Infrastructure built for data in motion

Rockset was founded in 2016 by former Facebook engineers and became known as a cloud platform for analyzing data from sources such as Kafka, Amazon S3, and DynamoDB. Its proposition was to eliminate some of the technical work required to prepare and update data before it could be queried.

For a company developing AI assistants, that capability has direct applications. A customer service agent needs to know the current status of an order. A financial assistant must consult the latest version of a report. An internal employee tool has to distinguish between a policy that is currently in force and one that has already been replaced.

Rockset does not turn that data into the model’s permanent knowledge. Its role is closer to that of a very fast library: it organizes and updates the information, then returns the documents or records the application needs at that moment.

The acquisition also shows where a significant part of the competition in enterprise AI lies. The model is just one layer of the product. Underneath it, companies need access permissions, connections to corporate systems, search mechanisms, databases, activity logs, and security controls. A useful answer depends as much on that infrastructure as on the model’s language capabilities.

More control over a critical component of enterprise products

OpenAI launched ChatGPT Enterprise for large organizations in August 2023 and has since expanded its offering for teams and developers. In that market, the ability to connect AI to proprietary data is a practical requirement for tools to move from eye-catching demos to everyday work systems.

Bringing in Rockset’s technology and team could give OpenAI more control over that retrieval layer, rather than leaving it entirely dependent on outside providers. It could also help narrow the gap between an update in a company’s systems and its availability to an AI-based assistant.

The acquisition does not by itself resolve the risks of using AI with corporate information. Questions such as who can access each document, how answers are verified, and what data is sent to each service will remain central. But it improves a basic prerequisite: ensuring that the model receives current, relevant context before answering.

The next step will be seeing how OpenAI integrates the technology into its products. The company has said it plans to strengthen data retrieval across its offering, without announcing a specific user-facing feature for now. For its enterprise customers, the message is clear: the AI race will not be fought solely over model size, but also over the quality of the data those models can access.

Retrieval is a chain, not a button

An enterprise assistant receives a question, checks permissions, searches records, selects passages, constructs context and generates an answer. Each stage has its own failure. A datum may not be synchronised, the index may retain an old version, search may select an irrelevant document or the model may contradict the retrieved material. “Uses RAG” guarantees none of these layers.

The most useful test combines freshness and permissions. Create a known record, measure how long it takes to appear, then change or delete it. At the same time, two accounts with different privileges ask the same question. The system should return the current version to the authorised person, abstain for the restricted account and preserve evidence of the passage supporting the answer.

Real time also needs a definition

A database may ingest changes quickly while an application continues to serve stale data through caches, connectors or intermediate jobs. Measure from the change in the source system to the visible answer, not merely the speed of an isolated query. The important unit is end-to-end delay under representative load.

Provenance must survive drafting. A useful reference leads to the document, version and passage used; a generic source list cannot verify a sentence. If two policies conflict, the assistant should show the conflict or ask for clarification, not silently select the one producing the smoothest answer.

The transferable skill is to audit retrieval through four tests: freshness, authorisation, provenance and abstention. They remain valid when the database or model changes. An eloquent answer becomes useful to a business only when the right datum reached the right person and can be traced.

A test set should contain contradictions

Easy questions with one correct document hide the most dangerous behaviour. Add a current policy and a superseded one, similar names, empty fields and a hostile instruction inside a retrieved file. The system should prioritise version and permissions, treat the document as data and not follow embedded commands. Link every failure to the stage that produced it.

Operational logs should reconstruct the answer without indiscriminately storing sensitive information. Version identifiers, timings, the consulted source, applied permissions and component outcomes are sufficient when access and retention are defined. That traceability supports correction and audit; a permanent dump of every conversation would create a new risk.

The result sheet should separate retrieval from generation. For each question, record the expected document, retrieved document, permissions, cited passage and answer fidelity. It then becomes possible to fix the index, prompt, model or access control instead of calling every failure a hallucination.

Every error should retain the name of the stage that failed.