OpenAI brings real-time voice to its developer API

OpenAI used its DevDay, held this Tuesday in London, to strengthen the less visible layer of its business: the tools that help other companies turn its models into products. The headline announcement is the Realtime API, an interface for building voice applications that can listen and respond with low latency.

The company also announced automatic prompt caching, a distillation system for moving tasks from larger models to smaller, more affordable ones, and image fine-tuning for GPT-4o. Together, the updates target a very specific problem: making AI-powered applications faster, cheaper and more specialized.

Voice conversations without chaining services

Until now, building a voice assistant with language models typically meant stitching together three components: speech recognition to transcribe the user, a text model to formulate a response and a synthesizer to turn it back into audio. That process adds delay and can lose elements of the conversation, such as a pause, an interruption or intonation.

The Realtime API works directly with incoming and outgoing audio through GPT-4o, OpenAI’s multimodal model. In other words, the system processes speech as speech instead of requiring every exchange to go through a complete text transcription first.

The API is available in public beta and supports connections through WebRTC — a common protocol for live communications in browsers and on mobile devices — and WebSocket. The first option is aimed at client-side applications, while the second is better suited to server connections.

OpenAI has set audio pricing at $100 per million input tokens and $200 per million output tokens. Tokens are the units into which the model breaks down the information it receives and generates; in audio, they do not correspond directly to words. Text can also be sent to the model for $5 per million input tokens and returned for $20 per million output tokens.

The promise is especially relevant for customer service, conversational tutors, accessibility tools and internal assistants. But natural-sounding voice does not solve the hard parts of these products on its own: integrating reliable data, deciding when a conversation should be handed off to a person and preventing the assistant from making up information in sensitive contexts.

Caching cuts the cost of repeating instructions

The second announcement addresses a common inefficiency in products built with language models. Many requests repeat the same long instruction: the assistant’s rules, company documentation, style examples or a conversation’s history.

With prompt caching, OpenAI temporarily stores repeated prefixes in those messages and automatically applies a 50% discount to reusable input tokens. The company says the mechanism can reduce latency by up to 80% for requests containing cached content.

The feature activates for prompts of at least 1,024 tokens and is available on the GPT-4o, GPT-4o mini, o1-preview and o1-mini models. Developers do not need to change the format of their API calls.

This is no small discount for those building assistants on extensive documents or complex instructions. In these applications, costs depend on more than the model’s response: resending the same context on every turn can account for a significant share of the bill. Caching also encourages a more orderly prompt design, with stable instructions at the beginning and variable information at the end.

Distilling models to avoid always paying for the most powerful one

OpenAI has also introduced a model distillation tool. The idea is to use a larger, more capable model to generate reference answers, review those results and then use them to fine-tune a smaller model for a specific task.

The company integrates that workflow into its platform: developers can store outputs from models such as GPT-4o and use them as training data to fine-tune GPT-4o mini. The goal is not for the smaller model to replace the larger one across the board, but to handle an application’s repetitive, well-defined tasks at a lower cost.

It is a familiar strategy in research and industry, but adding it to OpenAI’s dashboard reduces the work involved in building a custom collection, evaluation and training pipeline. Even so, the method’s usefulness depends on the quality of the examples: a small model trained on inaccurate or biased answers will reproduce those flaws more consistently and at a lower price.

GPT-4o can now be fine-tuned with images

The final piece is vision fine-tuning for GPT-4o, also in beta. Fine-tuning lets developers adapt a general-purpose model with their own examples so it performs better on a particular task, format or domain. In this case, those examples can include images as well as text.

For example, a company could use it to teach the model to classify catalog images using its internal categories, extract fields from visual documents or follow a specific inspection standard. That does not turn the model into an accuracy guarantee: its performance still needs to be measured on cases separate from those used in training, especially when a classification has operational consequences.

The updates show where OpenAI’s immediate competition lies. The model still matters, but developers are placing increasing weight on speed, price, ease of integration and the ability to adapt a system to a real-world use case. With the Realtime API, the company wants GPT-4o to be more than a chat engine: it aims to make it infrastructure for voice interfaces and specialized applications.