OpenAI brings real-time voice to its developer API

On October 1, 2024, OpenAI introduced the Realtime API in public beta for building low-latency speech-to-speech conversations. The original announcement describes modalities, prices and limits on that date; a natural interface must be evaluated as a chain of audio, model, tools and human escalation, not by how convincing it sounds.

The company also announced automatic prompt caching, a distillation system for moving tasks from larger models to smaller, more affordable ones, and image fine-tuning for GPT-4o. Together, the updates target a very specific problem: making AI-powered applications faster, cheaper and more specialized.

Voice conversations without chaining services

Until now, building a voice assistant with language models typically meant stitching together three components: speech recognition to transcribe the user, a text model to formulate a response and a synthesizer to turn it back into audio. That process adds delay and can lose elements of the conversation, such as a pause, an interruption or intonation.

The Realtime API works directly with incoming and outgoing audio through GPT-4o, OpenAI’s multimodal model. In other words, the system processes speech as speech instead of requiring every exchange to go through a complete text transcription first.

The API is available in public beta and supports connections through WebRTC — a common protocol for live communications in browsers and on mobile devices — and WebSocket. The first option is aimed at client-side applications, while the second is better suited to server connections.

OpenAI has set audio pricing at $100 per million input tokens and $200 per million output tokens. Tokens are the units into which the model breaks down the information it receives and generates; in audio, they do not correspond directly to words. Text can also be sent to the model for $5 per million input tokens and returned for $20 per million output tokens. Source

The promise is especially relevant for customer service, conversational tutors, accessibility tools and internal assistants. But natural-sounding voice does not solve the hard parts of these products on its own: integrating reliable data, deciding when a conversation should be handed off to a person and preventing the assistant from making up information in sensitive contexts.

Caching cuts the cost of repeating instructions

The second announcement addresses a common inefficiency in products built with language models. Many requests repeat the same long instruction: the assistant’s rules, company documentation, style examples or a conversation’s history.

With prompt caching, OpenAI temporarily stores repeated prefixes in those messages and automatically applies a 50% discount to reusable input tokens. The company says the mechanism can reduce latency by up to 80% for requests containing cached content. Source

The feature activates for prompts of at least 1,024 tokens and is available on the GPT-4o, GPT-4o mini, o1-preview and o1-mini models. Developers do not need to change the format of their API calls. Source

This is no small discount for those building assistants on extensive documents or complex instructions. In these applications, costs depend on more than the model’s response: resending the same context on every turn can account for a significant share of the bill. Caching also encourages a more orderly prompt design, with stable instructions at the beginning and variable information at the end.

Distilling models to avoid always paying for the most powerful one

OpenAI has also introduced a model distillation tool. The idea is to use a larger, more capable model to generate reference answers, review those results and then use them to fine-tune a smaller model for a specific task.

The company integrates that workflow into its platform: developers can store outputs from models such as GPT-4o and use them as training data to fine-tune GPT-4o mini. The goal is not for the smaller model to replace the larger one across the board, but to handle an application’s repetitive, well-defined tasks at a lower cost.

It is a familiar strategy in research and industry, but adding it to OpenAI’s dashboard reduces the work involved in building a custom collection, evaluation and training pipeline. Even so, the method’s usefulness depends on the quality of the examples: a small model trained on inaccurate or biased answers will reproduce those flaws more consistently and at a lower price.

GPT-4o can now be fine-tuned with images

The final piece is vision fine-tuning for GPT-4o, also in beta. Fine-tuning lets developers adapt a general-purpose model with their own examples so it performs better on a particular task, format or domain. In this case, those examples can include images as well as text.

For example, a company could use it to teach the model to classify catalog images using its internal categories, extract fields from visual documents or follow a specific inspection standard. That does not turn the model into an accuracy guarantee: its performance still needs to be measured on cases separate from those used in training, especially when a classification has operational consequences.

The updates show where OpenAI’s immediate competition lies. The model still matters, but developers are placing increasing weight on speed, price, ease of integration and the ability to adapt a system to a real-world use case. With the Realtime API, the company wants GPT-4o to be more than a chat engine: it aims to make it infrastructure for voice interfaces and specialized applications.

A real-time voice is an operational chain

Testing preserves input audio, transcript when available, response, tool call and final audio. Add noise, interruptions, accents, proper names and silence; measure end-to-end latency and whether the user can correct an action before execution. Naturalness adds no accuracy.

The separate Prompt Caching announcement explains that feature’s discounts and conditions. To verify savings, record cached tokens, cache hits, time and cost per session. Reordering instructions may improve caching while changing behaviour; test both effects.

Distillation and fine-tuning need an isolated test set. If the large model produces flawed examples, the small one learns to repeat them. Review a sample, retain difficult cases and compare against the untuned version. The transferable skill is to measure voice, caching and adaptation as separate components and calculate cost per resolved conversation, not per fluent demonstration.

A tool error changes the risk

An incorrect spoken answer can be corrected; a call that sends a message, buys something or changes a record has already altered the world. Before execution, the interface should display action, arguments and destination, request confirmation where harm is hard to reverse, and retain an understandable log.

Testing adds interruptions just before confirmation, similar names and changes of intent. Observe whether the system cancels the earlier instruction or blends contexts. Measuring only time to first audio rewards fluent experience while hiding the latency needed to act safely.

The protocol ends with failure recovery: disconnection, incomplete audio and unavailable tools. The system should report state and avoid repeating an action after reconnection. A fast experience that duplicates operations is not useful real time.