OpenAI unveils GPT-4o with real-time voice, vision and text

OpenAI unveiled GPT-4o on Monday, a new model that understands and generates text, images and audio. The “o” stands for omni: the company wants ChatGPT to stop feeling like a collection of separate tools and respond to voice, vision and language with the immediacy of a conversation.

The announcement matters for two reasons. First, OpenAI is bringing many capabilities previously associated with GPT-4 to the free version of ChatGPT. Second, it points to a way of interacting with AI assistants that does not depend on typing a question, waiting for an answer and starting over: the model can listen, look and respond throughout an exchange.

A model that processes audio end to end

Until now, voice conversations in ChatGPT have combined three systems: one transcribed audio into text, GPT processed that text, and another model converted the response back into speech. That pipeline works, but it loses important elements of conversation, such as tone, pauses, emotion and interruptions.

GPT-4o processes audio directly and can return audio as its response. OpenAI says it takes an average of 320 milliseconds to respond to a voice input, with a minimum of 232 milliseconds. That is close to the human response time in an everyday conversation.

In the launch demonstration, the model held a spoken conversation, detected emotion in a voice, helped solve an equation written in front of a camera, and acted as an interpreter between English and Italian. It could also analyze an image or a shared screen and comment on what it saw.

The demonstration should be distinguished from immediate availability. The new advanced audio features will first roll out over the coming weeks to a small group of ChatGPT Plus subscribers. Most users will initially continue to encounter the current voice experience.

GPT-4 for people using ChatGPT for free

The free version of ChatGPT will begin receiving GPT-4o with usage limits. Once that limit is reached, the service will switch to GPT-3.5. Plus users will have a message limit up to five times higher than that of free accounts.

OpenAI is also bringing several previously more restricted features to the free tier: web browsing, data analysis, file and image uploads, visual understanding and the use of custom GPTs. Memory, which allows ChatGPT to retain certain user preferences between conversations, will also roll out gradually.

This does not mean the free product will become identical to Plus. The subscription retains higher limits, priority access during periods of high demand and early access to some features. But the functional gap between the two versions is narrowing significantly.

Faster and cheaper for developers

GPT-4o will also be available through the API, the interface companies and developers use to integrate OpenAI models into their products. For text and image tasks, the company says it matches GPT-4 Turbo’s performance while running twice as fast and costing half as much.

Its announced price is $5 per million input tokens and $15 per million output tokens. A token is a small unit of text the model uses to read and generate language; it is not exactly equivalent to a word, but it provides a way to measure the volume processed.

The model supports up to 128,000 tokens of context, a useful capacity for working with lengthy documents, long conversations or large amounts of code. For now, the API supports text and images; audio capabilities will be added later for a limited group of partners.

A desktop app for the Mac

Alongside the model, OpenAI announced a desktop app for macOS. The company is rolling it out today to Plus users and plans to extend it to other users over the coming weeks. The app lets users open ChatGPT with a keyboard shortcut and work alongside other windows, without relying on a browser tab.

OpenAI also promises more direct integration with the content being viewed on screen. That idea captures GPT-4o’s ambition: an assistant that does not merely respond to text pasted into a dialog box, but can take in the visual and audio context of the user’s work.

The challenge will not be purely technical. The more present an assistant becomes in cameras, microphones, files and screens, the more important privacy controls, clear permissions and safeguards against abuse will be. OpenAI has limited output voices to predefined options and trained the model to reject certain audio content, but expanding these capabilities will require testing whether those safeguards work outside a demonstration.

For now, GPT-4o brings ChatGPT closer to the interface science fiction has promised for years: talking to a computer that listens, sees and answers. The difference is that this time it is reaching hundreds of millions of potential users first, including those who do not pay for a subscription.