OpenAI unveils GPT-4, a model that understands images

OpenAI today unveiled GPT-4, the next generation of the model that powered ChatGPT. The system can accept text and images as input, generate text responses and significantly outperform GPT-3.5 on reasoning, writing and academic exams.

The launch matters for two reasons. First, it turns conversations with an AI into a more useful tool for tasks that require interpreting documents, charts or photographs. Second, it raises the bar for language models once again, just four months after ChatGPT brought the technology to the mainstream.

From chatbot to a system that can see

GPT-4 is a multimodal model: it can process more than one type of information. In its presentation, OpenAI showed how the system interprets an image and answers questions about it. For example, it can explain what a photograph contains, extract information from a chart or suggest which ingredients are missing based on an image.

It does not generate images: its output is still text. The novelty is that it can combine what it sees with a written instruction to produce a response.

This capability will not be immediately available to everyone. OpenAI is initially testing it with Be My Eyes, an app that connects blind and low-vision people with volunteers who describe their surroundings. The integration points to one of the clearest uses for conversational computer vision: describing a scene, reading labels or helping people understand an image without relying on someone else being available.

Better scores, with important caveats

OpenAI evaluated GPT-4 using standardized and professional exams. In a simulation of the U.S. bar exam, the model ranked in approximately the top 10% of participants, compared with GPT-3.5’s bottom 10%. It also reached high percentiles on the U.S. SAT and GRE.

These figures show a real improvement in the ability to follow instructions, handle long texts and solve problems with a familiar structure. But they do not mean the system has general human understanding or can practice a profession autonomously.

An exam measures responses within a limited format and uses relatively well-defined grading criteria. Real-world work involves incomplete information, legal consequences and the need to verify sources. GPT-4 can write a convincing explanation and still include a false detail or fabricate a reference.

OpenAI acknowledges that limitation. The company says GPT-4 is 40% more likely than GPT-3.5 to produce factually accurate responses in its internal adversarial truthfulness evaluations, but it still hallucinates: it presents incorrect claims with confidence. That is a meaningful improvement, not a guarantee of reliability.

More context for conversations and documents

The initial version of GPT-4 can handle up to 8,192 context tokens, a measure that includes the instruction as well as the text it receives and generates. OpenAI also offers a 32,768-token variant for cases involving much longer documents.

In practical terms, that increase makes it possible to analyze contracts, reports, code or long conversations without splitting them into as many fragments. For businesses, it opens the door to assistants that work with their own documentation; for users, it improves tasks such as summarizing study materials, reviewing drafts or preparing questions about a complex text.

Its usefulness will depend, however, on how the data entered into these services is protected. A model can process internal documents, but that makes it essential to review terms of use, access controls and the handling of confidential information before incorporating it into a workflow.

Available in ChatGPT Plus as the AI assistant race heats up

GPT-4 is available starting today to people who pay for ChatGPT Plus, OpenAI’s $20-a-month subscription, although usage limits apply. The company has also opened a waitlist for its API, the route that allows developers to integrate the model into their own applications.

Microsoft has also confirmed that the latest version of its Bing search engine already uses GPT-4. The news explains why Bing had displayed more advanced conversational and summarization capabilities than earlier models would have suggested, although it had also produced episodes of erratic responses during its first few public weeks.

OpenAI has not disclosed GPT-4’s parameter count or details about its architecture, training data or the computational cost of developing it. Parameters are internal values that the model adjusts during learning, but their number alone does not determine quality: the data and training matter just as much, as do the later mechanisms used to align responses with human instructions.

The presentation solidifies a new phase for AI assistants. Until now, the general public had discovered chatbots that could write. GPT-4 adds a layer of visual analysis and a greater ability to work with complex instructions. The immediate challenge will be determining whether that improvement holds up beyond exams and demonstrations, where a plausible answer is not enough and getting it right is essential.