Anthropic launches Claude 3: Opus beats GPT-4 in several tests

Anthropic unveiled Claude 3 on Monday, a new family of language models comprising Opus, Sonnet and Haiku. The company says Opus, its most powerful version, outperforms GPT-4 and Gemini Ultra on several reasoning, knowledge and coding tests.

The announcement matters because it puts Anthropic back at the forefront of the race to build leading generative models. OpenAI set the commercial benchmark with GPT-4 over the past year; Anthropic is now offering an alternative with comparable capabilities, image understanding and a tiered lineup for different cost and speed requirements.

Three models for different tasks and budgets

Claude 3 is not a single model. Anthropic has divided the lineup into three tiers:

• Claude 3 Opus is designed for complex tasks such as analysis, research, coding and work that requires keeping large amounts of information in context.
• Claude 3 Sonnet aims to balance capability, cost and speed. As of today, it powers the free version of Claude.ai.
• Claude 3 Haiku is designed for fast, inexpensive responses, making it useful for customer support, document classification and large-scale information extraction. Anthropic plans to make it available soon.

All three support up to 200,000 tokens of context, a measure of how much text they can process in a conversation or document. In practical terms, that makes it possible to work with hundreds of pages, although the limit alone does not guarantee that the model will always find the relevant detail or reason correctly about it.

Opus and Sonnet are already available through the Claude.ai website and Anthropic’s API. Opus requires a Claude Pro subscription, which costs $20 a month in the United States. For developers, Anthropic prices Opus at $15 per million input tokens and $75 per million output tokens; Sonnet costs $3 and $15, respectively. Haiku will bring those rates down to $0.25 and $1.25.

Vision: Claude can now interpret images and documents

The clearest functional addition is that Claude 3 is multimodal: it can analyze both text and images. The system can interpret photographs, charts, diagrams and scanned documents, including pages with tables or other visual elements.

This brings Claude closer to GPT-4 with vision and Gemini. It is not simply a matter of describing an image: the goal is to let users ask about a financial chart, review a scanned form or extract information from a slide. For companies handling unstructured documentation, that leap may matter more than a marginal improvement on an academic test.

Anthropic also says Claude 3 gives fewer unnecessary refusals than its earlier models. That is an important issue for enterprise assistants: an overly cautious system may refuse to help with legitimate requests, while one that is too permissive may provide inappropriate instructions or make up information. Finding the right balance remains one of the sector’s central challenges.

Benchmarks put Opus ahead, with caveats

In its evaluations, Anthropic places Claude 3 Opus ahead of GPT-4 and Gemini Ultra on tests including MMLU, which measures knowledge across dozens of subjects; GPQA, focused on advanced-level scientific questions; GSM8K, which covers grade-school math problems; and HumanEval, for coding.

The company reports, for example, scores of 86.8% on MMLU and 84.9% on HumanEval for Opus. Those are notable results, but they should be read as a signal rather than a definitive verdict. Benchmarks measure closed sets of questions and may favor particular training techniques. Moreover, a model that scores highly on an exam can still make mistakes when summarizing a contract, interpreting an ambiguous instruction or citing a nonexistent source.

The comparison with GPT-4 is not entirely static, either. Commercial models change through updates, product configurations and external tools. For a user or a company, the relevant difference will not simply be which model tops a results table, but which offers the best reliability, latency, price, privacy and integration with existing workflows.

A competition no longer decided solely by the biggest model

With Claude 3, Anthropic is adopting a strategy similar to that of other providers: reserving maximum capacity for high-value tasks while offering lighter models for high-volume operations. That segmentation matters because running large models is expensive. A company does not need Opus to label thousands of emails, but it may need it to review complex technical documentation or assist an analyst.

Anthropic has deployed Opus under its AI Safety Level 2 standard, an internal framework with controls for models that, according to its assessment, do not yet present qualitatively new, high-impact risks. The company maintains that it has found no signs that Claude 3 could substantially help untrained actors develop chemical, biological, radiological or nuclear weapons.

The launch increases pressure on OpenAI, Google and the rest of the labs. The real comparison starts now: when Claude 3 faces imperfect documents, lengthy instructions and users who do not phrase their requests like exam questions.