Anthropic launches Claude 3: Opus beats GPT-4 in several tests
Anthropic unveils Claude 3, a family of three vision-capable models with a 200,000-token context window. Its Opus model leads GPT-4 on several benchmarks, although those tests do not fully measure its real-world usefulness.
On March 4, 2024, Anthropic introduced Claude 3 as a three-model family—Haiku, Sonnet and Opus—with vision and different trade-offs among capability, speed and cost. Anthropic’s announcement contains the results table placing Opus ahead of several rivals; it is evidence published by the vendor, not an independent verdict.
The announcement matters because it puts Anthropic back at the forefront of the race to build leading generative models. OpenAI set the commercial benchmark with GPT-4 over the past year; Anthropic is now offering an alternative with comparable capabilities, image understanding and a tiered lineup for different cost and speed requirements.
Three models for different tasks and budgets
Claude 3 is not a single model. Anthropic has divided the lineup into three tiers:
- Claude 3 Opus is designed for complex tasks such as analysis, research, coding and work that requires keeping large amounts of information in context.
- Claude 3 Sonnet aims to balance capability, cost and speed. As of today, it powers the free version of Claude.ai.
- Claude 3 Haiku is designed for fast, inexpensive responses, making it useful for customer support, document classification and large-scale information extraction. Anthropic plans to make it available soon.
All three support up to 200,000 tokens of context, a measure of how much text they can process in a conversation or document. In practical terms, that makes it possible to work with hundreds of pages, although the limit alone does not guarantee that the model will always find the relevant detail or reason correctly about it.
Opus and Sonnet are already available through the Claude.ai website and Anthropic’s API. Opus requires a Claude Pro subscription, which costs $20 a month in the United States. For developers, Anthropic prices Opus at $15 per million input tokens and $75 per million output tokens; Sonnet costs $3 and $15, respectively. Haiku will bring those rates down to $0.25 and $1.25.
Vision: Claude can now interpret images and documents
The clearest functional addition is that Claude 3 is multimodal: it can analyze both text and images. The system can interpret photographs, charts, diagrams and scanned documents, including pages with tables or other visual elements.
This brings Claude closer to GPT-4 with vision and Gemini. It is not simply a matter of describing an image: the goal is to let users ask about a financial chart, review a scanned form or extract information from a slide. For companies handling unstructured documentation, that leap may matter more than a marginal improvement on an academic test.
Anthropic also says Claude 3 gives fewer unnecessary refusals than its earlier models. That is an important issue for enterprise assistants: an overly cautious system may refuse to help with legitimate requests, while one that is too permissive may provide inappropriate instructions or make up information. Finding the right balance remains one of the sector’s central challenges.
Benchmarks put Opus ahead, with caveats
In its evaluations, Anthropic places Claude 3 Opus ahead of GPT-4 and Gemini Ultra on tests including MMLU, which measures knowledge across dozens of subjects; GPQA, focused on advanced-level scientific questions; GSM8K, which covers grade-school math problems; and HumanEval, for coding.
The company reports, for example, scores of 86.8% on MMLU and 84.9% on HumanEval for Opus. Those are notable results, but they should be read as a signal rather than a definitive verdict. Benchmarks measure closed sets of questions and may favor particular training techniques. Moreover, a model that scores highly on an exam can still make mistakes when summarizing a contract, interpreting an ambiguous instruction or citing a nonexistent source.
The comparison with GPT-4 is not entirely static, either. Commercial models change through updates, product configurations and external tools. For a user or a company, the relevant difference will not simply be which model tops a results table, but which offers the best reliability, latency, price, privacy and integration with existing workflows.
A competition no longer decided solely by the biggest model
With Claude 3, Anthropic is adopting a strategy similar to that of other providers: reserving maximum capacity for high-value tasks while offering lighter models for high-volume operations. That segmentation matters because running large models is expensive. A company does not need Opus to label thousands of emails, but it may need it to review complex technical documentation or assist an analyst.
Anthropic has deployed Opus under its AI Safety Level 2 standard, an internal framework with controls for models that, according to its assessment, do not yet present qualitatively new, high-impact risks. The company maintains that it has found no signs that Claude 3 could substantially help untrained actors develop chemical, biological, radiological or nuclear weapons.
The launch increases pressure on OpenAI, Google and the rest of the labs. The real comparison starts now: when Claude 3 faces imperfect documents, lengthy instructions and users who do not phrase their requests like exam questions.
A table does not choose the model for you
Every benchmark defines a small universe: questions, scoring method, assistance and exact version. An average combines knowledge, reasoning and code even when an organisation needs only one task. Before accepting the ranking, open the protocol, distinguish few-shot evaluation from best-of selection and check whether competitors received the same instructions.
Contamination is another limit. If questions or solutions appeared in training data, a result may measure recall as well as generalisation. Private tests and cases created after the training cut-off reduce that risk but still need a verifier. Executing tests is stronger than judging coding style; citations make document claims checkable.
A family represents trade-offs
Haiku, Sonnet and Opus are not a ladder on which the largest is always right. A high-volume verifiable task may favour speed; an expensive decision may justify more capability and review. Build an in-house set, fix a threshold and measure quality, latency, price and abstention. Choose the model by function, not prestige.
Vision needs a separate battery: photographs, charts, tables, small text and skewed documents. “Multimodal” means accepting more than one input type. It does not guarantee accurate reading of every cell or spatial reasoning. Requiring a cited region or transcription of the relevant passage localises error before the answer is used.
The transferable skill is to turn a vendor ranking into a comparable evaluation: version, protocol, in-house task, verifier, cost and failures. That record remains useful as table leaders change because it measures the work that matters rather than the publicity contest.
Safety claims need the same discipline. Fewer refusals may be an improvement when legitimate requests are recovered and a regression when harmful conduct opens up. Build close pairs—one allowed task and one prohibited task—and measure assistance, refusal and explanation. One average cannot capture that boundary.
Production also needs an escalation path. When evidence is insufficient, a person reviews or the system abstains. The best laboratory score does not remove the need to log incidents, compare revisions and withdraw a version whose risk worsens.
The final report includes error examples, not only averages. A repeatable failure can drive controls or reject a use; a high score without samples does not show where to review. Choosing a model also means choosing which errors the surrounding process can detect.
This article was produced with artificial intelligence under human editorial oversight.
