Meta launches Llama 4, fueling the benchmark fight

Meta has released Llama 4 Scout and Llama 4 Maverick, two new language models that also process images natively. The company presents them as the biggest leap forward for the Llama family to date: they feature a more efficient architecture and, in Scout’s case, a context window of up to 10 million tokens.

The announcement comes with an uncomfortable discussion for the industry. Maverick’s eye-catching LMArena score belongs to an experimental version optimized for chat, not the model developers can download today. That difference does not, by itself, invalidate the result, but it makes it harder to compare that ranking with a reproducible product.

Two expert models, not two small models

Scout and Maverick use an architecture known as mixture of experts, or MoE. Instead of activating the entire neural network for each piece of text, the system routes each task to part of the model. This allows it to store hundreds of billions of parameters while using only a fraction of them for each response.

Llama 4 Scout has 109 billion total parameters, of which it activates 17 billion, distributed across 16 experts. Meta says it can run on a single Nvidia H100 GPU with 4-bit quantization, a technique that reduces the required memory footprint at the cost of some numerical precision.

Maverick also keeps 17 billion active parameters, but has 400 billion in total and 128 experts. It requires an H100 DGX server to run on a single machine. It is a higher-capacity model designed for assistants, image understanding, and general business tasks.

Both models are natively multimodal from the start of training: text and images are fed into the same network rather than connecting a vision model and a language model separately. Meta also says it trained them on more than 30 trillion tokens and data in 200 languages. That scale matters especially outside English, although independent testing will have to determine how much of the improvement carries over to Spanish and other languages that are less represented in the training data.

Ten million tokens: a useful promise with practical limits

Scout’s most striking feature is its 10-million-token context window, which Meta calls an industry-leading capability. A token is a unit of text; in Spanish, one million tokens amounts to roughly several hundred thousand words.

On paper, that limit would make it possible to analyze code repositories, extensive collections of contracts, or large document archives without splitting them into chunks. But accepting that much information does not mean every application can use it cheaply or quickly. Memory, inference cost—the process of generating responses—and the model’s ability to find a relevant detail inside a massive document remain real constraints.

The LMArena ranking does not correspond to the released model

Meta announced that an experimental version of Llama 4 Maverick reached an Elo score of 1,417 on LMArena. The platform pits model responses against one another anonymously and asks human users to choose which they prefer; it then ranks the systems using a chess-inspired score.

That figure placed Maverick among the highest-scoring assistants at the time and above closed models such as GPT-4o and Gemini 2.0 Flash in the ranking. However, Meta’s own documentation makes clear that the model tested was an experimental chat version optimized for that environment and different from the open checkpoint released alongside the launch.

The distinction matters. A benchmark helps guide decisions when the system being measured is one that users can try, deploy, or audit. When a company presents the score of an unavailable variant, the leaderboard measures a real capability, but not necessarily the experience someone who downloads the model will get. LMArena is not a comprehensive test either: it favors answers that users like in a brief conversation and does not replace evaluations of coding, factual reliability, safety, or performance on long documents.

Behemoth remains a preview

Meta has also shown Llama 4 Behemoth, a model still in training with 288 billion active parameters and nearly two trillion in total. The company says it outperforms GPT-4.5, Claude 3.7 Sonnet, and Gemini 2.0 Pro on scientific and mathematical tests such as MATH-500 and GPQA Diamond, but it has not released it.

Scout and Maverick are available on llama.com and Hugging Face under Llama’s community license. They are open weights: they can be downloaded and run, although they are not the same as unrestricted open-source software, since Meta’s license imposes terms of use and requires a separate commercial license for organizations with more than 700 million monthly active users.

Meta has also begun integrating Llama 4 into Meta AI, which is available through WhatsApp, Messenger, Instagram Direct, and the web in markets where the service is offered. The next test will not be a score chart; it will be whether the downloadable models live up to their promises in real products, with manageable costs and verifiable results.