Mixtral 8x7B: why activating fewer parameters does not mean loading a small model

On December 11, 2023, Mistral AI released Mixtral 8x7B, an open-weight model that activates part of its network to process each token. The company’s release page reports 46.7 billion total parameters, 12.9 billion used per token, a 32,000-token context and an Apache 2.0 license. It also says Mixtral beats Llama 2 70B and matches or exceeds GPT-3.5 on most of its tests.

The architecture separates two resources that are often conflated: stored capacity and computation per token. Mixtral can contain many routes without traversing all of them at every step. But using 12.9 billion parameters to compute a token does not make the file as small as a 12.9-billion-parameter model or guarantee the same memory requirement. That distinction is the key to reading any mixture-of-experts release.

Eight experts, two routes per token

Mixtral is a decoder Transformer whose feed-forward layer contains eight parameter groups. At each layer, a router assigns every token to two groups and combines their outputs. “Expert” is a technical name: it does not imply that one block practices medicine and another speaks French, or that a person can manually select a profession. They are learned blocks, and their specialization may be partial or difficult to interpret.

A simplified example helps. The word “bank” enters a layer; the router calculates eight scores and chooses the top two. Their blocks process the representation and the outputs are weighted. The next layer may select another pair. The route depends on token and context. The model is not running eight complete answers or holding a final vote among eight chatbots.

The idea predates Mixtral. The Switch Transformers paper, published by Google researchers in 2021, explained how selecting different parameters for each input can increase total capacity while containing computation. It also documented difficulties: device-to-device communication, training instability and load balancing. Sparse activation offers efficiency, not free magic.

Training efficiency and inference efficiency must also be separated. Switch Transformers reported up to sevenfold pre-training speed gains with the same resources and a fourfold speed-up over T5-XXL in its setup. Those results show that sparse activation can save compute in a particular experiment; they do not guarantee Mixtral’s latency when serving users. Training measures steps and communication while parameters are learned; serving adds batching, cache and expert movement. A number from one phase should not be used to sell the other.

Active computation is not resident memory

To generate a token, Mixtral uses only two experts at each layer plus shared components. This yields Mistral’s stated 12.9 billion active parameters—not 14 billion obtained by multiplying two by seven. The 8x7B name is a useful label, not exact architectural accounting. Total parameters are 46.7 billion, not 56 billion, because parts are shared and the name is rounded.

Computation falls relative to a dense model of similar total size, but all eight experts’ weights must remain available for routing. A normal implementation loads them into GPU memory or distributes them across devices. At BF16, two bytes per parameter gives an estimate of about 93.4 GB for 46.7 billion weights alone, before cache, activations and software. Quantization or sharding reduces pressure on each device but may change quality and add communication.

“Cheaper than Llama 2 70B” therefore needs conditions. Mistral reports six-times-faster inference in its comparisons, but speed depends on hardware, batch size, context length, precision, library and expert distribution. An MoE may save operations while still waiting for tokens to move between devices. The useful business metric is tokens per second and cost per correct output under the intended configuration, not active parameters alone.

The GPT-3.5 comparison has qualifiers

The claim that Mixtral matches or exceeds GPT-3.5 comes from its manufacturer. The table combines knowledge, reasoning, comprehension, mathematics, coding and multilingual tests. It is a reason to evaluate the model, not a certificate that the products are equivalent. “GPT-3.5” can refer to different versions and dates; a conversational API also includes instructions, moderation and tuning that are not identical to a base model.

Mixtral was released in two variants. The base model predicts text and, according to the official weight repository card, contains no moderation mechanism. Mixtral Instruct received supervised fine-tuning and preference optimization for instruction following. Comparing the base model with a chatbot or using Instruct’s score to describe the base mixes systems. Full model name, version, chat template and configuration should accompany every figure.

A minimum reproduction fixes five variables: exact weight commit, precision, inference engine, prompts and dataset. It then records quality, latency, peak memory and failures. A comparison with a closed API must record version or date because the provider may update it. A result shown only as an image on a vendor page, without outputs or code, is less auditable even if the number itself is genuine.

Open weights do not mean everything is open

Mistral released the weights under Apache 2.0, a permissive license allowing use, modification and distribution under its terms. This is material openness: the repository makes the model downloadable and runnable on private infrastructure. Users can inspect files, quantize, fine-tune and avoid relying on an API for every request.

Yet open weights do not provide full reproducibility. The release says pretraining used data extracted from the open web, but does not publish the corpus, training code, compute budget or every filter. The finished artifact can be studied and transformed; the process cannot be rebuilt from the information supplied. Calling it “open source” without stating which components are available erases this difference.

The license does not provide automatic safety either. The repository warns that the base model has no moderation. A self-hosting organization controls its data and gains flexibility, but also assumes evaluation, filtering, service protection, privacy, monitoring and compliance. Running on private servers may improve sovereignty; it does not make outputs reliable or remove the need to review input provenance.

How to decide whether an MoE fits

A practical test begins with the bottleneck. If computation is scarce but memory and fast device interconnects are available, sparse activation can be attractive. If the constraint is fitting every weight on one GPU, the 46.7-billion total remains decisive. If the application serves many short responses, batch throughput matters; if it reads long documents, the attention cache and context cost grow.

Next comes the task. Ten general benchmarks cannot replace a hundred local examples containing the actual language, format and errors. Include instructions with no answer, fictional private data, long inputs and cases requiring refusal. Instruct may suit conversation; the base model may suit specialized fine-tuning. The correct choice depends on the final system, not the most favorable table.

Mixtral matters because it combines a permissive license, downloadable weights and an efficient architecture that organizations can measure. The durable lesson is more precise than “8x7B equals GPT-3.5”: for an MoE, separate total parameters, active parameters, memory, computation and evaluation protocol. Those five questions show when sparsity cuts cost and when a name merely hides a large model.