Phi-4 and the test that matters: when a small model is enough

On December 11, 2024, Microsoft’s official Hugging Face repository made Phi-4, a 14-billion-parameter language model aimed at reasoning, visible. The news is not that “small” has come to mean weak. It is that size is no longer a complete answer: deciding whether a model is useful requires examining its training data, evaluation method, memory requirements and license.

The first verifiable version of the model card and files, dated December 11, lets readers separate facts from promises. It describes a dense, decoder-only Transformer with 14 billion parameters and a 16,000-token context window. It also reports 9.8 trillion training tokens, 21 days of computation on 1,920 H100 80 GB GPUs, and a mixture of synthetic material, filtered public documents, acquired academic books, and question-and-answer datasets. Those figures describe training; by themselves, they do not establish the cost of running a particular application.

“Small” is a comparison, not a specification

Fourteen billion parameters are few beside models with hundreds of billions, but they still represent a substantial system. If each weight is stored in 16 bits, a back-of-the-envelope calculation gives about 28 GB for weights alone: 14 billion multiplied by two bytes. Activations, the attention cache, context, libraries and system headroom are additional. Storing weights in eight or four bits may reduce the total, but quantization can also alter speed and quality. “Runs on one GPU” is therefore not a universal property; it depends on precision, device, input length, user concurrency and the inference engine.

This distinction prevents a common mistake. The card says training used 1,920 H100s, but that figure must not be confused with the resources needed to use the model. Training adjusts billions of values through repeated passes and requires a very different system; inference loads learned values and generates tokens. Parameter counts also need context: a dense architecture activates all its parameters at every step, while a mixture-of-experts model can contain more parameters in total but activate only a subset. The number is a useful orientation point, not a purchasing decision.

The practical question should run in the other direction: what memory, latency and concurrency budget can the task tolerate? An internal assistant serving one person, a batch proofreader working overnight and a service receiving a thousand simultaneous requests may need different models and infrastructure even if they share a prompt. The relevant cost is the whole system’s cost per useful answer, not the name’s prestige or the isolated parameter count.

Phi’s bet: select data that teaches

Microsoft attributes much of Phi-4’s ability to data composition. The initial card says it combines filtered public documents with synthetic, “textbook-like” data meant to teach mathematics, coding, common-sense reasoning and general knowledge. This was not invented for the new release. Textbooks Are All You Need, published by Microsoft researchers in 2023, proposed training small code models with synthetic material resembling carefully written textbooks and exercises. The Phi-3 technical report, available since April 2024, continued the approach by prioritizing heavily filtered and synthetic data to improve capability per parameter.

“Synthetic” does not mean “false,” nor does it guarantee quality. It means that part of the corpus was produced or transformed by models and automated processes. Such material can supply graded explanations, controlled-difficulty exercises and examples scarce on the web. It can also repeat errors, narrow diversity or teach the generating model’s style and blind spots. The useful audit question is not whether synthetic data exists but how it was generated, filtered, deduplicated and checked.

The card gives the broad mixture but does not publish the complete corpus or make every selection decision reproducible. It also states that multilingual material is about 8% of the total and that the model is primarily focused on English. That limitation matters more to a Spanish-language support team than a win on an English mathematics test. A strong aggregate result cannot replace an evaluation using the target language, documents and costly errors of the real deployment.

How to read benchmarks without turning them into a league table

Microsoft reports results on MMLU, MATH, GPQA, DROP, MGSM, HumanEval and SimpleQA. The initial table assigns Phi-4 a score of 80.0 on MATH and 58.1 on GPQA; on SimpleQA, which focuses on short factual answers, it reports only 7.1. The card itself acknowledges that model size limits the amount of world knowledge the system can store. That combination is more informative than an average: it suggests strength on some reasoning tasks and far less reliability as an unsupported encyclopedia.

Some results were produced with Simple Evals, evaluation code published by OpenAI. Opening the protocol is valuable, but it does not answer every question. A reader still needs to check the dataset version, prompt template, number of included examples, answer-extraction method, temperature, scoring rule and possible training-set exposure. Two tables bearing the same benchmark name may not be comparable when these conditions differ.

Three verbs should also remain separate. “Measured” means there is a result under a protocol. “Compared” means the other models were tested under equivalent conditions. “Generalized” means the advantage survives outside that test. A vendor can honestly support the first without proving the third. The responsible reading turns the table into a hypothesis: Phi-4 deserves a local test for jobs resembling those exercises. It is not a universal certificate of superiority.

A local test need not contain thousands of questions. It can begin with fifty real and deliberately awkward cases: ambiguous instructions, long documents, questions with no answer in the supplied material, Spanish inputs, figures that must be copied exactly and requests the system should refuse. Success criteria should be fixed beforehand, memory and latency recorded, and outputs compared blindly against an alternative. Changing the prompt to favor one model after seeing results means the systems are no longer being compared on the same test.

The license is part of the model too

The version posted on December 11 did not carry a permissive commercial license. Its included file was the Microsoft Research License, which authorized non-commercial research use and restricted distribution. This defeats another common shortcut: downloadable weights do not automatically mean open source or permission to embed the model in any product. The repository, exact license and date are all part of the artifact being audited.

For an organization, the minimum checklist has four columns: rights to use and redistribute; hardware requirements under the chosen configuration; quality on its own tasks; and safety controls. Microsoft’s card warns that Phi-4 can produce inaccurate information, was trained primarily on English and was not evaluated for every downstream purpose. It also calls for additional evaluation and mitigation in high-risk settings. Those cautions are operational requirements, not decorative fine print: they determine where source retrieval, human review or a ban on automation is needed.

A sufficiency test, not a size race

Phi-4 makes a durable idea visible: the right model is not the largest one an organization can buy, but the smallest one that clears a representative test with margin, inside the available budget and license. If factual recall is weak, it may be paired with search or document retrieval; if the target language is weak, a MATH score does not repair that; if the license bars the product, the benchmark ceases to matter.

The same method applies to the next release. First, open the original card with its date and version. Then separate training from inference and estimate memory at the precision actually planned. Next, treat every benchmark as a protocol rather than a trophy. Finally, check language, license and performance on local cases. This sequence turns a claim of “more capability with fewer parameters” into a verifiable decision: it asks not whether the model is small, but whether it is sufficiently good for a specific task and can be used lawfully, measurably and safely.