Humanity's Last Exam: The Test AI Fails With Under 10%

Scale AI and the Center for AI Safety (CAIS) today unveiled the results of Humanity's Last Exam, a benchmark designed to test whether AI models can reason and know as much as the best human experts in their fields. The result is stark: none of the models evaluated scored above 10%.

The test arrives at a moment when the exams that used to measure AI progress have stopped being useful. MMLU, the benchmark that has served for years as the reference for evaluating general knowledge in language models, has had top models scoring above 90% for some time now. When an exam becomes saturated—when nearly every system passes with flying colors—it stops telling us anything about who's actually better or how much room for progress remains.

How the exam was built

CAIS and Scale AI's creation process was a global call for submissions. The two organizations collected more than 70,000 proposed questions, from which 13,000 were selected for human expert review and eventually narrowed down to the 3,000 questions that make up the public version of the exam. Behind it are nearly 1,000 contributors from more than 500 institutions across 50 countries, most of them active researchers or university professors.

The questions cover mathematics, the humanities and the natural sciences, in both text-only and multimodal formats, including images and diagrams. The required level is deliberately extreme. One example from the ecology category, cited by Scale AI, asks how many paired tendons are supported by a specific sesamoid bone found in hummingbirds of the order Apodiformes, demanding an exact, unambiguous numerical answer.

To encourage genuinely difficult questions, Scale AI and CAIS offered $5,000 for each of the 50 best submitted questions and $500 for the next 500, plus the chance to be listed as a co-author on the final paper.

The results: under 10%

The exam was administered to several frontier multimodal models: OpenAI's GPT-4o and o1, Anthropic's Claude 3.5 Sonnet, and Google's Gemini 1.5 Pro. According to Summer Yue, Scale AI's Director of Research, in the final round of testing some models began answering a fraction of the questions correctly—always below 10%—though she cautioned that variations in these results are common in model testing and could partly be due to randomness.

Dan Hendrycks, CAIS co-founder and executive director, framed the result in historical perspective. "We wanted problems that would test the capabilities of the models at the frontier of human knowledge and reasoning," he explained. He recalled that when he released the MATH benchmark in 2021 —a set of competition-level math problems— the best model scored below 10%, and few predicted at the time that scores above 90% would be reached just three years later. "Right now, Humanity's Last Exam shows that there are still some expert closed-ended questions that models are not able to answer. We will see how long that lasts," he added.

Summer Yue summed up the project's purpose, noting that they designed what might be the ultimate test—built to demand precise, multi-step logical reasoning with unambiguous answers, at a level that pushes even the most advanced systems to the limits of their capabilities.

Why it matters

A saturated benchmark is a broken yardstick: if every model gets top marks, it no longer distinguishes real progress from statistical noise. Humanity's Last Exam was created precisely to restore that margin for measurement at a moment when large language models are comfortably clearing conventional academic tests.

The fact that today's most powerful systems—including o1, OpenAI's first model explicitly designed to reason step by step before answering—score below 10% on this exam doesn't mean AI hasn't progressed. It means the bar has shifted to territory where the comparison is no longer with average encyclopedic knowledge, but with the frontier of specialized human expertise.

CAIS and Scale AI have announced they will open the dataset to the research community so other teams can evaluate their own systems and study the observed variations in detail, though they will keep a small subset of questions in reserve to preserve the integrity of future evaluations. The question Hendrycks leaves hanging—how long it will take models to approach 90% here too—is what will determine whether this exam remains useful three years from now or meets the same fate as MMLU.