Language and AGI: Measuring Generality Without Mistaking It for Fluency

Re-edited on July 30, 2026, this article begins with a necessary separation: producing convincing language does not demonstrate artificial general intelligence. Language is an extraordinarily broad interface; translation, code, explanations, and plans can all be requested in the same format. A common interface, however, does not automatically turn those tasks into one common capability or prove that a system can learn any new skill.

The claim “this is approaching AGI” needs a definition, a set of tests, and an accounting of resources. Without them, a selected conversation, an academic score, or model size becomes a certificate with no way to falsify it. A useful method decomposes generality, performance, and autonomy and asks what evidence supports each axis.

Predicting text is not a test of generality

An autoregressive model is trained to assign probability to the next token given a context. By repeating that operation, it can generate paragraphs, code, or dialogue. The objective rewards the capture of syntactic, semantic, and factual regularities in the data, but it does not specify truth, planning, experience of the world, or a persistent goal by itself.

GPT‑3 documented a 175-billion-parameter model in 2020, evaluated with zero, one, or a few examples written into its context. The study showed that one interface could address many tasks without gradient updates for each one. It also reported uneven results, limits on several tasks, and risks associated with data and generation. The finding was in-context learning under particular tests, not an experimental declaration of AGI.

Fluency creates an observational problem: the same output presents both an answer and an apparent explanation. A well-formed sentence may come from a correct association, a retrieved source, a tool, a clue in the question, or a plausible invention. Conditions must be varied and components separated before assigning a capability. One conversation does not reveal which mechanism produced success.

Generality, performance, and autonomy are different axes

Performance asks how well a system solves a task. Generality asks how broad the task set is and how much transfers to new ones. Autonomy asks how much the system can do without intervention and for how long. A model may score highly while remaining narrow; a broad-purpose tool may need constant supervision; an autonomous agent may be incompetent yet still cause harm because it executes actions.

The Levels of AGI framework proposes classifying systems by depth of performance and breadth of generality while treating deployment risk, including autonomy, separately. It is the authors’ conceptual proposal, not a universal standard, but it repairs a common confusion: “more capable” does not automatically mean “more general,” and “more general” does not mean “more autonomous.”

The cost of acquiring a skill matters too. In On the Measure of Intelligence, François Chollet proposes evaluating skill-acquisition efficiency while accounting for experience and prior knowledge. Memorizing many tasks during training may create broad coverage without showing efficient adaptation to a genuinely new rule. A transfer claim must state what the system saw, what it did not see, and how much help it received.

Benchmarks illuminate one slice

MMLU collects multiple-choice questions from 57 subjects, ranging from elementary mathematics to law and ethics. It supports comparisons of knowledge and question answering under a common format. It does not measure conversation, action, continual learning, or safe use of knowledge by itself. Choosing the right option also does not expose the process: success may involve reasoning, pattern recognition, or previous exposure to similar material.

BIG-bench assembled more than 200 tasks contributed by a broad community to explore language-model capabilities and limits. Diversity is a strength, but it does not turn the average into a natural scale of intelligence. Tasks differ in difficulty, format, sensitivity to instructions, and likelihood of appearing in training data. Selecting only the tasks that improve after results are known creates a biased story.

HELM proposes holistic evaluation: defined scenarios and multiple metrics instead of one number. Alongside accuracy, it considers properties such as calibration, robustness, fairness, bias, toxicity, and efficiency. No suite exhausts a system, but the structure teaches a discipline: describe a capability through its scenario, permitted adaptation, metrics, and limits of comparison.

Benchmarks also expire. Examples, solutions, and analyses circulate publicly and may enter later corpora. A new version can recognize material without acquiring the intended skill. Controls require held-out sets, new variants, cutoff dates, overlap detection, and adversarial tests. If a developer has seen the test during development, the score is no longer an independent surprise.

Saturation is another warning: when almost every system answers correctly, the set no longer separates capabilities in that region. Adding increasingly obscure questions does not always solve the problem; it can turn evaluation into a contest of trivia. A better approach is to design task families with controlled difficulty, record performance curves, and repeat them with perturbations that preserve the rule. This helps reveal whether the system acquired the pattern or one particular surface form.

Language, tools, and the world

A linguistic response can describe an action without performing it. Booking a trip, analyzing a spreadsheet, or controlling a robot requires perception, state, permissions, tools, and verification of the result. Adding those components expands the system but also creates failure points: retrieval of a wrong source, a malformed parameter, excessive authorization, or lack of confirmation.

Evaluation should follow the complete path. Did the model choose the correct tool? Did it cite the document it used? Did it verify that the action completed? Did it detect a contradiction? Did it ask for confirmation before an irreversible step? A base-model score does not answer those questions. A system with search access may answer recent facts absent from its weights; credit should go to the workflow and be measured with the tool available.

Language does not replace grounding in a situation either. “The box is on the left” depends on viewpoint; medical advice depends on clinical history, examination, and risk thresholds; a legal term depends on jurisdiction and date. When a variable is missing, competent behavior may be to ask or abstain. Forcing an answer every time increases the appearance of capability while reducing reliability.

How to audit a claim of progress toward AGI

First, copy the claim literally and define the capability. “Performs better” needs a task, example population, metric, and baseline. “Generalizes” needs a new distribution and a degree of novelty. “Learns” needs permitted experience and cost. “Is autonomous” needs a horizon, actions, supervision, and stopping condition. “Understands” must be translated into behavior that can fail.

Then hold the comparison constant: same version, instructions, tools, budget, latency, and access to examples. Publish disaggregated results and failures, not only the average. Separate a public evaluation from a held-out test and disclose possible contamination. A narrow improvement remains valuable; calling it narrow makes it possible to use where it works without turning it into a promise about everything else.

An adversarial review tries to break the claim with simple changes: rephrase the instruction, alter names and order, remove a clue, require a citation, or introduce a contradiction. If the result disappears, the capability description should become narrower. Abstentions and confidently wrong answers should be recorded too; a system that recognizes its limits may be more useful than one with more correct answers and failures that users cannot detect.

Finally, maintain a capability matrix: tasks mastered, tasks failed, observed transfer, resources used, limits, and test date. The matrix can grow with evidence and retreat when the environment changes. The transferable skill is requiring that map before accepting “closer to AGI.” Language can be a window onto many skills; generality is demonstrated beyond the window, with controlled novelty, comparable costs, and visible failures.