Manus and the agent test: completing a task is not proof of autonomy

On March 6, 2025, Manus introduced itself as a “general agent” able to receive a goal and deliver a finished result: research candidates, compare apartments, analyze data or build a web page. Its official launch video shows planning, browsing and execution inside a virtual computer. The demonstration drew enormous attention and invitation-only demand, but a selection of successful cases cannot reveal the failure rate or how much human work occurred behind them.

The comparison with DeepSeek mixes two layers. DeepSeek gained attention for foundation models: systems that predict and generate content. Manus is an agent product that may use third-party models and add instructions, memory, tools, an execution environment and a loop that observes results and chooses the next step. Evaluating it only by its underlying model would be like judging a delivery service by the van’s engine.

What turns a chatbot into an agent

A chatbot produces an answer. An agent maintains state and chains actions to change something outside the conversation. A minimal architecture receives a goal, divides it into steps, selects a tool, executes it, observes what happened, revises the plan and decides when to stop. It may open pages, write code, read files or generate an artifact. The value lies in the loop, not in the “autonomous” label.

That loop already existed in other products. OpenAI described deep research on February 2 as a capability that searches, interprets and synthesizes sources over several steps using browsing and Python. Two days later, Hugging Face published an open reproduction of a research agent and explained its parts: a model, a framework organizing actions, and tools for searching or running code. Manus extends the promise toward a broader range of finished products, but it did not invent the agent structure.

“Autonomous” also does not mean “unsupervised.” A user defines the goal, grants permissions, may intervene and validates the result. The provider chooses tools, limits and system instructions. If a person has to correct step three, the task may still be useful even though autonomy is partial. It is better to measure levels: recommend actions; execute with confirmation; execute inside a sandbox; or act on external systems without approval. Each level changes the risk.

What GAIA measures—and what it leaves out

Manus supported part of its launch with GAIA results. The original GAIA paper describes 466 questions requiring reasoning, web browsing, tool use and sometimes files or images. Answers are short and verifiable; 300 are withheld to maintain a private leaderboard. Difficulty levels roughly track the number of steps and tools needed. It is closer to execution than a multiple-choice exam.

The paper itself provides a useful control for reading the score: human participants solved 92% of the questions, versus 15% for GPT-4 with plugins in the evaluation published in 2023. That gap is neither a current product ranking nor a direct comparison with Manus; it states GAIA’s purpose: testing robustness on tasks that are simple for a person but difficult for a system. A percentage improvement must therefore be tied to the experiment’s version, tools and date, not treated as a universal measure of autonomy.

Still, answering a GAIA question does not equal safely completing a business assignment. The design itself says web interaction is essentially limited to clicks: it does not aim to test posting comments, booking meetings or performing actions that might create spam. It also does not fully measure monetary cost, elapsed time, number of retries, damage from a mistaken action or susceptibility to instructions planted on a hostile page.

A reader should demand the protocol before accepting a table. Was the public or private split evaluated? Is the score pass@1, the first attempt, or selected from many runs? Which tools were available? What time and budget did each task receive? Was human intervention counted? In February, Hugging Face reported 55.15% on GAIA validation for its open reproduction and showed that replacing code actions with JSON actions sharply reduced performance. The lesson is important: scaffolding can change the score while the model remains the same.

A test that users can reproduce

Before trusting Manus or any agent with real work, build a small evaluation containing ten representative assignments. They should not be polished demos but ordinary trouble: a source that does not respond, two similarly named files, a contradictory figure, an ambiguous instruction and a form that must never be submitted without approval. Define in advance what counts as a correct output and which action is forbidden.

The minimum log has six columns. Outcome: did it deliver the requested artifact? Accuracy: can every fact be checked? Process: does the trace expose sources and actions? Recovery: did it notice an error and change strategy? Resources: how much time, money and how many calls did it consume? Safety: did it respect permissions and request confirmation before an irreversible action? A completion percentage that ignores any of these can reward a fast but dangerous system.

Run the test more than once. Models are variable and the web changes; one success may be luck. Three attempts per task reveal dispersion. Compare against two baselines as well: a person using ordinary tools and the same model used as a chatbot. If the agent does not improve time or quality over chat, autonomy has added complexity without value.

For sensitive tasks, the environment matters as much as intelligence. Start the agent in a sandbox with fictional data, limited credentials, an explicit domain allowlist and read access before write access. External effects—send, buy, delete, publish—need a confirmation gate. A visible “agent computer” helps observation, but an animation is not a substitute for an exportable trace and enforceable permissions.

Four failures a demo usually hides

The first is false completion: the agent declares success although a file is missing or a page contains invented data. The second is looping, when it repeats searches or calls and raises cost without progress. The third is goal drift: it optimizes an easy part and forgets the condition that made the task valuable. The fourth is prompt injection: a page or document tells it to reveal information or ignore the user.

These failures do not make agents useless. They move the engineering burden. Systems need deliverable validation, step and budget limits, separation between untrusted content and instructions, minimum necessary privileges and human approval for important effects. A better model may improve planning, but it does not remove controls that would also be needed for a new employee.

Beyond “the second DeepSeek”

As of March 6, the solid claim about Manus is limited: an official demonstration shows an agent combining planning and tools, the company reports strong GAIA results, and public access is too restricted for broad independent evaluation. There is no basis for saying it is the first general agent, that it always works without supervision or that one score proves superiority across tasks.

The useful news is not that China has produced another viral symbol. Competition is shifting from the isolated model to the system around it. An agent is a model plus tools, state, permissions, verifiers and a recovery strategy. A reader who keeps that formula can judge the next release by asking not which video looks most autonomous, but what share of real tasks it completes correctly, at what cost, with what traceability and without crossing which boundaries.