o3 and o4-mini use tools: how to audit the whole agent
OpenAI gave its reasoning models coordinated access to search, Python, files and images. The unit to evaluate is the model, its tools and permissions.
On April 16, 2025, OpenAI released o3 and o4-mini with a change more consequential than any isolated benchmark score: inside ChatGPT, both could decide when to combine web search, file analysis, Python, images and image generation. The change did not turn the chat product into an infallible operator. It turned one answer into a workflow: observe, select a tool, receive a result, update the plan and repeat.
That pattern is the durable skill worth learning. When a model uses tools, the outcome no longer depends only on what it “knows.” It also depends on the query it formed, the source retrieved, the code executed, the permissions granted and whether the result was checked. OpenAI’s original announcement positions o3 as the higher-capability option and o4-mini as the faster, more efficient one. The practical choice must be made on the whole job, not the name.
What tool use actually means
A model without external access generates a continuation from its input and training. An equipped model can additionally produce a structured request: search for a phrase, call a function with arguments, run code or query a file. The surrounding system performs the operation and returns the result. The model then decides whether it can answer or needs another step.
Headlines commonly merge three actors. The model proposes an action; the application supplies and executes tools; an external service provides data or changes state. If search fails, Python has a faulty dependency or the application exposes the wrong file, “the agent” can fail even when its language reasoning is good. If the application does not offer a tool, the model cannot summon it by force of will.
ChatGPT integrated those pieces behind one interface. In the API, developers configure the environment. The official o3 model page lists web search, code interpreter, file search, image generation, functions and MCP among supported tools. The o4-mini model page lists search, code, files, functions and MCP but not image generation in its current tool list. Product, model and tools are not synonyms.
This distinction corrects an attractive shortcut: o3 and o4-mini do not “turn ChatGPT into an agent” on their own. The models contribute planning and action selection; ChatGPT contributes the toolset, permissions, execution and interface. Agentic behavior emerges from the combination.
A useful chain can also propagate an error
Consider a request to compare California electricity consumption with the previous summer. A system might break it into period, data, normalization and chart; locate a source; download figures; write Python; run the calculation; and draft an explanation. Each step produces evidence that should be inspectable.
The same autonomy can amplify an early mistake. If search retrieves peak demand instead of energy consumed, code may calculate the wrong variable flawlessly. If megawatts and megawatt-hours are mixed, a polished chart makes the error more persuasive. If one source offers provisional data and another revised data, the comparison may look precise without being equivalent.
Control therefore does not consist of asking the model to “think better.” It consists of defining invariants: acceptable sources, units, period, handling of missing values, discrepancy thresholds and evidence format. Before approving the chart, a person should be able to identify the series used, the transformation applied and whether the result can be reproduced.
A tool supplies capability, not truth. Search opens a door; it does not guarantee a primary source. Python follows instructions precisely; it does not certify that the formula answers the question. Image analysis can enlarge or rotate a photograph; it does not automatically repair a visual misreading.
“Thinking with images” has observable limits
OpenAI described o3 and o4-mini as able to incorporate images into their problem-solving process and transform them by cropping, zooming or rotating. The official visual-reasoning demonstration shows inverted handwriting, diagrams and screenshots. This goes beyond assigning a label to an image: the system can change the view and inspect it again.
The same page lists limitations: redundant tool calls or manipulations, basic perception mistakes and different paths across attempts that can produce wrong answers. That disclosure is more useful than the slogan. It identifies what to test: small text, rotated images, low contrast, crops that remove context and repeated runs of the same task.
In a serious process, visual output should retain the original image, transformations applied and the region supporting each extracted fact. For an invoice, radiograph or configuration screenshot, a plausible explanation is not enough. Ask for coordinates, verbatim transcription where appropriate and explicit uncertainty. Human review remains part of the system in high-risk domains.
How to read the benchmark numbers
OpenAI reported that o3 made 20% fewer major errors than o1 in difficult external-expert evaluations and that o4-mini with a Python interpreter achieved 99.5% pass@1 and 100% consensus@8 on AIME 2025. The announcement itself warned that those results should not be compared with models lacking tools. The caveat is central: the evaluated object is model plus environment.
Pass@1 is the share solved on a first attempt under stated conditions; consensus@8 combines multiple runs. Neither is the probability of answering any user’s mathematics problem correctly. AIME is a specific examination, and Python reduces the difficulty of some calculations. A team analyzing contracts, handling complaints or maintaining software must evaluate those tasks with its own tools and failure modes.
Configuration matters too. OpenAI says its comparisons used high reasoning effort, similar to o4-mini-high in ChatGPT. More reasoning can increase latency and consumption. A production metric should be quality per cost and time at a known error rate, not an unconstrained maximum score.
Safety does not end at the model boundary
The o3 and o4-mini system card says OpenAI rebuilt safety-training data, added mitigations and evaluated biological and chemical capability, cybersecurity and AI self-improvement. The company concluded that neither model reached the “High” threshold under its Preparedness Framework in those categories. That is a vendor evaluation under defined scenarios, not automatic approval for every deployment.
Tools change the risk. A wrong answer stays in a conversation; a function with permissions can send an email, alter a database or initiate a purchase. The baseline principle is least privilege: expose only necessary tools, restrict recipients and amounts, separate reads from writes and require confirmation for actions that are hard to reverse.
Every step also needs a trace. It should record model version, selected tool, arguments, response, cost, duration and final outcome. Secrets should be excluded or protected, but the decision must be reconstructable. Without traces, a more autonomous system is harder both to improve and to stop.
An evaluation that measures the whole system
Before adopting o3, o4-mini or a successor, build a set of real cases with a verifiable answer or acceptance criterion. Measure final success, correct tool selection, argument errors, source quality, cost, latency and human intervention. Include cases where the tool is unavailable, returns contradictory data or tries to induce the model to disregard instructions.
Compare configurations, not two labels: model, version, reasoning effort, tools, permissions, instructions and limits. o4-mini may win at volume despite needing occasional review; o3 may justify its cost on rare tasks where failure is expensive. A simpler non-agentic flow may win when the sequence is stable.
The April 16 launch moved the frontier from chatbot toward orchestrator. The responsible way to use that shift is to treat every call as a verifiable operation. A reader who retains that discipline can recognize any future “agent” by more than a persuasive answer: whether the system selects well, acts within limits, leaves evidence and fails safely.
This article was produced with artificial intelligence under human editorial oversight.
