A Solver Can Perfectly Solve the Wrong Problem

On May 24, 2026, VeriSimpl was submitted as a study of a silent failure in AI-assisted optimization: code can run, a solver can declare an optimum, and the formulation can still represent a different problem. A misindexed constraint, a cost counted twice, or a variable with the wrong meaning is enough to produce a mathematically impeccable and operationally useless number.

The VeriSimpl preprint proposes using the solver not only at the end to calculate, but during verification to construct small diagnostic questions. An LLM reasons over those simplified versions from the original description. If its answer and the program's behavior diverge, the translation from natural language to mathematics may be unfaithful.

Four layers that must stay separate

A formulation contains decision variables, constraints, and an objective function instantiated with data. Code can be executable without being correct. A solver can find a feasible solution compatible with the written constraints. It can also certify that the solution is optimal for the written objective. None of those properties alone establishes the fourth: semantic fidelity to what the person intended to decide.

Consider inventory planning. The business asks to maximize profit while charging storage cost once per quarter. The program adds that cost inside an extra loop and counts it repeatedly. The solver has no basis to object: it precisely applies the rules received. The mistake occurred before solving, when the sentence became an expression.

Successful execution is therefore a syntax and integration test, not a meaning test. Even a reasonable solution can hide the defect if the input does not activate the faulty constraint. Verification needs questions designed to make every part of the contract observable.

How VeriSimpl simplifies

The system generates several candidate programs. For each, the solver constructs queries that isolate a property. During constraint verification, it holds the rest fixed and tests values representing strict satisfaction, a boundary, or a violation. The LLM receives those values and the natural-language description and decides whether they should be feasible.

The paper shows a storage constraint that incorrectly divides capacity of 200,000 by 10,000, limiting inventory to 20. The solver creates examples around the boundary: 19 and 20 feasible, 21 infeasible. From the original description, the LLM can see that this boundary does not follow. An error buried in a large formula becomes a concrete question about three numbers.

The second family checks variables and the objective. The solver calculates a candidate solution; almost all variables are then fixed and one part is hidden for the LLM to infer from the description. If purchases, sales, and inventory are known, calculating profit no longer requires solving the full plan. A mismatch between the expected value and the program's value exposes incorrect aggregation.

VeriSimpl also adds a light type check: whether a variable should be binary, integer, or continuous and whether its domain fits the described decision. Constraint, variable, and type signals are combined to select the candidate with the highest verification score before running the complete problem.

What was measured

The evaluation is not one exam. It includes 269 test problems from NL4Opt after infeasible cases were removed; 67 LP and MILP problems from textbooks and lectures in NLP4LP; 17 checked feasible complex operations-research problems in CompOR; and 100 industrial cases spanning aviation, manufacturing, logistics, and energy in IndOR.

NLP4LP comes from OptiMUS, which organized model and code generation into modules for long descriptions. IndOR was introduced by ORLM. Linking those origins reveals what each benchmark means: they are not four replicas of one problem type, and their sizes differ sharply.

With GPT-4o as the base model, average direct-generation accuracy was 56.8% across the four sets and VeriSimpl reached 65.5%. With R1, it rose from 62.5% to 72.8%. On NL4Opt, VeriSimpl scored 88.1% with GPT-4o and 88.8% with R1; on CompOR it scored 76.5% with both. These are results from the experimental setup, not a guaranteed rate for a new business problem.

Precision and coverage tell different stories

The authors call a case fully verified when every query passes. With GPT-4o, those cases had 91.5% average precision: that share of everything marked fully verified was correct. But average coverage was 34.2%: the signal reached only that portion of all correct answers. With R1, average precision was 78.5% and coverage 23%.

A signal can be selective and useful. High precision means “all checks passed” adds confidence; low coverage means many correct solutions do not receive that seal. It does not justify automatically rejecting everything that fails verification or saying the method certifies every result. It reduces part of manual review and leaves another part untouched.

Ablations strengthen the qualification. Constraint feasibility alone was a weak proxy for correct interpretation. Reasoning over variables alone was insufficient too. Removing components reduced accuracy or precision, and no individual signal was consistently strongest across all domains.

The most important blind spot

The paper's failure section describes incorrect answers that passed verification. The program and verifying LLM had made the same mistaken interpretation of language such as “start time” in shift scheduling or omitted certain costs from profit. Because they shared the mistake, their answers were consistent.

The method assumes decision variables are given and does not yet check everything a formulation may have omitted completely. If a duty in the description never enters the program, queries derived from that program may never ask about it. Internal coherence does not necessarily discover external absence.

This limit recurs in hybrid systems. A runtime-independent verifier can still share data, categories, or assumptions with the generator. Genuine independence requires diverse evidence: the source text, cases prepared by a domain expert, invariants, extreme data, and review of definitions—not merely another pass by the same model.

A protocol for a real formulation

First, write the objective, variables, constraints, units, and assumptions separately. Every quantifiable noun in the request should appear somewhere in the formulation or be explicitly declared out of scope. A coverage table exposes omissions before anyone inspects the result.

Second, build minimal cases whose answers can be calculated by hand: one product, one period, two routes, or one constraint. Include boundaries and impossible states. Third, apply directional changes. If a cost rises, capacity falls, or a prohibition appears, record before execution which properties should remain and which should change.

Fourth, compare alternative formulations of the same requirement. An equation can have equivalent forms; if they yield different solutions, a semantic decision remains unresolved. Fifth, ask a person responsible for the process to sign off on the specification and cases, not merely the code. The solver expert validates mathematics; the domain expert validates the represented world.

The skill that remains

VeriSimpl contributes not an automatic certificate but a way to manufacture small questions capable of contradicting a large formulation. It turns part of semantic review into observable experiments and reports both the gains and the places where consistency misleads.

The transferable skill is demanding four distinct demonstrations: the program runs, the solution satisfies written constraints, it is optimal for the written objective, and its variables, constraints, and objective correspond to the real decision. The solver answers the first three relative to its model. The fourth needs traceability to language, adversarial cases, and a person able to say, “yes, this is the problem we meant to solve.”