Apple proposes fine-tuning models without losing sight of what they know

On May 11, 2026, a paper from Apple proposed approaching the fine-tuning of language models not by choosing a fixed data mixture, but by recomputing it during training under explicit limits. Tuning a model to code better, answer medical questions, or follow a particular instruction has a well-known cost: it can get worse at what it already knew how to do — the phenomenon the field has spent decades calling catastrophic forgetting. The manuscript, signed by Eleonora Gualdoni, Sonia Laguna, Louis Bethune, Joao Monteiro, Pierre Ablin, and Marco Cuturi, turns that dilemma into a constrained optimization problem solved on the fly, while the model trains.

The technique is called DynaMiCS, for Dynamic Mixtures for Constraint Satisfaction. The idea is simple to state and demanding to execute: before deciding how much to train on each dataset, the system briefly probes what effect each one has on several evaluations. It then uses those measurements to pick a mixture that improves the chosen objectives without exceeding an acceptable degradation in the capabilities the team wants to keep.

Learning without forgetting

In multi-domain fine-tuning, a team may want to improve code generation while making sure the model does not lose general knowledge, instruction following, or its scores on a safety evaluation. The usual practice is to fix data proportions — uniform, by dataset size, or hand-tuned — and check the result at the end. If the mixture does not work, the whole training run has to be repeated with another recipe, and every repetition costs compute time that does not always exist.

DynaMiCS turns that process into constrained optimization. For each training dataset, it runs a short series of isolated steps and measures how the losses change across the evaluation domains. Those measurements form what the paper calls a slope matrix: a local estimate of how much each evaluation rises or falls per step trained on each dataset — in other words, which transfer helps and which interference hurts. With it, the system computes weights for the next stretch of training, measures again later, and corrects the mixture.

The schedule of those corrections is not continuous, because probing costs too. The method spaces them over intervals that keep doubling, with three variants depending on how much probing happens early: with no warmup there are six mixture recomputations per run; with light warmup, nine; with dense warmup, eleven. Each probe is also capped at 128 steps per dataset, so the extra spending is bounded in advance.

With the slopes in hand, choosing the mixture becomes a linear program with a squared relaxation of the constraints. At each recomputation the system generates 45 candidate solutions — fifteen penalty strengths combined with three tolerance margins — and keeps the feasible one that most improves the objectives. The important distinction is that the domains to protect do not have to be training data at all: they can be safety benchmarks, commonsense suites, or instruction-following tests. The method tries to keep their losses below a reference taken before fine-tuning began and, if no combination satisfies everything, it relaxes the constraint and picks the least-violating option.

What the study measured

The authors evaluated the proposal on 50 fine-tuning scenarios with between one and three target domains and between three and ten constrained domains. They used Qwen2.5-3B with LoRA adapters of rank 8, 32, and 64, and Qwen3-8B and Gemma3-12B at rank 32; LoRA tunes a model through low-rank matrices without touching all of its weights, and is today the most common route for fine-tuning on a contained budget. Each run trains for 2,048 steps with batches of eight 512-token sequences — about eight million tokens per experiment — and the manuscript accumulates some 7,000 runs across configurations, scenarios, and seeds. The targets span mathematical reasoning, code, instructions, medical questions, function calling, and multilingual tasks, with datasets such as MetaMathQA, NuminaMath, GSM8K, and CodeFeedback; the constraints include safety, common sense, and general knowledge, measured with suites such as tinyMMLU, tinyTruthfulQA, and NemotronSafety.

The comparison is not against a straw man. The baselines include uniform mixtures, mixtures proportional to dataset size, a sweep of five different splits between target and protected data, and a dynamic method based on gradient alignment. In those experiments, DynaMiCS beat them all. On Qwen3-8B, the manuscript reports a median 11% reduction in target-domain perplexity versus 0% for the fixed baselines compared. On Gemma3-12B, it reports 24% versus 18% for the best baselines; on Qwen2.5-3B, between 8% and 9% versus 0.4% for the best of ten baselines. And on the part that gives the method its name — how often the constraints end up being respected — the authors report satisfaction rates consistently above any fixed mixture they compared. The figures are taken at each run's best feasible checkpoint — the point in training where the objectives improve most without breaking any constraint — and on feasibility the advantage holds even against sweeps allowed to spend ten times more compute.

The extra cost comes from the probes and intermediate evaluations, but it stays far from the alternative. According to the paper, even a modest sweep of ten independent mixtures costs ten times a full training run; DynaMiCS, with all its probing included, accumulates between 1.4 and 1.7 times, depending on how much warmup is used. For a team that today hunts for the right mixture by repeating full runs, that is the method's economic argument: pay a bounded, known overhead instead of multiplying blind trainings. The manuscript also stresses that all of this works without reference models or per-example scoring, two common — and expensive — ingredients in other data-mixture selection methods.

A constraint is not a guarantee

The result does not mean the model keeps every important capability by definition. DynaMiCS preserves what is measured and included as a constraint. An incomplete evaluation, a poorly chosen metric, or a behavior that never shows up in the tests will remain outside the optimization. The authors write it themselves in their limitations: the method depends on the quality of the evaluation domains and may fail to preserve broader capabilities when those evaluations are narrow.

Nor is it mathematical magic. The slope matrix is a local linear approximation: it assumes the effects of the datasets combine additively and predictably enough over the short interval until the next update. The authors concede that the approximation loses accuracy when the intervals grow long, that interactions between domains fall outside the model, and that aggressive early specialization can bias the rest of the training trajectory.

That is precisely why the work is useful. It forces a team to declare what it wants to improve, what it is not willing to degrade, and how each thing will be measured. Instead of treating the data mixture as a fixed recipe, it turns it into a decision that gets revisited throughout training. And it leaves the reader with a question that outlives this manuscript: when someone claims they fine-tuned a model "without losing capabilities," ask for the list of which capabilities were measured as constraints, and with which benchmarks. Whatever was not on that list was not protected. For anyone tuning models to specific tasks, that shift in framing can be as valuable as the improvement percentage in a table.