Apple proposes fine-tuning models without losing sight of what they know

On May 11, 2026, a paper from Apple proposed addressing language-model fine-tuning not by choosing one fixed data mix, but by recalculating the mix during training with explicit limits. Fine-tuning a model to code better, answer medical questions or follow a particular instruction comes with a familiar cost: it can get worse at what it already knew how to do.

The technique is called DynaMiCS, for Dynamic Mixtures for Constraint Satisfaction. Its idea is easy to state, though demanding to run: before deciding how much to train on each dataset, the system briefly probes the effect of each one on several evaluations. It then uses those measurements to select a mix that improves chosen targets without exceeding an acceptable decline in capabilities that need to be preserved.

Learning without forgetting

In multi-domain fine-tuning, a team may want to improve code generation while avoiding losses in general knowledge, instruction following or a safety evaluation. The usual practice is to fix data proportions—uniform, dataset-size based or manually tuned—and check the result at the end. If the mix fails, training has to be repeated.

DynaMiCS turns that process into constrained optimization. For each training dataset, it runs a short isolated series of steps and measures how losses change across evaluation domains. Those measurements form a slope matrix: a local estimate of which transfer helps and which interference hurts. It uses that matrix to compute weights for the next stage of training, measures again later, and adjusts the mix.

The important distinction is that domains to be protected do not have to be data used for training. They can be safety evaluation suites, common knowledge or instruction-following tests. The method aims to keep their losses below a reference recorded before fine-tuning. If no combination can satisfy everything, it relaxes the constraint and chooses the option that violates it the least.

What the study measured

The authors evaluated the proposal in 50 fine-tuning scenarios with one to three target domains and three to ten constrained domains. They used Qwen2.5-3B, Qwen3-8B and Gemma3-12B, with LoRA adapters in the experiments, and report more than 7,000 fine-tuning runs. Targets include mathematical reasoning, code, instructions, medical questions, function calling and multilingual tasks; constraints include safety, commonsense reasoning and general knowledge.

In those experiments, DynaMiCS outperformed fixed mixtures tried at several weights. On Qwen3-8B, the manuscript reports an 11% median reduction in target-domain perplexity versus 0% for the compared fixed baselines. On Gemma3-12B, it reports 24% versus 18% for the best baselines. The additional cost comes from temporary probes and evaluations, but the authors place it well below running many full training jobs to search for a suitable mixture.

A constraint is not a guarantee

The result does not mean that a model automatically retains every important capability. DynaMiCS preserves what is measured and included as a constraint. An incomplete evaluation, a poorly chosen metric or a behaviour absent from the tests remains outside the optimization. The method also uses a local linear approximation: it assumes that dataset effects can be combined predictably enough over the short interval until the next update.

That is exactly why the work is useful. It forces teams to declare what they want to improve, what they are unwilling to degrade, and how each will be measured. Rather than treating the data mix as a fixed recipe, it makes it a decision that can be revisited throughout training. For people adapting models to specific tasks, that change in framing may matter as much as the improvement percentage in a table.