Transfer learning: when to reuse a model—and when not to

As of 30 July 2026, much applied AI does not begin with an empty model. It starts from a network trained to recognise images, represent language or follow instructions, then adapts it with data from another task. This pattern has reduced costs and enabled projects that could not assemble the compute or data required to train from scratch. Its name, transfer learning, can mislead: it does not move “knowledge” as if copying a file.

What is reused is a set of parameters and representations learned under an objective, a dataset and design choices. The new task wagers that some of that structure remains useful. Sometimes it does; sometimes it causes negative transfer: the starting point restricts learning, carries errors or performs worse than a simpler alternative. The important skill is not knowing that pretrained models exist, but deciding when reuse is justified.

Transfer is a hypothesis

During pretraining, a model adjusts parameters to reduce an error. In vision, it may learn edges, textures and shapes; in language, regularities useful for predicting or reconstructing text. Those representations are not a neutral database of the world. They reflect the task, corpus, labels, language and omissions of training.

How transferable are features in deep neural networks? showed in vision networks that early layers tended to learn more general features while later ones became more specific to the original task. It also found that transferability declined as the distance between tasks increased. This is evidence from specific architectures and datasets, not a law assigning a fixed meaning to every layer. It offers the right intuition: the greater the distance from source to target, the less the advantage should be assumed.

BERT popularised the same pattern in language processing. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding pretrained representations on unlabelled text and then fine-tuned the model, with an output layer, for tasks such as question answering and inference. The advance did not show that one representation is universal; it showed that this pretraining provided an effective starting point across eleven evaluations reported by the authors.

The first decision is measuring distance

Before selecting a model, compare source and target along five dimensions:

• Modality: text, image, audio, video, time series or a combination.
• Domain: general web, radiology, contracts, crops, conversation or industrial sensors.
• Task: classification, localisation, retrieval, prediction, generation or control.
• Population: languages, regions, devices, groups and capture conditions.
• Cost of error: a reversible suggestion, an administrative decision or a physical action.

A model of everyday photographs may help identify industrial defects because it shares basic visual patterns. It may fail when the useful signal appears at another scale or through another sensor. A general language model may summarise clinical notes fluently while missing local abbreviations or turning medical uncertainty into a categorical sentence. Superficial similarity of format does not guarantee similarity of problem.

Four adaptation strategies, four trade-offs

Use the representation without changing it

The base is frozen and a small layer is trained for the task. This is inexpensive, preserves original behaviour and tests whether the representations already separate useful classes. Its limit is clear: if the relevant information is absent, a new head cannot create it.

Update every parameter

Full fine-tuning allows the whole network to adapt to the domain. It needs more memory, data and training control. With few examples it may overfit; substantial changes can erase prior abilities or alter safety behaviour. “More trainable parameters” does not automatically mean “better transfer.”

Update a fraction

Adapters and other parameter-efficient methods change a small part. LoRA, for example, freezes pretrained weights and introduces trainable low-rank matrices into model layers. It reduces the number of parameters that must be updated and stored, making it easier to maintain several adaptations over one base. It does not remove the dependency: an adapter remains tied to a particular version and inherits abilities and risks from the frozen model.

Keep weights fixed and supply context

Some systems use a model without retraining it, adding instructions, examples or retrieved documents. This may meet the need at lower cost and update more easily. It is not equivalent to fine-tuning: context consumes space, can be ignored and does not stably change representations. It is an alternative to compare, not guaranteed transfer.

Savings exist only when total cost is measured

Reusing a model avoids some initial training, but adds download, storage, inference, adaptation, evaluation, security and maintenance. A huge frozen model may cost more to operate than a small model trained for a narrow task. The comparison should include engineering time, energy, hardware, latency, cost per query and update frequency.

Licensing matters too. Available weights do not imply unrestricted use; conditions may govern redistribution, commercial use or attribution. If data provenance is undocumented, the adopting organisation receives uncertainty that fine-tuning does not repair.

Transfer also transfers failures

The base model may contain biases, vulnerabilities, false associations, unwanted memorisation or unequal language coverage. Fine-tuning on a small dataset may hide a failure in the average metric without removing it for subgroups. It may also inherit a backdoor in third-party weights or introduce one through adaptation data.

Traceability and known limits are therefore essential. Model Cards for Model Reporting proposes documenting intended use, relevant factors, metrics, evaluation data and disaggregated results. A card does not guarantee quality, but it shows which claims accompany a model and which ones must be retested at the destination.

The most important rule is that an evaluation cannot be inherited. Strong benchmark results for the base do not establish the adapted system in its new setting. Changing the population, sensor, language, interface or authority given to a model changes the object that must be evaluated.

The minimum experiment that reveals transfer

A defensible decision compares at least three starting points:

• A simple reference: a rule, small model or training from scratch that reveals what transfer adds.
• The frozen base: tests whether the representation already contains useful signal.
• The selected adaptation: partial or full fine-tuning under the same test set.

Training, validation and test data should be separated by the real unit. If images from the same patient, versions of the same document or samples from the same device appear on both sides, the result may measure memory rather than generalisation. The report should include variation, not just the best run, and subgroup results where impact is uneven.

Two controls should follow: regression tests for abilities that must remain, and stress tests for the target setting. If an assistant is specialised for customer service, test the new task alongside safety, languages, recovery from ambiguous instructions and behaviour when data is missing.

From experiment to maintenance

Transfer does not end at deployment. Data changes, new base versions arrive and an adapter may become incompatible. The core of NIST’s AI Risk Management Framework calls for identifying data and evaluation suitability, validating in deployment contexts and monitoring pretrained models as part of regular maintenance. It is a voluntary framework, but it captures a sensible technical duty: treat the base as a living dependency.

A transfer record should retain the version and hash of the weights, licence, adaptation data, configuration, tests, owner and withdrawal criteria. Without them, a later base-model upgrade may break reproducibility or silently change the outcome.

The decision card

Before adapting a pretrained model, answer:

• Source: under which objective, data, population and licence was it trained?
• Distance: what does it genuinely share with the target task?
• Method: what remains frozen, what changes and why?
• Reference: does it beat a simple alternative under the same test?
• Inheritance: which failures, limits and obligations travel with the weights?
• Maintenance: how will drift, incompatibility and regressions be detected?

The transferable skill is this: treat a pretrained model as a hypothesis about useful representations, not universal knowledge, and require a comparison that measures gains, costs and inherited failures. Transfer saves resources when source and target fit. When they do not, an apparent head start can be the most expensive way to be wrong.