Transfer Learning

Transfer learning is the machine learning technique of reusing knowledge a model has acquired on one task or source domain to solve a different but related task, the target domain, instead of training from scratch. The survey by Sinno Jialin Pan and Qiang Yang, published in IEEE Transactions on Knowledge and Data Engineering in 2010, did not originate the idea: it traces modern research to the 1995 Learning to Learn workshop; its contribution was to consolidate the field and give a unified definition using source and target domains and tasks.

The motivation is practical: labeled data is expensive and scarce in many real-world problems, while representations learned on data-rich tasks often carry over.

How it is practiced: pretrain, then fine-tune

The dominant pattern is pretraining followed by fine-tuning. In computer vision, convolutional networks pretrained on ImageNet became the standard starting point for new image tasks; in language, BERT and GPT cemented the same recipe, with a model pretrained on massive unlabeled text later adapted to classification, translation or question answering. A lighter alternative is feature extraction, which freezes the pretrained layers and trains only a small classifier on top.

The foundation-model era and PEFT

The 2021 Stanford report On the Opportunities and Risks of Foundation Models, led by Rishi Bommasani, calls a model a foundation model when it is trained on broad data —generally through self-supervision at scale— and can be adapted to many downstream tasks. It presents the paradigm as an evolution based on deep learning and transfer learning whose scale brings new capabilities, and warns that defects in the base model pass to its adapted derivatives.

Fine-tuning every weight of a model with tens or hundreds of billions of parameters is prohibitive, which drove the rise of parameter-efficient fine-tuning (PEFT). LoRA (Hu et al., 2021) freezes the pretrained weights and injects trainable low-rank matrices: on the 175-billion-parameter GPT-3 it cut trainable parameters by roughly 10,000 times and GPU memory to a third, while matching or beating full fine-tuning. The adapters proposed by Houlsby et al. (2019) add task-specific modules while keeping the original network fixed; on GLUE, the published PDF reports 3.6% additional parameters per task and performance within 0.4% of full fine-tuning. Prompt tuning by Lester et al. (2021) learns soft prompts to condition a frozen model and, in its T5 experiments, matches full model tuning once scale exceeds billions of parameters.

When it works — and when it does not

The gains are not universal: they depend on how similar source and target are, on how much data the final task has, and on the model's capacity. Negative transfer occurs when using source-domain data or knowledge reduces performance in the target domain. The survey by Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu, published in February 2023 in the IEEE/CAA Journal of Automatica Sinica, explains that it can arise when tasks are unrelated, distributions differ too much, or the algorithm is unsuitable, and reviews about fifty strategies spanning secure transfer, domain-similarity estimation, distant transfer, and mitigation. Predicting in advance how much a given transfer will help remains an open problem.