Tools and Techniques for Training Custom Language Models

Adapting a general language model to a specific domain — contracts, medical records, customer support in one industry — is now within reach of small teams. These are the techniques that make it possible, and one case that teaches something more useful than any of them: what to do when the success story everyone cites cannot be checked.

The starting point: don't train from scratch

Practically nobody trains a language model from the beginning. You start from a pre-trained one and adapt it, which is what transfer learning means: the model arrives knowing the general structure of the language, and you only have to teach it the specialism.

The reference infrastructure for this is the Transformers library, presented by the Hugging Face team at the 2020 EMNLP conference and maintained in an open repository with more than twenty-three thousand commits on its main branch.

One point worth keeping current: it is no longer a natural language processing library. Its own current description defines it as the model-definition framework for text, vision, audio and multimodal models, for both inference and training. Anyone who knows it only as «the NLP library» is two or three years behind.

The techniques, in order of cost

Fine-tuning. Training continues on domain data. It is the most effective and the most expensive, and it demands care: fine-tuning the whole model at once with a high learning rate overwrites the general knowledge — the phenomenon known as catastrophic forgetting. Hence the usual precautions of freezing layers and tuning gradually.

Specialized representations. Training the vectors representing words on domain text lets the model distinguish senses that general language blurs. «Charge», «provision» or «resolution» do not mean the same in a contract as in a newspaper.

Hyperparameter search. The values governing training — learning rate, batch size, number of epochs — change the outcome considerably. Exploring them at random works; doing it with Bayesian optimization, which decides where to try next from what it has already observed, usually gets to the same place sooner.

Multi-task learning. Training one model on several objectives at once, so what is learned in one helps the others. Its difficulty is not conceptual but a matter of balance: defining a loss function that stops one task from swallowing the rest.

Knowing what the model is looking at

A specialized model can be right for the wrong reason — learning that documents from a certain client are nearly always approved, rather than learning to assess them. Interpretability techniques exist to catch this: LIME, which approximates the model with a simple one in the neighbourhood of a specific prediction, and SHAP, which distributes responsibility for the prediction among the variables using game theory's Shapley values.

In a custom model these are not a luxury: they are how you check that the specialization learned what you meant to teach it.

The case everyone cites and nobody can check

Here comes the most useful part of this article, and it is not a technique.

When you look for an example of a custom model with real impact in banking, the same one always appears: LOXM, from J.P. Morgan, a machine learning system for optimizing order execution. It has circulated since 2017, and specific figures circulate with it: execution efficiency improvements, training on billions of historical trades.

In preparing this article we looked for the primary source of those figures. It does not exist publicly. There is no bank statement, no product page, no accessible technical document. The official electronic trading FAQ that J.P. Morgan publishes — dated November 2025 — does not mention LOXM. All that can be found is the 2017 press coverage and the texts repeating it.

That is why you will find none of those figures in this article. Not because we doubt the system exists, but because a figure living only in articles that cite one another is not data: it is a rumour with a good reputation.

And it is worth seeing what that means. The most cited success story of artificial intelligence applied to banking has gone almost a decade without its owner publishing a single checkable specification. That is valuable information in itself about how much of what this sector claims can be verified.

The evaluation error that ruins half these projects

One failure is so widespread it deserves its own section, and it is not technical but methodological. A model is specialized for a domain — contracts, records, case files — and then validated on a general test set.

The result is predictable and misleading. The specialized model scores much like the starting one on the generic exam, because that exam contains none of the cases that separate them, and the conclusion drawn is that specialization «added nothing». Or the reverse: it scores well on the general exam and is deployed into the domain, where it fails the moment real casuistry appears.

The rule is uncomfortable because it costs work: a specialized model requires a specialized evaluation set, built from domain cases and annotated by someone who knows the domain. There is no shortcut. If a project budgets nothing for building that set, it is not measuring what it thinks — it is measuring something else very precisely.

How to check a custom-model claim

The LOXM case leaves a method applicable to any announcement of this kind, and it requires no technical knowledge.

Ask about the evaluation set before the result. Whoever did the work properly can describe it: how many cases, where they came from, who annotated them. Whoever did not will change the subject to architecture.

Ask what it was compared against. A specialized model should be measured against the general model without specialization, and against how the work was done before. Without those two references, an accuracy figure says nothing about whether the project was worth it.

And ask who else can repeat the measurement. A figure only its publisher can reproduce is not a result: it is a testimonial. That is exactly the difference between the papers linked here — whose numbers anyone can check in the document — and the system we have been hearing about for almost a decade without a single public specification.

The capability: what to do when the source does not turn up

1. Look for the owner's document, not the news story. If a company has achieved something notable, it normally publishes it: a statement, a technical paper, a product page. If there is only press, the right question is why.

2. Distrust the figure that travels without a document. A percentage repeated across twenty sites all citing the same original article has one source, not twenty. Repetition is not corroboration.

3. Say that you could not check it. This is the part almost nobody does, and it is what sets you apart. Declaring the limit costs one sentence and saves years of propagating something nobody verified.

The deep end

• Wolf and others (2020), the Transformers paper in the EMNLP proceedings, open access.
• The repository, where the project's real state and current scope are visible.
• LIME and SHAP, to check what your specialized model learned.
• On LOXM there is no link to offer, and that absence is the data point.