Tokenization and Preprocessing: What a Model Keeps and Loses
Cleaning is not deleting signals. A guide to preserving originals, choosing a tokenizer, and measuring cost, coverage, and loss by language.
Re-edited on July 30, 2026, this article corrects a dangerous recipe: preprocessing text does not mean lowercasing everything and deleting URLs, punctuation, common words, or spelling errors. Those signals may be the data. “US” and “us” do not mean the same thing; “not” may be frequent and decisive; a web address may reveal provenance; a misspelling can distinguish a real query from edited prose.
Tokenization does not simply mean “split words” either. A token is the unit a model receives and produces, selected through a particular vocabulary and algorithm. That choice determines cost, context length, and treatment of languages, code, numbers, and rare characters. Responsible practice preserves the original, makes every transformation explicit, and measures its effects before training.
First, preserve a layer that is not changed
A raw corpus needs provenance, date, permission, encoding, and a stable identifier. Transformations are applied to a copy and recorded with a version. This makes it possible to reproduce a dataset, explain why a document disappeared, and correct a rule without collecting everything again. “Clean” is not a universal property: it depends on the task, population, and risk.
Unicode normalization addresses the fact that one visible character can be represented by different sequences. Unicode Standard Annex #15 defines canonical and compatibility forms. They are not interchangeable: compatibility normalization can merge forms that the producer of the text distinguished. A team should select a form, test examples from affected languages and domains, and preserve the original string whenever the distinction may matter.
Lowercasing may help a classifier that is intended to ignore case, but it can harm entities, abbreviations, or code. Removing HTML can be reasonable when the target is visible text, yet useful structure, links, and metadata should be extracted first. Automatic spelling correction may erase dialect, age, intent, or the trace of an attack. Every rule needs a hypothesis: which error it reduces, which information it destroys, and how both effects will be tested.
A token is not necessarily a word
Splitting on spaces fails for languages that do not mark every boundary that way and for items such as emoji, contractions, compound names, addresses, and code. A vocabulary of complete words creates many unknown units. A character vocabulary covers almost any form but produces long sequences. Subword methods seek a compromise: retain frequent units and divide rare forms into reusable fragments.
In 2016, Sennrich, Haddow, and Birch adapted Byte Pair Encoding to word segmentation for neural translation. The technique starts with small units and merges frequent pairs until it reaches a set vocabulary size. It can therefore represent open vocabulary using a limited symbol inventory. It does not discover “true” morphemes: merges follow corpus frequencies and may segment the same root in different ways.
SentencePiece makes it possible to train subword models directly from sentences without relying on prior whitespace tokenization. It treats whitespace as an explicit symbol, supports both BPE and a unigram model, and facilitates reconstructing text from pieces. This does not mean that it ignores spaces or that context decides segmentation afresh in every sentence: it learns a tokenization model and then applies it in a defined way.
The algorithm’s name does not specify the tokenizer
Two tokenizers called BPE can differ in normalization, pretokenization, corpus, vocabulary size, reserved symbols, and treatment of bytes. BERT used WordPiece and added control tokens to mark the beginning, separation, and masked positions. The GPT‑2 report describes a byte-level BPE variant designed to combine byte-level coverage of inputs with frequent merges.
Byte coverage avoids depending on an unknown symbol for representable characters, but it does not guarantee linguistic efficiency. The same word can occupy very different numbers of tokens depending on language, script, spacing, or accents. Longer sequences consume context and compute, leaving less room for a user’s content. Unnatural splits in numbers, file paths, or code can also make operations that a person regards as elementary harder.
The tokenizer is part of the model, not an interchangeable accessory. Changing its vocabulary changes the identifiers entering the network and makes learned weights incompatible unless a migration is designed. A model record should therefore preserve the tokenizer files, their hashes, library version, normalizer, special tokens, and truncation and padding rules.
Special tokens deserve tests of their own. They mark functions such as beginning, end, padding, or separation of messages and must not be confused accidentally with user text. Documentation should state which visible string produces each token, how an input resembling it is escaped, and which component adds every marker. A tokenizer does not make an instruction safe: it only establishes the numerical boundary received by the model and the layers around it. Security behavior has to be tested at the complete system level.
Quality is measured by language, domain, and task
A basic indicator is fertility: how many tokens are needed to represent a word or comparable unit. Other useful measures include length per document, the share of isolated bytes or characters, fragmentation of names and numbers, truncation rate, and cost by language. Distributions matter more than one average because affected texts may be concentrated in a community or domain.
The study How Good is Your Tokenizer? compared multilingual and monolingual tokenizers and related tokenization quality to downstream performance. Its lesson is not that one segmentation is optimal for every language, but that sharing a vocabulary allocates capacity in a way that must be measured. Adding a language’s text to training does not guarantee that the vocabulary grants it efficient units.
Testing should include real difficult pairs: uppercase text, composed and decomposed Unicode, scripts without spaces, emoji with modifiers, mixed words, URLs, email addresses, decimals, dates, code, and control sequences. Teams verify round-trip behavior when reconstruction is promised, stability across machines, and behavior on invalid input. They then measure the complete task: fewer tokens are an improvement only if the relevant quality is maintained or increased.
The vocabulary should be learned only from the training split and then frozen for validation and testing. If it is readjusted after inspecting reserved data, the evaluation no longer simulates unknown input. For a new domain, the existing tokenizer and a candidate should be compared under the same budget: length, latency, memory, task quality, and compatibility. Reduced fragmentation does not justify a migration that breaks models, indexes, or caches without a testable plan.
From document to training batch, without leakage
After normalization and tokenization come decisions about filtering, duplication, and splitting. Duplicates can give one source disproportionate weight and make a test look easy when a copy entered training. Deduplicating Training Data Makes Language Models Better examines repetition, memorization, and overlap between training and evaluation, showing why deduplication must inspect sequences within documents rather than only identical files.
Training, validation, and test data should be separated before learning decisions that may inspect reserved content. Filtering thresholds, blocklists, and detectors are versioned; the reason for exclusion is retained; false positives are audited by language. A rule that removes “low-quality” text according to a classifier trained on one register can erase legitimate voices written in another.
The transferable skill is turning an opaque pipeline into a reproducible record: original and provenance; reversible transformations; exact tokenizer; measures by language and domain; and non-overlapping splits. The question is not whether data became “clean,” but which information was retained, which was lost, and what test shows that the loss helps the task without shifting the cost to other users.
This article was produced with artificial intelligence under human editorial oversight.
