From Words to Vectors: How a Neural Network Represents Language

As of July 30, 2026, nearly every general-purpose language model accepts text, but no neural network operates directly on words. It operates on numbers. Between a sentence typed by a person and the next word predicted by a system lies a chain: tokenization, vector lookup, the addition of position and context, and a transformation into probabilities. Understanding that chain reveals what an embedding actually is, why two terms become neighbors, and why proximity is not the same as understanding.

The problem: a word is not a number

A program can assign a different identifier to every word, but identifiers are labels rather than quantities: making “cat” 20 and “dog” 21 does not make them more similar than “cat” and “airplane.” A one-hot representation avoids that false ordering with a vector containing one 1 and zeros everywhere else. It distinguishes vocabulary items, but all are equally separated and the vector grows with the dictionary.

An embedding replaces that sparse table with a dense one. Each token points to a learned row containing, for example, hundreds of coordinates. The coordinates are not pre-labelled “animal” or “plural.” They move during training as the model reduces prediction error. Bengio and colleagues’ 2003 paper formulated a neural probabilistic language model that jointly learned distributed word representations and sequence probabilities.

Distributed means the pattern matters

In a classic symbolic representation, a category may have an explicit slot. In a distributed representation, information is spread across many dimensions and each dimension participates in many concepts. Two vectors count as nearby under a geometric measure such as cosine similarity because training has pushed them toward similar uses.

Word2vec experiments showed that learning to predict a word from its context—or context from the word—produced useful relationships at lower cost than earlier neural models. GloVe began with global co-occurrence statistics and trained vector differences to reflect relationships derived from those counts. They use different methods, but both operationalize one intuition: contexts of use carry signals about similarity.

That is the distributional hypothesis in operational form. It does not say that words used in similar contexts are identical or that a model knows their real-world referents. “Doctor” and “nurse” may be near because of topic; “rise” and “fall” because they enter similar constructions. Geometry records regularities in a corpus and objective, not an infallible dictionary.

Before the vector comes the token

Modern systems do not always store one row per complete word. They split text into common units, characters, or fragments. This limits vocabulary size and lets novel terms be composed. Research on subword units for neural machine translation applied a form of byte-pair encoding to handle rare words through segments.

Segmentation has consequences. A frequent word may occupy one token; a technical term, surname, or underrepresented language may require several. That changes length, cost, and learning difficulty. Before comparing two models by a “100,000-token window,” inspect their tokenizers: the same paragraph does not necessarily consume the same budget.

A fixed vector cannot resolve polysemy

With static embeddings, “bank” receives the same vector in “river bank” and “central bank.” Context arrives later as the network processes the sequence, but the initial row stays fixed. Recurrent networks handled sequences through a state updated one word at a time. LSTM introduced gates and a memory path to mitigate difficulties in learning long dependencies, as described in the original 1997 paper. Even so, temporal computation limits parallelism.

ELMo marked an important step toward contextual representations: it generated each word vector as a function of the whole sentence using a bidirectional model. The two occurrences of “bank” could therefore diverge. A representation was no longer merely a dictionary entry; it became the state produced for that particular use.

Attention: each position gathers context

The original Transformer replaced recurrence in its core with self-attention. At each position it constructs queries, keys, and values; compatibility between a query and the keys determines how much each value contributes to the new representation. Multiple heads apply different projections, and successive layers transform the result again.

“Attending” does not mean looking consciously, nor does it guarantee that every head corresponds to a human linguistic rule. It is a differentiable operation for mixing information conditional on content. Its practical advantage was greater parallelism and shorter paths between distant positions. Its cost grows with the relationships considered, and an attention score alone is not a complete explanation of a model’s decision.

Order does not appear by itself

Basic self-attention would treat a sequence as a set without positional information. A common misconception needs correction here: the 2017 Transformer added sinusoidal or learned positional encodings to input embeddings; it did not introduce relative positions as a universal architectural feature. Those variants came in later work.

The distinction helps when reading architectures. An absolute encoding says where a token is; a relative one represents distances or relationships between positions; a rotary scheme changes queries and keys to incorporate position. All aim to prevent “the dog bit the man” from collapsing into “the man bit the dog,” but they have different extrapolation properties and costs.

BERT makes the contextual shift visible

BERT pretrained a Transformer encoder by masking tokens and learning to reconstruct them from context on both sides. The output associated with a word depends on the other positions and the layer inspected. The model could then be fine-tuned for a task with an additional output layer. This does not mean BERT contains a single internal definition for every term: it produces states useful for its training objective and the tasks on which it was evaluated.

An autoregressive generative model uses a different mask: while predicting the next token, each position can depend only on earlier ones. Both systems may share attention operations and embeddings, but objective and access to context change which representation is useful. “Transformer” names a family of components; it does not specify a task.

From a contextual state to a probability

After many layers, the state at the final position is projected into one score for every vocabulary token. A softmax function normalizes those scores into a distribution. Always choosing the maximum, sampling, or restricting candidates changes generated text, but not what the weights learned.

This final stage prevents a conceptual error: the model does not retrieve a stored sentence or consult a secret definition on every step. It computes probabilities conditioned on earlier tokens and learned parameters. It may produce a factual continuation or a plausible invention because both are possible sequences under that distribution.

The geometry also inherits the corpus

If the data contains social associations, vectors may reproduce them. Semantics derived automatically from language corpora contain human-like biases measured associations in embeddings and related them to established findings in psychology. Removing one geometric direction is not enough to solve every harm: context, task, population, and product decision all matter.

A nearest-neighbor chart is therefore a diagnostic, not a certificate of neutrality. It should be compared across languages, domains, and groups, then connected to errors in the actual application. An embedding can be useful for document retrieval and dangerous as a substitute for a decision about people.

How to explain it without a single acronym

Take one ambiguous word in two sentences and follow six steps: how it is tokenized; which initial vector it receives; how position is added; which context each layer mixes; which final state reaches the output; and which distribution it produces. Then ask which corpus and objective moved those numbers. If you can narrate that path, you can distinguish a static embedding from a contextual one and an architecture from its task.

The durable skill is this: when someone says a model “understands” a word, ask for the concrete representation, the context it includes, the training signal, and the validating test. A network turns language into useful geometry; deciding what that geometry means remains a matter of evidence.