OpenAI upgrades GPT-4 Turbo and cuts API prices with new embeddings

On January 25, 2024, OpenAI updated its developer catalogue: it released a new GPT-4 Turbo preview, announced a GPT-3.5 Turbo price cut and introduced the text-embedding-3-small and text-embedding-3-large models. The changes could lower both text-generation and semantic-search costs, but a price table or benchmark average is not enough to justify a migration.

The transferable lesson is to separate three layers that are often conflated. A generative model writes an answer; an embedding model represents queries and documents as vectors; retrieval infrastructure selects the passages supplied to the generator. Improving one layer does not automatically repair the others. In a document assistant, a fluent response can still rest on the wrong paragraph when retrieval fails.

What actually changed

OpenAI’s original announcement positioned text-embedding-3-small as an economical successor to text-embedding-ada-002. It cost $0.00002 per thousand tokens —$0.02 per million— compared with $0.0001 for the older model. text-embedding-3-large, with as many as 3,072 dimensions, cost $0.00013 per thousand tokens, or $0.13 per million.

OpenAI also reported higher averages on two evaluation collections. On MTEB, ada-002, the new small model and the large model scored 61.0, 62.3 and 64.6 respectively; their MIRACL multilingual-retrieval averages were 31.4, 44.0 and 54.9. These are vendor-reported results on public suites and support a hypothesis: the new models may retrieve better, particularly across languages. They do not establish that the right item will be found in a particular legal, medical or commercial archive.

The benchmark’s own design explains why. The paper that introduced MTEB assembled 58 datasets, 112 languages and eight task types, including retrieval, clustering and similarity. Its authors found that no method dominated every task. An average combines different problems; it can conceal that a model strong at classification is merely ordinary on the search task a product needs.

An embedding is not an answer

An embedding turns text into a sequence of numbers. Distances between vectors estimate which passages relate to a query even when they use different words. In retrieval-augmented generation, or RAG, that calculation first selects documents; a model such as GPT-4 then writes from the retrieved passages.

This separation makes failures diagnosable. If the correct document never appears among the results, the problem precedes generation: the text may have been split poorly, metadata may be missing, the query may need rewriting or the embedding may represent the domain badly. If the correct document appears but the answer contradicts it, inspect the prompt, supplied context and generator. Replacing GPT-4 will not fix a poor index; replacing embeddings will not prevent a generator from inventing.

Dimensions introduce another choice. Longer vectors consume more memory and storage and can make search more expensive. The new models allowed developers to request fewer dimensions. OpenAI’s example shortened text-embedding-3-large from 3,072 to 1,024 while accepting a trade-off between size and accuracy. The underlying idea is Matryoshka Representation Learning, which arranges information from coarse to fine so shorter prefixes remain useful. “Can be shortened” does not mean “free to shorten”: the loss must be measured on the actual collection.

A sound migration needs two indexes

Vectors produced by different models should not be mixed as though they occupied the same space. A query encoded by the new model is not reliably comparable with documents encoded by the old one. Migration requires recalculating embeddings for the full corpus and building a separate index. The old system remains available until the new one passes evaluation and can safely take traffic.

Running both avoids a blind switch. Send a sample of real queries to each index and retain rankings, latency and cost. For every question, define in advance which document or passage counts as relevant. The basic measure is recall near the top: for example, how often at least one correct source appears among the first five results. Measure hard groups separately as well: abbreviations, names, negation, cross-language questions and near-duplicate documents.

The sample should not contain only questions the system already handles. Include known failures, common searches and mistakes with high consequences. An internal-policy search tool may tolerate a cafeteria-menu query that needs rewording; it should not confuse the current leave policy with a superseded version. Date, jurisdiction and status metadata can matter as much as vector similarity.

API price is not total cost

The per-token reduction affects one part of the bill. An embedding migration incurs a one-time cost to re-index every document, followed by charges for new documents and queries. Vector databases charge for storage, memory or operations, and additional dimensions can increase those costs. The generator, volume of retrieved context, monitoring and human review of errors also belong in the calculation.

A useful decision sheet records five quantities: tokens to index, monthly queries, dimensions per vector, estimated storage and the cost of the writing model. Add two non-monetary measures: tail latency and correct-retrieval rate. The small model may win when it preserves quality at high volume; the large model may pay for itself in a modest archive where every missed document matters. There is no universal winner outside a use case.

The announced price for gpt-3.5-turbo-0125 put input at $0.50 per million tokens, 50% lower, and output at $1.50, 25% lower. OpenAI also said it fixed an encoding issue affecting non-English function calls. Those are reasons to test the model on structured work, not to replace a capable model without measuring accuracy, formatting and the cost of repairing errors.

Versions and aliases are part of the system

The new gpt-4-0125-preview was intended to complete tasks such as code generation more thoroughly and reduce what OpenAI called “laziness,” where work was left unfinished. This was a qualitative vendor claim, not a success rate. To test it, retain a suite of assignments with explicit completion criteria and compare outputs without simultaneously changing prompts, tools and temperature.

OpenAI also offered aliases that pointed to the latest preview. This convenience permits automatic changes. A reproducible experiment or regulated workflow should pin a particular version, record the date, parameters and test set, and upgrade deliberately. A moving alias can be reasonable for prototypes where early improvements matter. The choice is operational, not a contest over which model name sounds newer.

The January 25 update combined lower prices, new models and better key and usage controls. Its useful reading is not “more points for less money,” but an invitation to measure the complete chain. A team that builds a parallel index, labels real queries and compares retrieval, latency and cost can know whether the offer improves its product. A team that only replaces a model name has changed a dependency without demonstrating an improvement.