Intel launches Gaudi 3 to challenge Nvidia in AI market

On April 9, 2024, Intel introduced Gaudi 3 as an accelerator for training and inference of generative models. Intel’s technical release publishes memory, networking, availability and comparisons with NVIDIA H100; these are tests selected and run by the Gaudi vendor, not a universal advantage for every model.

The announcement comes as Nvidia’s H100 has become a scarce and expensive component for companies, research labs and cloud service providers. It is not just a question of which chip is the most powerful: availability, power consumption and the price of complete systems have become decisive factors in putting generative AI into production.

More memory and integrated networking for large models

Gaudi 3 follows Gaudi 2 and is designed for two distinct tasks. The first is training, the process of tuning a model with huge collections of data. The second is inference—using an already-trained model to answer questions, generate text or summarize documents.

Intel has equipped the chip with 128 GB of high-speed HBM2e memory and integrated Ethernet connections running at 200 gigabits per second. That integration is one of the defining features of the Gaudi family: rather than relying on a proprietary network to connect large numbers of accelerators, Intel is betting on Ethernet, the standard that already dominates data centers. Primary source.

The idea has a practical consequence: building AI servers and server clusters could require fewer specialized components. That does not eliminate the complexity of deploying thousands of chips, but it could make it easier for manufacturers and customers to use networking equipment they already know.

Intel promises advantages over the H100

Intel’s published figures are based on tests using Llama 2, the family of language models released by Meta. In a configuration with 64 accelerators, the company says Gaudi 3 can train the 70-billion-parameter version of Llama 2 up to 40% faster than Nvidia’s H100. Primary source.

For inference, Intel puts the advantage at between 1.5 and 2 times that of the H100, depending on the model size and the configuration tested. These results should be viewed with caution: vendor benchmarks use specific models, numerical precision, software and numbers of simultaneous users. A strong result on Llama 2 does not guarantee the same difference across all models or every workload at a company. Primary source.

The most aggressive selling point is price. Intel estimates that a system with eight Gaudi 3 accelerators will cost around $125,000, compared with roughly $300,000 for a comparable configuration with eight H100s. That is the company’s own estimate, and the final cost will depend on the server manufacturer, networking, storage and support contracts. Still, it illustrates the strategy: compete where Nvidia has left a wide margin amid strong demand. Primary source.

The challenge is not just making chips

Gaudi 3 will be available to equipment manufacturers during the second quarter of 2024. Dell, Hewlett Packard Enterprise, Lenovo and Supermicro are among the partners preparing servers with the new accelerator.

Intel must nevertheless solve a problem that cannot be measured in teraflops alone. Nvidia has spent years building a software ecosystem around CUDA, its GPU programming platform. Many tools, libraries and engineering teams are already designed for that environment. Switching suppliers requires companies to adapt workflows, validate models and train staff.

Intel offers its SynapseAI software stack and has worked to make Gaudi compatible with widely used frameworks such as PyTorch. But adoption will depend on whether its cost and performance claims hold up outside the company’s labs and with the models customers actually use.

For the market, Gaudi 3 matters even before it proves an outright technical victory. The industry needs more accelerator supply to ease generative AI bottlenecks. If Intel can deliver systems at scale and maintain its price advantage, it will give companies and cloud providers negotiating leverage they currently have little of against Nvidia.

Evaluate the accelerator as a system

Memory, compute and network must balance. Training distributes a model and synchronises updates; inference moves weights and caches while users arrive. Ethernet may ease integration with familiar networks, but performance depends on topology, software and congestion. The chip sheet does not replace cluster design.

A comparison fixes model, revision, batch, sequence, precision, accelerator count and objective. For training, measure time to agreed quality rather than samples per second alone. For inference, record percentile latency, throughput and energy under load. An average may hide excessive waits for some users.

Server price is not total cost

Include network, storage, power, cooling, licences, people and availability. Porting and optimisation work count too. A cheaper system may lose its advantage after months of adaptation; an expensive one may be unjustifiable when a smaller task fits lesser hardware.

Test portability before buying. Run the same model with supported libraries, inspect incompatible operators and preserve a reference version. Vendor benchmarks indicate what to test; the real workload decides. An enterprise gains negotiating power when it can move work with evidence, not merely when another chip exists.

Promised availability needs delivery and support dates. A competitive slide does not relieve a bottleneck if hardware does not ship at volume. Distinguish samples, partner servers and operational capacity, and record the contracts supporting each stage.

The transferable skill is to turn an accelerator comparison into cost per accepted task under equal conditions. That unit joins hardware, software and operation and avoids crowning a chip through a figure chosen by its vendor.

Precision changes the race

Accelerators support different numerical formats. Fewer bits may increase throughput and reduce memory, but model quality must hold. A comparison using different precision for each rival may be legitimate if it represents the best deployment, but it should declare the criterion and verify both reach the same threshold.

For generative inference, separate prompt processing from token-by-token generation. A platform may be fast at one and slow at the other. Applications with long documents and short answers have a different profile from chat. Measuring both phases prevents one figure being extrapolated to every use.

Express energy per correct task rather than maximum power alone. Measure at the system plug under stable load and include network and CPU. If the model needs retries, those cycles count. Efficiency without quality may cheapen answers nobody should use.

The buying process preserves results and scripts so another supplier can repeat them. An in-house benchmark should not become an informal secret controlled by a vendor. Reproducibility turns a second offer into real competition.

The team also tests node failure and job recovery. Peak throughput matters less when an incident restarts hours of training. Reliability, observability and support belong to the accelerator platform even when absent from the chip name.

The conclusion identifies uncertainty: pending in-house tests, contracted price and actual delivery date. Naming it does not weaken analysis; it defines work remaining before infrastructure is committed.