Meta unveils LLaMA, efficient language models for research
Meta has unveiled LLaMA, a family of language models ranging from 7 billion to 65 billion parameters for research. Its bet is that a better-trained model can compete with much larger, more expensive systems.
Today, Meta unveiled LLaMA, a family of language models built for researchers that ranges from 7 billion to 65 billion parameters. The novelty is not just its size: Meta says it achieved competitive results against much larger models by training on more data and using a refined architecture.
LLaMA arrives at a time when large language models have become infrastructure reserved for a handful of companies. Systems such as OpenAI’s GPT-3 and Google’s PaLM require enormous computing resources, and their weights—the numerical values learned during training—are not freely distributed. Meta wants some of that research to be possible with more manageable models.
Four sizes, one bet
The family includes versions with 7 billion, 13 billion, 33 billion and 65 billion parameters. Parameters are the internal values a model adjusts as it learns language patterns; they do not directly equate to intelligence, but for years they have been the most visible measure of these systems’ scale.
Meta’s thesis is that this measure is not enough. The 7-billion-parameter model was trained on 1 trillion tokens, the units of text the system processes, while the remaining models used 1.4 trillion. That is notably more than the amount used to train many earlier models of a similar size.
According to Meta’s technical paper, the 13-billion-parameter version of LLaMA outperforms OpenAI’s 175-billion-parameter GPT-3 on most of the benchmarks compared by the authors. The 65-billion-parameter version delivers competitive results against DeepMind’s Chinchilla, which has 70 billion parameters, and Google’s PaLM, which reaches 540 billion.
These comparisons do not automatically make LLaMA a replacement for those products. Results depend on the benchmarks selected, and a language model can perform well on standardized tests without being reliable in real-world conversations or professional tasks. But they do challenge a widely held idea: that continually increasing the number of parameters is the main path to improvement.
More text before more scale
The work builds on a conclusion that had already begun to reshape the race to develop giant models: a system can fall short if it grows in parameters without receiving enough training data. Rather than focusing all its resources on making the model bigger, LLaMA seeks a more balanced relationship between its size and the volume of text it reads.
Its data comes primarily from Common Crawl, a massive collection of web pages. It also incorporates cleaned text datasets such as C4, GitHub code, Wikipedia, books, scientific papers from arXiv and Stack Exchange conversations. Meta says it removed duplicates and filtered out some low-quality content, a crucial step because models reproduce both the knowledge and the biases and errors found in their data.
The efficiency has an important practical consequence. Training a 65-billion-parameter model remains beyond the reach of most universities and companies, but studying, fine-tuning and running 7-billion- or 13-billion-parameter versions may be far more accessible than working with models containing hundreds of billions of parameters.
Access for researchers, not full openness
Meta will make LLaMA available to researchers by application and under a noncommercial license. The company is distributing the inference code—the stage when the model generates a response—and plans to provide the weights to institutions and individuals with a research justification.
This is therefore neither an unrestricted open release nor a consumer product comparable to ChatGPT. The restriction reflects a familiar concern: models capable of generating convincing text can also be used to create disinformation, impersonation campaigns or automated content at scale.
Even so, the release could shift the balance of power in research. If the results hold up outside Meta, labs with smaller budgets will be able to experiment with powerful models without relying entirely on the interfaces and decisions of major platforms. The next test will be seeing how well LLaMA performs when adapted to languages, specialized domains and specific uses not covered by the paper’s evaluations.
