The emerging skills of advanced linguistic models: an illusion

In the bustling arena of artificial intelligence (AI), the race to create ever more efficient linguistic models never ceases to surprise. An ambitious project called Beyond the Imitation Game benchmark (BIG-bench) focused on this race, involving 450 researchers in the development of 204 tasks intended to push the limits of large language models. These models, which bring chatbots like ChatGPT to life, have shown variable performance, oscillating between predictable improvements and sudden leaps forward. A phenomenon that researchers have described as “revolutionary” behavior, reminiscent of phase transitions observed in physics.

**Large language models (LLM)**, such as the famous GPT-2, GPT-3.5, and the very recent GPT-4, have demonstrated an ability to process and understand enormous quantities of text by establishing connections between words. The ability of these models to accomplish complex, even unexpected, tasks lies in their number of parameters, essentially the various ways in which words can be interconnected. GPT-3.5, for example, uses 350 billion parameters, while the new arrival GPT-4 has a whopping 1.75 trillion parameters.

The increase in performance with model size seems logical, but some behavior defied expectations. We observed almost zero performance levels for a time, followed by a spectacular improvement, a phenomenon that intrigued the scientific community. Some researchers have seen in these “jumps” in capacity signs of emergence, these collective behaviors that arise in a system reaching a high level of complexity.

However, a team from Stanford University offers a different vision of these phenomena. According to them, the apparent unpredictability of these leaps in capacity is less a question of sudden emergence than of the way in which performance is measured. Sanmi Koyejo, lead author of a study on the subject, argues that so-called « phase transitions » in LLM abilities may be much more predictable than many believe, attributing the confusion to measurement methodology rather than to the true capabilities of the models.

This vision contrasts with the idea of ​​fluid, linear progress in AI. It suggests that our understanding of qualitative leaps in LLM capabilities depends closely on how we choose to evaluate and understand them. While major language models continue to advance, offering impressive improvements in efficiency and performance, the interpretation of these advances remains a matter of debate. The Stanford team’s findings challenge the notion of emergence as a mirage, offering a revolutionary perspective on how we perceive progress in the ever-evolving field of AI.