Why Do We Need World Models When LLMs Exist? How World Models Will Impact Physical AI

Yifeng Yin, Co-founder and CEO of TEA Intelligent System.

During my tenure as a machine learning engineer and AI researcher at Hugging Face and now as the CEO of an AI startup, ​I have watched and participated in the evolution of large language models (LLMs) from being cute to being kind of scary.

Now, almost everyone knows how powerful LLMs are. But if LLMs seem so powerful​, why are influential computer scientists like Yann LeCun and Fei-Fei Li betting on world models? ​​

The Difference Between LLMs And World Models​

All technology is valued for the problems it can potentially solve. ​The main difference between LLMs and world models is that, fundamentally, they are designed to solve two different sets of problems.

To understand this, let’s get back to first principles.

LLMs input and output tokens. That is, if a problem can be described and solved by combinations of tokens, it can potentially be solved by LLMs. If an LLM exhausted its theoretical potential, it would output the optimal token combination for any input. Since many things can be represented by a combination of tokens, LLMs can potentially solve a lot of problems.

The case for world models is more interesting and less straightforward. As explained in LeCun’s paper and this blog post by Fei-Fei Li, world models are functions that take in an observation of the “environment” and an action to be taken. They then output, in the form of embeddings, a prediction of how the state of the “world” would be if we take that action. The embedding should contain all information of interest about the new state of the environment.

In short, world models do not generate sentences but a prediction of the state of the environment after a certain action possibly taken by some agents.

As with LLMs, if we exhaust all theoretical potentials of a world model, it will output a very accurate prediction of what will happen to the world given an observation of the world and the action to be taken.

To sum it up: