π0 and π0-FAST: Vision-Language-Action Models for General Robot Control

We have ported the first robotics foundation models to Hugging Face LeRobot! Both π0 and π0-FAST, developed by Physical Intelligence, are now available in the LeRobot repository, bringing generalist robotic intelligence to the Hugging Face ecosystem. If you are curious about how Vision-Language-Action (VLA) models differ from Vision-Language Models (VLMs) and how actions are represented, dive into this blog post to find out!

Explore the model collection and the PyTorch Version of the model in our repository: Huggingface collection of Pi0 models | Huggingface collection of Pi0+FAST models | LeRobot repo

Introduction

Robert Heinlein suggests that a well-rounded person should be capable of handling a wide range of tasks—both intellectual and physical—rather than being narrowly specialized in one field. Drawing a parallel between a well-rounded person and machine intelligence: AI systems vary widely, but human intelligence excels in versatility—adapting to tasks, environments, and surprises. While large language and vision-language models (LLMs, VLMs) show promise, they lack interaction with the physical world. To bridge this gap, we need models trained on robotic data. Generalist robot models can enhance adaptability, using diverse data to improve generalization and robustness. Instead of training on isolated tasks, pre-training on varied robotic data—similar to LLMs—boosts efficiency and performance.

Developing generalist robot policies, or robot foundation models, presents three key challenges:

In this post, we introduce π0 and π0-FAST, prototype models and learning frameworks developed by Physical Intelligence, designed to overcome these challenges.

🔍 What is π0?

Paper | Jax Code

π0 is trained on data from 7 robotic platforms and 68 unique tasks, demonstrating strong zero-shot and fine-tuned performance on complex, real-world tasks such as laundry folding, table bussing, grocery bagging, box assembly, and object retrieval.

How to Use π0 in LeRobot?

pip install -e ".[pi0]"