LeRobot goes to driving school: World’s largest open-source self-driving dataset

TL;DR of L2D, the world's largest self-driving dataset!

• 90+ TeraBytes of multimodal data (5000+ hours of driving) from 30 cities in Germany
• 6x surrounding HD cameras and complete vehicle state: Speed/Heading/GPS/IMU
• Continuous: Gas/Brake/Steering and discrete actions: Gear/Turn Signals
• Environment state: Lane count, Road type (highway|residential), Road surface (asphalt, cobbled, sett), Max speed limit.
• Environment conditions: Precipitation, Conditions (Snow, Clear, Rain), Lighting (Dawn, Day, Dusk)
• Designed for training end-to-end models conditioned on natural language instructions or future waypoints
• Natural language instructions. F.ex "When the light turns green, drive over the tram tracks and then through the roundabout" for each episode
• Future waypoints snapped to OpenStreetMap graph, aditionally rendered in birds-eye-view
• Expert (driving instructors) and student (learner drivers) policies

State-of-the art Vision Language Models and Large Language Models are trained on open-source image-text corpora sourced from the internet, which spearheaded the recent acceleration of open-source AI. Despite these breakthroughs, the adoption of end-to-end AI within the robotics and automotive community remains low, primarily due to a lack of high quality, large scale multimodal datasets like OXE. To unlock the potential for robotics AI, Yaak teamed up with the LeRobot team at 🤗 and is excited to announce Learning to Drive (L2D) to the robotics AI community. L2D is the world’s largest multimodal dataset aimed at building an open-sourced spatial intelligence for the automotive domain with first class support for 🤗’s LeRobot training pipeline and models. Drawing inspiration from the best practices of source version control, Yaak also invites the AI community to search and discover novel episodes in our entire dataset (> 1 PetaBytes), and queue their collection for review to be merged into future release (R5+).

Expert policy — Driving instructor	Student policy — Learner driver