SmolVLM2: Bringing Video Understanding to Every Device

TL;DR: SmolVLM can now watch 📺 with even better visual understanding

SmolVLM2 represents a fundamental shift in how we think about video understanding - moving from massive models that require substantial computing resources to efficient models that can run anywhere. Our goal is simple: make video understanding accessible across all devices and use cases, from phones to servers.

We are releasing models in three sizes (2.2B, 500M and 256M), MLX ready (Python and Swift APIs) from day zero. We've made all models and demos available in this collection.

Want to try SmolVLM2 right away? Check out our interactive chat interface where you can test visual and video understanding capabilities of SmolVLM2 2.2B through a simple, intuitive interface.

Table of Contents

• SmolVLM2: Bringing Video Understanding to Every Device
  • TL;DR: SmolVLM can now watch 📺 with even better visual understanding
  • Table of Contents
  • Technical Details
    • SmolVLM2 2.2B: Our New Star Player for Vision and Video
    • Going Even Smaller: Meet the 500M and 256M Video Models
    • Suite of SmolVLM2 Demo applications
      • iPhone Video Understanding
      • VLC media player integration
      • Video Highlight Generator
  • Using SmolVLM2 with Transformers and MLX
    • Transformers
      • Video Inference
      • Multiple Image Inference
    • Inference with MLX
      • Swift MLX
    • Fine-tuning SmolVLM2
  • Citation Information

Technical Details

We are introducing three new models with 256M, 500M and 2.2B parameters. The 2.2B model is the go-to choice for vision and video tasks, while the 500M and 256M models represent the smallest video language models ever released.

While they're small in size, they outperform any existing models per memory consumption. Looking at Video-MME (the go-to scientific benchmark in video), SmolVLM2 joins frontier model families on the 2B range and we lead the pack in the even smaller space.

SmolVLM2 2.2B: Our New Star Player for Vision and Video

It’s so memory efficient, that you can run it even in a free Google Colab.

Python Code