nanoVLM: The simplest repository to train your VLM in pure PyTorch

nanoVLM is the simplest way to get started with training your very own Vision Language Model (VLM) using pure PyTorch. It is lightweight toolkit which allows you to launch a VLM training on a free tier colab notebook.

We were inspired by Andrej Karpathy’s nanoGPT, and provide a similar project for the vision domain.

At its heart, nanoVLM is a toolkit that helps you build and train a model that can understand both images and text, and then generate text based on that. The beauty of nanoVLM lies in its simplicity. The entire codebase is intentionally kept minimal and readable, making it perfect for beginners or anyone who wants to peek under the hood of VLMs without getting overwhelmed.

In this blog post, we cover the core ideas behind the project and provide a simple way to interact with the repository. We not only go into the details of the project but also encapsulate all of it so that you can quickly get started.

TL;DR

You can start training a Vision Language Model using our nanoVLM toolkit by following these steps:

# Clone the repo
git clone https://github.com/huggingface/nanoVLM.git

# Execute the training script
python train.py

Here is a Colab notebook that will help you launch a training run with no local setup required!

What is a Vision Language Model?

As the name suggests, a Vision Language Model (VLM) is a multi-modal model that processes two modalities: vision and text. These models typically take images and/or text as input and generate text as output.

If you are interested in learning more about VLMs, we strongly recommend reading our latest blog on the topic: Vision Language Models (Better, Faster, Stronger)