PaliGemma 2 Mix - New Instruction Vision Language Models by Google

TL;DR

Last December, Google released PaliGemma 2: a new family of pre-trained (pt) PaliGemma vision language models (VLMs) based on SigLIP and Gemma 2. The models come in three different sizes (3B, 10B, 28B) and three different resolutions (224x224, 448x448, 896x896).

Today, Google is releasing PaliGemma 2 mix: fine-tuned on a mix of vision language tasks, including OCR, long and short captioning and more.

PaliGemma 2 pretrained (pt) variants are great vision language models to transfer on a given task at hand. All pt checkpoints are meant to be fine-tuned on a downstream task and were released for that purpose.

The mix models give a quick idea of the performance one would get when fine-tuning the pre-trained checkpoints on a downstream task. The main purpose of the PaliGemma model family is to provide pretrained models that can learn better on a downstream task, instead of providing a versatile chat model. Mix models give a good signal of how pt models perform when fine-tuned on a mix of academic datasets.

You can read more about PaliGemma 2 in this blog post.

You can find all the mix models and the demo in this collection.

Parameter Count	Framework	Resolution
3B	HF Transformers	224
448
JAX	224
448
10B	HF Transformers	224
448
JAX	224
448
28B	HF Transformers	224
448
JAX	224
448

Table of Contents

• PaliGemma 2 Mix Models
• Comparing PaliGemma 2 Mix Variants
• Inference and Fine-tuning using Transformers
• Demo

PaliGemma 2 Mix Models

PaliGemma 2 mix models can accomplish a variety of tasks. We can categorize them according to their subtasks as follows.

Note that this list of subtasks is non-exhaustive, and you can get more information on the full list of tasks in the PaliGemma 2 paper.