Visual Salamandra: Pushing the Boundaries of Multimodal Understanding

The Language Technologies Lab takes a major step forward in multimodal artificial intelligence with the release of Visual Salamandra, extending the capabilities of the Salamandra large language model (LLM) to both images and video. Visual Salamandra is based on the 7 billion parameters foundational model maintaining its compactness and efficiency while extending it to multimodal tasks.

Designed with vision-language alignment at its core, Visual Salamandra builds on top of the Salamandra Instructed 7B model by integrating Google’s SigLIP encoder (SigLIP-So400m), a 2-layer MLP projector, and advanced late-fusion techniques to bridge the gap between visual and textual modalities.

The resulting architecture enables Visual Salamandra to comprehend and generate contextually accurate responses from diverse inputs, ranging from single and multiple images and videos to purely textual instructions. This development reflects a broader commitment by the Lab to support robust, multilingual, and multimodal AI systems—especially those that prioritize European linguistic diversity.

Training Visual Salamandra: A Deep Dive into Vision Experiments

To adapt Salamandra for visual inputs, the Lab implemented a four-phase training process centered on late-fusion architecture. In this setup, a pre-trained image encoder (SigLIP, 14 patches at 384x384 resolution) generates image embeddings, which are then aligned with the LLM via a custom-trained multilayer perceptron (MLP) projector.

The four training phases include:

• Phase 1: Projector Pre-training – Only the projector is trained to map image features into the LLM’s latent space.

• Phase 2: High-Quality Vision Pretraining – Using refined datasets (e.g., OCR and re-captioned images), the entire architecture (encoder, projector, and LLM) undergoes joint training.

• Phase 3: Instruction Tuning – The model learns to follow user instructions via visual question answering (VQA), OCR, and other grounded vision tasks.