New ViT and ALIGN Models From Kakao Brain

Kakao Brain and Hugging Face are excited to release a new open-source image-text dataset COYO of 700 million pairs and two new visual language models trained on it, ViT and ALIGN. This is the first time ever the ALIGN model is made public for free and open-source use and the first release of ViT and ALIGN models that come with the train dataset.

Kakao Brain’s ViT and ALIGN models follow the same architecture and hyperparameters as provided in the original respective Google models but are trained on the open source COYO dataset. Google’s ViT and ALIGN models, while trained on huge datasets (ViT trained on 300 million images and ALIGN trained on 1.8 billion image-text pairs respectively), cannot be replicated because the datasets are not public. This contribution is particularly valuable to researchers who want to reproduce visual language modeling with access to the data as well. More detailed information on the Kakao ViT and ALIGN models can be found here.

This blog will introduce the new COYO dataset, Kakao Brain's ViT and ALIGN models, and how to use them! Here are the main takeaways:

• First open-source ALIGN model ever!
• First open ViT and ALIGN models that have been trained on an open-source dataset COYO
• Kakao Brain's ViT and ALIGN models perform on par with the Google versions
• ViT and ALIGN demos are available on HF! You can play with the ViT and ALIGN demos online with image samples of your own choice!

Performance Comparison

COYO DATASET

How ViT and ALIGN work

So what do these models do? Let's briefly discuss how the ViT and ALIGN models work.

A Visualization of How ViT Works from Google Blog ALIGN Model from Google Blog

How to use the COYO dataset

>>> from datasets import load_dataset

>>> dataset = load_dataset('kakaobrain/coyo-700m')
>>> dataset

How to use ViT and ALIGN from the Hub

import requests
from PIL import Image
import torch
from transformers import ViTImageProcessor, ViTForImageClassification