The State of Computer Vision at Hugging Face 🤗

At Hugging Face, we pride ourselves on democratizing the field of artificial intelligence together with the community. As a part of that mission, we began focusing our efforts on computer vision over the last year. What started as a PR for having Vision Transformers (ViT) in 🤗 Transformers has now grown into something much bigger – 8 core vision tasks, over 3000 models, and over 100 datasets on the Hugging Face Hub.

A lot of exciting things have happened since ViTs joined the Hub. In this blog post, we’ll summarize what went down and what’s coming to support the continuous progress of Computer Vision from the 🤗 ecosystem.

Here is a list of things we’ll cover:

• Supported vision tasks and Pipelines
• Training your own vision models
• Integration with timm
• Diffusers
• Support for third-party libraries
• Deployment
• and much more!

Enabling the community: One task at a time 👁

The Hugging Face Hub is home to over 100,000 public models for different tasks such as next-word prediction, mask filling, token classification, sequence classification, and so on. As of today, we support 8 core vision tasks providing many model checkpoints:

• Image classification
• Image segmentation
• (Zero-shot) object detection
• Video classification
• Depth estimation
• Image-to-image synthesis
• Unconditional image generation
• Zero-shot image classification

Each of these tasks comes with at least 10 model checkpoints on the Hub for you to explore. Furthermore, we support tasks that lie at the intersection of vision and language such as:

• Image-to-text (image captioning, OCR)
• Text-to-image
• Document question-answering
• Visual question-answering

These tasks entail not only state-of-the-art Transformer-based architectures such as ViT, Swin, DETR but also pure convolutional architectures like ConvNeXt, ResNet, RegNet, and more! Architectures like ResNets are still very much relevant for a myriad of industrial use cases and hence the support of these non-Transformer architectures in 🤗 Transformers.