Launching the Artificial Analysis Text to Image Leaderboard & Arena

In two short years since the advent of diffusion-based image generators, AI image models have achieved near-photographic quality. How do these models compare? Are the open-source alternatives on par with their proprietary counterparts?

The Artificial Analysis Text to Image Leaderboard aims to answer these questions with human preference based rankings. The ELO score is informed by over 45,000 human image preferences collected in the Artificial Analysis Image Arena. The leaderboard features the leading open-source and proprietary image models : the latest versions of Midjourney, OpenAI's DALL·E, Stable Diffusion, Playground and more.

Check-out the leaderboard here: https://huggingface.co/spaces/ArtificialAnalysis/Text-to-Image-Leaderboard

You can also take part in the Text to Image Arena, and get your personalized model ranking after 30 votes!

Methodology

Comparing the quality of image models has traditionally been even more challenging than evaluations in other AI modalities such as language models, in large part due to the inherent variability in people’s preferences for how images should look. Early objective metrics have given way to expensive human preference studies as image models approach very high accuracy. Our Image Arena represents a crowdsourcing approach to gathering human preference data at scale, enabling comparison between key models for the first time.

We calculate an ELO score for each model via a regression of all preferences, similar to Chatbot Arena. Participants are presented with a prompt and two images, and are asked select the image that best reflects the prompt. To ensure the evaluation reflects a wide-range of use-cases we generate >700 images for each model. Prompts span diverse styles and categories including human portraits, groups of people, animals, nature, art and more.

Early Insights From the Results 👀

How to contribute or get in touch

Other Image Model Quality Initiatives

Check out the following for other great initiatives: