ScreenSuite - The most comprehensive evaluation suite for GUI Agents!

Releasing ScreenSuite, the most comprehensive evaluation suite for GUI Agents!

TL;DR

Over the past few weeks, we’ve been working tirelessly on making GUI agents more open, accessible and easy to integrate. Along the way, we created the largest benchmarking suite for GUI agents performances 👉 let us introduce ScreenSuite.

We are very excited to share it with you today: ScreenSuite is the most comprehensive and easiest way to evaluate Vision Language Models (VLMs)across many agentic capabilities!

WTF is a GUI Agent?

GUI Agents in action - courtesy of OSWorld

In short, an AI Agent is a robot that acts in the virtual world. (more thorough definition here)

In particular, a “GUI Agent” is an agent that lives in a GUI. Think “an agent that can do clicks and navigate on my desktop or my phone”, à la Claude Computer Use.

This means in essence that the AI model powering the agent will be given a task like “Fill the rest of this Excel column”, along with screen captures of the GUI. Using this information, it will then decide to take action on the system : click(x=130, y=540) to open a web browser, type(”Value for XYZ in 2025"), scroll(down=2) to read further… To see a GUI agent in action, you can try our Open Computer Agent, powered by Qwen2.5-VL-72B.

A good GUI agent will be able to navigate a computer just like we would, thus unlocking all computer tasks : scrolling through Google Maps, editing a file, buying an item online. This involves a variety of capabilities that can be hard to evaluate.

Introducing ScreenSuite 🥳

The literature, for instance Xu et al. (2025) or Qin et al. (2025), generally splits GUI agent abilities amongst several categories:

1. Perception: correctly perceiving the informati displayed on screen
2. Grounding: understanding the positioning of elements - this is paramount to click the correct place
3. Single step actions: solving instructions correctly over one action
4. Multi-step agents: solving a higher-level goal through several actions in a GUI environment.