Smol2Operator: Post-Training GUI Agents for Computer Use

TL;DR: This work shows how a lightweight vision–language model can acquire GUI-grounded skills and evolve into an agentic GUI coder. We release all training recipes, data-processing tools, resulting model, demo and datasets to enable full reproducibility and foster further research 🫡. Find the collection here.

Introduction
1. Data Transformation and Unified Action Space
- The Challenge of Inconsistent Action Spaces
- Our Unified Approach
- Example Data Transformation
- Custom Action Space Adaptation with Action Space Converter
- Key Features
- Usage Example
- Transformed and Released Datasets
2. Phase 1: From Zero to Perception
- Training Data
- Optimization Experiments
- Image Resolution and Coordinate System Analysis
- Key Findings
- Phase 1 Results
3. Phase 2: From Perception to Cognition
- Training Data
- Phase 2 Results
4. All you need is Open Source
5. Conclusion
What's Next?

Introduction

Graphical User Interface (GUI) automation is one of the most challenging frontiers in computer vision. Developing models that see and interact with user interfaces enables AI agents to navigate mobile, desktop, and web platforms. This will reshape the future of digital interaction.

In this blog post, we present a comprehensive approach to training vision-language models for GUI automation through a multi-phase training strategy. We demonstrate how to transform a model with zero grounding capabilities into an agentic coder capable of understanding and interacting with graphical interfaces.

Rather than aiming for a SOTA model, our goal is to demonstrate the entire process, from data processing to model training, and, in doing so, show how to unlock GUI-grounding capabilities in VLMs.

GUI capabilities combine understanding of the interface and precise element localization. These abilities enable the model to translate high-level tasks into low-level GUI actions such as clicking, typing, …