SyncAI.news, a Varaisys broadcasting
T2T Technical Report: Synthesizing GUI Agent Trajectories at Scale from Video Tutorials
QX

Qingsong Xie, Yanlin Fei, Nazhou Liu, Jiacheng Tang, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das

· 1 min read

ResearcharXiv cs.CV

T2T Technical Report: Synthesizing GUI Agent Trajectories at Scale from Video Tutorials

arXiv:2611.18715v1 Announce Type: new Abstract: Training generalist GUI agents requires large volumes of interaction trajectories comprising screenshot sequences paired with grounded UI actions, yet collecting such data through human demonstration is expensive and difficult to scale. Although public software tutorial videos contain rich operational demonstrations, their continuous pixel streams lack discrete action boundaries, semantic intents, and precise spatial coordinates, making them difficult to use directly for model training. To address this, we present T2T (Tutorial-to-Trajectory), a fully automated pipeline for synthesizing GUI agent trajectories at scale. T2T spans desktop, web, and mobile platforms to convert unannotated video tutorials into structured, executable interaction trajectories. To guarantee trajectory quality, T2T combines active segment filtering, keyframe identification via localized UI change, shortcut-free observation frame selection, presence-aware grounding, and two-stage action-space normalization. These refined trajectories are then converted into multi-task training views spanning navigation, planning, forward dynamics, and grounding. We fine-tune Qwen3-VL-8B on a subset of the more than one million human-annotation-free training samples generated by this pipeline. Evaluations show that the model achieves consistent gains across multiple benchmarks covering desktop, web, and mobile environments, including OSWorld-G, ScreenSpot-Pro, AndroidControl, and AgentNetBench, indicating improved core capabilities in grounding, step-level instruction execution, and trajectory-level decision-making. Notably, all gains are achieved without any in-distribution training data from target domains. Learning exclusively from trajectories extracted from public videos via the T2T pipeline, the model exhibits cross-application and cross-task generalization capabilities.

Original source

This story was published by arXiv cs.CV and written by Qingsong Xie, Yanlin Fei, Nazhou Liu, Jiacheng Tang, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News