SyncAI.news, a Varaisys broadcasting
Separators in Enhancing Autoregressive Pretraining for Vision Mamba
HL

Hanpeng Liu, Zidan Wang, Shuoxi Zhang, Kaiyuan Gao, Kun He

· 1 min read

ResearcharXiv cs.AI

Separators in Enhancing Autoregressive Pretraining for Vision Mamba

arXiv:2603.03806v2 Announce Type: replace-cross Abstract: The state space model Mamba has recently emerged as a promising paradigm in computer vision, attracting considerable attention for its efficient handling of long-sequence tasks. Its inherent causal structure makes it particularly well suited for autoregressive pretraining. However, existing autoregressive pretraining methods in vision are largely limited to short-sequence settings and may not fully use Mamba's capacity to model longer contexts. To investigate this setting, we introduce SeparaTors for AutoRegressive pretraining (STAR), a new autoregressive pretraining method for Vision Mamba that explicitly marks the boundaries between different images. STAR increases the patch-token sequence length from 144 to 640 by packing four images and four separator clusters. This is approximately $4.4\times$ the ARM patch-token sequence length. The increase is achieved without changing the resolution of any individual image: we use $192\times192$ inputs for autoregressive pretraining and $224\times224$ inputs for downstream classification fine-tuning. With this long-sequence pretraining scheme, STAR-B achieves 83.5\% EMA top-1 accuracy on ImageNet-1K after 1,600 epochs of pretraining. The learned representation also transfers beyond in-distribution classification: compared with ARM, STAR-B improves COCO box AP from 46.11 to 46.84 and mask AP from 40.74 to 41.45, while raising the mean top-1 accuracy across five ImageNet robustness benchmarks from 55.1\% to 56.8\%. Under the evaluated four-image setting, these results indicate that separator-based long-sequence pretraining improves recognition robustness and dense visual prediction relative to ARM.

Original source

This story was published by arXiv cs.AI and written by Hanpeng Liu, Zidan Wang, Shuoxi Zhang, Kaiyuan Gao, Kun He. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News