
YW
Yiqin Wang, Nuri Cingillioglu, Charles Pert
· 1 min read
ResearcharXiv cs.CL
Log-Depth Recurrent Language Modeling
arXiv:2609.28212v1 Announce Type: cross
Abstract: Language modeling using Transformers has become commonplace despite their fixed computational depth and quadratic runtime with respect to input tokens. Recurrent models on the other hand offer linear depth but no parallel execution. In this work, we extend balanced-tree recursive operators from sequence encoding to autoregressive prediction, enabling all prefix representations to be computed with logarithmic depth and linear runtime. Our experiments provide an initial characterization of this model class, demonstrating robust length extrapolation and performance approaching that of ALiBi-based Transformers, highlighting its potential as an alternative architecture for language modeling.
Original source
This story was published by arXiv cs.CL and written by Yiqin Wang, Nuri Cingillioglu, Charles Pert. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


