
JV
Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser, Tomasz Steifer, Felipe Urrutia, Jos\'{e} Verschae, Przemys{\l}aw Andrzej Wa{\l}\c{e}ga
· 1 min read
ResearcharXiv cs.AI
Transformer Heads Looking for Order
arXiv:2609.25588v1 Announce Type: new
Abstract: In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transformer. Unlike similar previous results, our results assume the model where transformers have an output MLP.
Original source
This story was published by arXiv cs.AI and written by Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser, Tomasz Steifer, Felipe Urrutia, Jos\'{e} Verschae, Przemys{\l}aw Andrzej Wa{\l}\c{e}ga. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


