SyncAI.news, a Varaisys broadcasting
The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale
F\

F{\i}rat \"Oncel, Cem Subakan, Mirco Ravanelli, \c{C}a\u{g}atay Y{\i}ld{\i}z

· 1 min read

ResearcharXiv cs.LG

The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale

arXiv:2611.22947v1 Announce Type: new Abstract: Kernelized attention methods, which aim to replace the $O(N^2)$ complexity of softmax attention with $O(N)$ alternatives, have been the focus of intense research in the last five years. Despite growing interest, systematic evaluations and fair comparisons across approaches and scales are still missing. We analyze kernelized attention in two halves and highlight prominent limitations on each. On efficiency, FlashAttention achieves lower memory consumption and faster wall-clock time than SOTA kernelized methods during training and in vision transformers across most sequence lengths (with inference-time memory in causal language models the only clear exception), contradicting the core assumption for pure kernelized attention used as a drop-in replacement for softmax. We also discover that reported efficiency gains often fail to reproduce against modern baselines. On quality, current kernelized methods leave gaps of 2-10 points relative to softmax under a unified distillation protocol, and model scala alone does not close these gaps. We show this gap is not intrinsic to kernelization: LARA++, a principled extension of LARA that approximates softmax via importance sampling with adaptive proposal distributions, recovers up to 99% of softmax accuracy. We establish that the gap is closable; however, the computational overhead remains substantial, leaving the efficiency wall intact. Code is available at https://github.com/firatoncel/KernelRealityCheckNeurIPS2026

Original source

This story was published by arXiv cs.LG and written by F{\i}rat \"Oncel, Cem Subakan, Mirco Ravanelli, \c{C}a\u{g}atay Y{\i}ld{\i}z. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News