SyncAI.news, a Varaisys broadcasting
When Can Attention Heads Be Statically Defined?
WW

Weixian Waylon Li, Yintao Tai, Marcio Fonseca, Shay B. Cohen

· 1 min read

ResearcharXiv cs.CL

When Can Attention Heads Be Statically Defined?

arXiv:2609.34650v1 Announce Type: cross Abstract: Some attention heads learn similar patterns across inputs. Reusing these patterns could reduce training cost by avoiding repeated query-key score computation and softmax. Through controlled pretraining comparisons, we identify Selective Attention Freezing (SAF), which selects heads with low attention-pattern variance and replaces their attention weights with fitted post-softmax means halfway through training. We represent these fixed patterns with absolute-position and relative-distance preferences, reducing storage from quadratic to linear in sequence length. A fused kernel reconstructs the patterns and executes ordinary-attention and replaced heads together. At matched training-token budgets, replacing 25% of attention heads gives 1.056x faster post-replacement optimiser updates at 124M parameters and 4K context, with a 0.77% perplexity increase. At 1B and 8K context, post-replacement updates are 1.068x faster on four GPUs including communication, with a 0.51% perplexity increase. The resulting models also accelerate long-input finetuning and causal prefill. After associative-recall adaptation, the 124M model with 25% replacement generalises to more key-value pairs at a fixed length better than ordinary attention and two pruning controls.

Original source

This story was published by arXiv cs.CL and written by Weixian Waylon Li, Yintao Tai, Marcio Fonseca, Shay B. Cohen. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News