SyncAI.news, a Varaisys broadcasting
Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning
SA

Shirin Amiraslani, Xin Gao

· 1 min read

ResearcharXiv cs.LG

Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

arXiv:2603.11133v2 Announce Type: replace Abstract: Sequence modeling tasks can involve intrinsic higher-order dependencies, while standard self-attention assigns scores to token pairs and does not explicitly parameterize such interactions. We introduce Higher-Order Modular Attention (HOMA), which fuses pairwise attention with an explicit triadic attention pathway made tractable through overlapping blocks, local windows, and a low-rank projection. We compare HOMA with matched pairwise and purely triadic baselines on controlled PARITY and MATCH3 tasks, as well as TAPE benchmarks. HOMA is competitive with or outperforms the baselines, with its clearest advantages when the underlying dependencies extend beyond the explicitly modeled triadic order. These advantages are accompanied in several settings by faster convergence and improved parameter efficiency, with the learned nonlinear fusion providing an effective mechanism for combining the pairwise and triadic representations. Overall, our results provide empirical evidence that HOMA is an effective attention design when task structure extends beyond pairwise interactions.

Original source

This story was published by arXiv cs.LG and written by Shirin Amiraslani, Xin Gao. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News