Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning

arXiv:2602.06600v2 Announce Type: replace Abstract: Recent work improves reasoning in large reasoning models (LRMs) by test-time scaling through self-consistency and parallel thinking, adding generic "thinking tokens", or presenting the question twice, as in re-reading (RE2). However, these approaches either inject task-agnostic tokens or mandate heuristics that do not explain, and often ignore, the spontaneous repetition that many LRMs exhibit at the head of their long chain-of-thought (CoT). In this work, we analyze and harness the model's tendency to restate the question, dubbed Echo of Prompt (EOP), as a front-loaded, compute-shaping mechanism, which can be viewed as self-generated prompt repetition inside the reasoning trace rather than in the input. Specifically, we formalize its probabilistic cost by casting echo removal as rejection-based conditioning, and define the Echo Likelihood Gap $\Delta\mathcal{L}$ as a computable proxy linking early repetition to likelihood gains and downstream accuracy. We find that (1) in a causal intervention, reinserting an echo into failed GSM8K traces raises exact match by 7.9-10.4 points for two reasoning models but not for a non-reasoning base model; and (2) in length- and suffix-controlled likelihood analyses and layer-wise attention studies conducted to show that EOP's benefits go beyond verbosity, EOP increases answer to answer-prefix attention in middle layers, consistent with an attention refocusing mechanism. Building on these findings, we develop Echo-Distilled SFT (ED-SFT) to instill an "echo-then-reason" pattern through supervised finetuning, and training-free Echoic Prompting (EP) to re-ground the model mid-trace. We evaluate on GSM8K, MathQA, Hendrycks-MATH, AIME24, and MATH-500 under identical decoding settings and budgets, and find consistent gains over baselines, including thinking-token insertion. Our code is publicly released at https://github.com/hhh2210/echoes-as-anchors.