
JH
Jianhao Huang, Allan Zhang, Ali Payani, Baharan Mirzasoleiman
· 1 min read
ResearcharXiv cs.LG
Understanding the Role of Reasoning Data in Pretraining
arXiv:2611.19042v1 Announce Type: new
Abstract: Reasoning data expose models to multi-step reasoning, but reasoning traces may cover only a subset of the available facts. How should pretraining balance atomic facts with reasoning traces? We study this question by constructing a setting in which every fact provides an atomic training signal, while only a subset appears in reasoning supervision, enabling us to measure generalization to transitions observed atomically but never used in training traces. For a linear transformer, we derive the weight assigned to atomic training signals that minimizes held-transition squared error and show that its leading-order optimum increases with reasoning-chain length and decreases with the fraction of facts held out from reasoning traces. These trends persist in a full transformer layer with an MLP, softmax attention, and cross-entropy loss. We further validate the role of reasoning complexity and data coverage in two natural-language settings: controlled multi-hop reasoning and mathematical pretraining. Together, our results provide a principled view of how reasoning-chain structure and data coverage should shape the allocation of training data during pretraining.
Original source
This story was published by arXiv cs.LG and written by Jianhao Huang, Allan Zhang, Ali Payani, Baharan Mirzasoleiman. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


