Do Heavy Tails Help Diffusion? On the Subtle Trade-off Between Initialization and Training

arXiv:2605.13175v3 Announce Type: replace Abstract: Recent works have proposed incorporating heavy-tailed (HT) noise into diffusion- and flow-based generative models, with the goals of better recovering the tails of target distributions and improving generative diversity. This motivation is intuitive: if the data are heavy-tailed, HT noise may appear better matched than light-tailed (LT) Gaussian noise. However, replacing Gaussian noise by HT noise also changes the underlying estimation problem. In this paper, we revisit this paradigm through a combined theoretical and empirical study, establishing sampling-error bounds for two representative diffusion models driven by HT and LT noise. We show that HT diffusion models can achieve exponentially fast initialization, but accumulated training errors may offset this advantage. Experiments on synthetic and real-world datasets support our theoretical findings. Our results call into question a growing design trend in generative modeling and challenge the use of HT noise to improve rare-region exploration.