
CT
Charan Teja Nagisettygari, Siva Raja Sindiramutty
· 1 min read
ResearcharXiv cs.LG
How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips
arXiv:2611.19135v1 Announce Type: new
Abstract: Greedy decoding at zero temperature is widely assumed to be deterministic, although the exact same prompt may yield different tokens depending on the number of other prompts sharing the batch for inference. The currently available solutions operate during inference, trading determinism for throughput. Can we train a model in such a way that fewer decoding steps would be susceptible? We characterize the perturbation empirically across six model families and three numerical precisions and observe two empirical facts about the measurement: the perturbation depends exactly on the representable precision (8.00x for bfloat16 vs float16, 8192.0x for float16 vs float32, in line with predictions based on mantissa bits) and the batch invariance happens accidentally at certain tensor shapes, allowing a practitioner to achieve zero nondeterminism merely by setting the sequence length. Then we apply hinge penalization of the model's top-1 / top-2 logit gap, scaled to twice the measured noise quantile. On GSM8K dataset it leads to a 35% reduction in token flips and divergence from 53.8% to 38.1%, at the cost of 1.3 - 2.9 accuracy points, increasing monotonically with the margin size. It does not affect the gap threshold sufficient for determinism: the logit gaps of surviving flips are unchanged (median 0.125 for all arms), as the hinge is a soft penalty and the largest hardware perturbation remains the same. We reject seven hypotheses, including a per-model viability diagnosis, holding on six models and falsified on the seventh.
Original source
This story was published by arXiv cs.LG and written by Charan Teja Nagisettygari, Siva Raja Sindiramutty. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


