SyncAI.news, a Varaisys broadcasting
Train After You Measure: a Pre-Registered Learning Curve for Fine-Tuning Typed Decision Models on One User's Labels
FL

Federico Lucca

· 2 min read

ResearcharXiv cs.CL

Train After You Measure: a Pre-Registered Learning Curve for Fine-Tuning Typed Decision Models on One User's Labels

arXiv:2611.21393v1 Announce Type: new Abstract: Fine-tuning a small typed decision model on one user's email labels did not clear the margin we pre-registered over a logistic regression trained on the same labels. In earlier work, about two hundred labels and a TF-IDF logistic regression had the highest point accuracy on this inbox, though not significantly above the best hosted model; we ask whether the same labels, given to a decision model, close the gap. A first registered study, on a generic 4B base with cross-validation and message type only, stayed below the baseline at every label budget (-1.8 to -7.3 points). The main study was registered before any fine-tuning run: a fixed test set of 135 emails never used for selection, LoRA adapters on the public Tev1-0.8B and Tev1-4B decision models, nested subsets of 50 to 587 labels, two seeds and five decision fields on a free T4 GPU. Four of its five hypotheses are refuted--two in the fine-tunes' favour--and one is confirmed for one model. On message type the 4B model misses the registered three-point margin at 400 labels (+2.6) and passes it with all 587 (+3.3); with all labels both models are above the baseline by +0.7 and +3.3 points, where one email is 0.74 points and a 95% interval is about 15 points wide, and the 4B model is clearly ahead in type macro-F1 (0.581 vs. 0.406). On the three action switches the baseline is ahead at every label budget for both models (0.931 vs. 0.904 and 0.899 mean accuracy with all labels), at under a millisecond per question against 306 and 1681 ms. The 0.8B type curve barely improves after 300 labels and then falls. The baseline never predicts the two rarest classes (7 offers and 8 spam in the training pool); the fine-tuned models find spam in some runs, not consistently across seeds. We list the explanations we did not test and say when a new attempt would be worth making. Scripts are public; weights and emails are not.

Original source

This story was published by arXiv cs.CL and written by Federico Lucca. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News