SyncAI.news, a Varaisys broadcasting
Why your fine-tuning job takes longer than the FLOPs say
HF

Hugging Face Blog

· 2 min read

AI LabsHugging Face Blog

Why your fine-tuning job takes longer than the FLOPs say

Predicting end-to-end fine-tuning time on shared GPU clusters, from 272 production runs.

A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The full version will be on arXiv soon.

GPUs are among the most expensive and scarcest resources in AI today. Anyone running a shared cluster has to decide, every day, which jobs go on which GPUs and in what order. That plan is only as good as the estimate of how long each job will take. When a job finishes hours early, GPUs sit idle until the next one is dispatched. When it runs hours late, everything queued behind it slips, experiments wait and delivery dates move.

Better duration estimates turn this into planning instead of guesswork: what runs, when, and on how many GPUs, with less idle capacity in between. Yet the estimate most teams use is a back-of-the-envelope FLOP calculation. It says nine hours, and the job finishes the next afternoon.

We wanted to know how far off that math really is, and whether we could do better before a job is even dispatched. So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs, about 7,382 GPU-hours in total, and compared analytical estimates against learned predictors.

The short version: the usual FLOP-based estimate explains little of what we observed (R² = 0.31, MAE 12.41 h). A CatBoost regressor that only sees information available before dispatch reaches R² = 0.98, with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation. The gap comes from two things a closed-form estimate cannot do. The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.

The estimate everyone uses is compute-only

T_theoretical = (6 · P · D) / (N_GPU · FLOPS_peak · MFU)

What we built

Two design choices shaped the work:

Original source

This story was published by Hugging Face Blog. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on huggingface.co

Similar News