Supervised fine-tuning (SFT) typically applies a uniform cross-entropy loss to all target tokens. This approach can lead to over-sharpening of already mastered tokens while amplifying learning pressure on uncertain tokens. The research proposes TrimSFT, a token-level reweighting method, to address this issue. TrimSFT scales the SFT loss based on the logit gap between the gold token and its strongest competitor, concentrating learning within an intermediate logit-gap region.
Experiments evaluated TrimSFT on six base models – Llama, Qwen, and DeepMath families – across five mathematical reasoning benchmarks. The results consistently showed TrimSFT improved over standard SFT, achieving the best average performance on five out of six models. The gains ranged up to +26.9 points on MATH500. Analysis revealed that the bandwidth { au} was more influential than the exact margin location.
Furthermore, half-trim variants, removing supervision from only one side, yielded inferior results. A token-level logit-gap distribution analysis demonstrated that TrimSFT reshaped model confidence in a more balanced manner compared to uniform SFT or monotonic reweighting methods. This suggests that reasoning SFT can benefit from trimming both extremes rather than treating all tokens uniformly.
Source: https://arxiv.org/abs/2609.09707