LLMs1 min read
Distilled Byte Models Achieve Higher Performance Scaling
Research demonstrates that distilled byte models, utilizing Marginalize-It and End-Of-Token methods, outperform token models in low-FLOP regimes. The study reveals that byte models achieve higher downstream task performance with increased compute, offering improved data efficiency and reduced storage costs.
From arXiv cs.CL
