The research investigates the scaling behavior of smaller models through distillation. It compares models trained with tokenization (Tokens) versus byte tokenization (Bytes, Bytes w/ eot) and examines the impact of distillation versus cross-entropy objectives. The experiments involved training decoder-only dense transformer models with approximately 1 billion parameters, scaling up to 1 trillion bytes of data.
Across eight benchmarks covering multiple choice QA, language generation, and machine translation, the results indicate that Token-1B models initially outperform byte models. However, with increased compute, byte models surpass Token-1B models, achieving a higher performance ceiling. Extrapolating scaling laws predicts that distilled End-Of-Token-1B models asymptotically outperform distilled Token-1B models by up to 4%.
Furthermore, the byte models demonstrate improved data efficiency, matching the performance of distilled Token-1B models using only one-sixth of the training data. The models operate over a small vocabulary of 256 bytes, eliminating the need for top-k truncation and reducing logit storage costs to approximately one-fifth.
The study also found that distilled End-Of-Token-1B models asymptotically surpass Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively. This suggests a viable approach to achieving high performance with significantly smaller model sizes.
Source: https://arxiv.org/abs/2609.12303
