The research program focused on BabyLM 2026 Strict-Small, trained on a corpus of 10 million words and 100 million cumulative word presentations. The project progressed through three stages, beginning with frontier advancement through compact restatements and residual incremental learning. Subsequent analysis revealed patterns in context use dependent on target relations and prediction windows, demonstrating that recovering familiar performance did not guarantee generalization to unseen inputs. This led to the formulation of a data-efficient learning principle: organizing experience around contextual dependencies.
Stage III implemented strategies including retaining source text, masking local clues, supervised target selection, and preserving predictions on ordinarily masked inputs. Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. The Overall score rose from 42.02 to 42.25 across two generations, with the second generation achieving the highest score in the public Strict-Small snapshot of 8 September 2026.
Further research addressed compression, relational anchors, shared representations, and measurement. Model weights and associated code and research records are available on Hugging Face and through a GitHub repository. This work illustrates Research RSI, a recursive self-improvement of the research process.
Source: https://arxiv.org/abs/2609.10702