The research introduces SynthSentry, a method for identifying synthetic data contamination within language model training corpora before training begins. The system operates at the corpus level and is model-agnostic, meaning it does not require access to the generating model, its generation history, or synthetic labels. It calculates a score based on three statistical measures: lexical diversity collapse, n-gram tail truncation, and perplexity variance across reference models. Evaluation was conducted on corpora contaminated by small open-weight generators and an instruction-tuned open-weight model using a leave-one-generator-out protocol. A domain-stratified study assessed false positives on naturally repetitive human text, including legal, clinical, and source code data. The scoring system ranked corpora by severity with minimal loss of accuracy when holding out entire generator families. Calibration techniques, including covariance shrinkage and a bootstrap threshold, improved the per-domain false-positive budget. A downstream fine-tuning check showed no accuracy deficit caused by contamination at the scale tested. The research focuses on English-language, batch-mode corpus screening and contamination sources are single-generation or hand-authored rather than recursively generated. The scope of the results is small-scale.
Further analysis involved evaluating the robustness of the scoring system. The system’s performance remained relatively stable even when holding out entire generator families, indicating its effectiveness in identifying contamination across diverse generator families. Calibration methods improved the false-positive budget, and a downstream fine-tuning check revealed no contamination-driven accuracy deficit. The research acknowledges that pruning to recover accuracy remains an open question. The study also highlights the risk of over-pruning when the contamination fraction exceeds the true contamination fraction.
The authors released a scoring toolkit for this purpose. The research frames this screening process as a data curation defense rather than a post-hoc diagnosis. The system’s effectiveness is demonstrated through a series of experiments, but the results are limited to small-scale, English-language corpora. The research does not address recursion depth or recursively generated data.
Source: https://arxiv.org/abs/2609.12353
