NCP-ArchPreview introduces a latent-space language model that utilizes next-concept prediction alongside next-token prediction. The model’s architecture constructs a product-quantized concept vocabulary from hidden states and learns to predict future concepts via a dedicated Concept Module. These predicted concepts guide subsequent token-level generation, with both objectives trained end-to-end. The model was trained on 5.73T tokens from the Dolma-3 dataset, scaling to 8.9B parameters.
Notably, NCP-ArchPreview achieves a final pretraining loss comparable to OLMo-3-7B after consuming only 51.3% of the total training tokens. Performance improvements were observed across several downstream tasks, including a 2.45-point gain on the macro-average and a 5.99-point gain on GSM8K. Controlled experiments highlighted the impact of both the latent architecture and the NCP objective.
Further experimentation revealed efficiency gains. Utilizing 85% of standard computation, NCP-ArchPreview approaches the training loss of a strictly parameter-aligned 8.9B baseline. The learned latent space remains valuable; updating the 17M-parameter VQ module enables domain adaptation, and injecting concept representations into a DFlash2 drafter improved the mean accepted length by 4.17% with minimal overhead.
This architecture represents the largest demonstration of a latent-space language model to date.
Source: https://arxiv.org/abs/2609.10715