Hybrid language models combine attention mechanisms with a fixed-size recurrent state, but the specific roles of each component were previously unclear. Two cache-level interventions were introduced to analyze their functions.
The split-prefill method isolates either the key-value (KV) cache or the recurrent state from a prefilled context, then generates an answer. The state-swap technique pairs the KV cache from one context with the recurrent state from another in a single forward pass. Results show that retrieval accuracy relies on attention, with a sharp decline in recurrence-only setups.
The study found that output language and persona are primarily influenced by the recurrent state, which can accept words not in the context but related in meaning. Attention provides a lookup over what was said, while the recurrent state shapes how the model articulates it next. This understanding can inform model design and debugging.
Source: https://arxiv.org/abs/2609.04434