Grounded language-model pipelines consist of three stages: selecting an object, retrieving passages, and using evidence to answer. If the selected object does not reach the reader, it breaks the handoff.
The audit involved 600 HybridQA questions across three selector families. Results show that exact key lookup and title matching reliably return objects when they match dataset passages. However, BM25 retrieval omits selected objects in some cases, especially at lower cutoff thresholds.
Discrepancies between selector identities occur in a notable portion of records, affecting the accuracy of object retrieval. Removing the passage containing the object significantly reduces exact match scores, indicating the importance of correct object handoff.
The study introduces the Returned-Object Profile (ROP), an executable record capturing the target, returned ID, cutoff, and membership rules, supporting offline replay and analysis.
Source: https://arxiv.org/abs/2609.04579