ContractEval addresses a problem in LLM agent behavior: failures can appear correct, even when procedural steps are missed. Existing evaluation methods, such as output-only and trace-aware judging, do not consistently identify these structural failures. ContractEval represents procedural instructions as query-active obligations, allowing for a direct comparison against response or trace evidence. This framework detects failures like skipped checks, incorrect branching, ordering errors, and invariant breaches. Evaluations on a controlled suite of audited contracts demonstrated that ContractEval detected all structural failures under gold expected and observed graphs, while standard LLM judges missed many injected failures. LLM-backed extraction preserves this signal but remains sensitive to calibration. The framework makes procedural conformance auditable rather than implicit in final-answer quality.
Source: https://arxiv.org/abs/2609.09458