The research investigated a method for enhancing the reliability of large language models (LLMs) when performing arithmetic, specifically within the context of clinical calculators. The standard approach involves hardcoding each calculator as a validated function, a process that is labor-intensive and difficult to scale. This work proposes an alternative: the model does not perform the calculation itself. Instead, the model generates case-specific Python code that a restricted local executor then runs as a deterministic solver. The model’s task becomes selecting the appropriate solver to use.
Evaluation was conducted using MedCalc-Bench, a benchmark consisting of 1,100 cases and 55 calculators. Qwen2.5-7B and Qwen2.5-32B-AWQ models were compared against direct model arithmetic and a hand-written 22-calculator library. The benchmark was created by auditing clinical formulas against current guidelines, identifying 16 cases with version, use, or coefficient concerns.
The results showed a benefit at the 32B scale, with a 90.53% success rate compared to 83.47% for the 7B model (+7.05 points, 95% confidence interval of [0.47, 14.60]). However, the 7B model showed a slight improvement over direct arithmetic (75.31% vs 72.02%). The hand-written library achieved 100% accuracy on its 440 supported cases but failed on other cases (40.0%).
Ultimately, the study suggests that the Program-Solve interface provides a more reliable solution for certain open-weight models, particularly at the 32B scale, but it does not replace the need for verified formulas or reliable variable extraction. Source: https://arxiv.org/abs/2609.10728v1