PerfReasoning is a new benchmark designed to evaluate large language models (LLMs) in the context of hardware performance modeling. It tests models both as direct reasoners and as generators of analytical performance-model code.
Given workload, architecture, and mapping specifications, models compare mappings and predict off-chip traffic and buffer requirements. The strongest closed-source models exceed 90% accuracy on reasoning-based questions, while the best open-weight model reaches 82.4%.
Model construction is notably more difficult; for example, GPT-5.6 Sol exceeds an 80% pass rate, but other configurations average below 15% and show high variability across runs. Task-specific reinforcement learning improves mapping-reasoning accuracy, but feedback-free multi-round self-revision prompting is less reliable.
PerfReasoning highlights the gap between architectural reasoning and reliable performance-model building. The benchmark will be publicly released to support reproducible evaluation and future progress tracking.
Source: https://arxiv.org/abs/2609.04476