Large language models are increasingly used in ensembles or series to balance cost and performance. However, practitioners lack a principled method for determining which model to activate at each step.
The authors introduce inference networks as a graph-based framework where nodes represent LLMs with distinct costs and expertise levels. The core problem is finding the optimal topology that minimizes expected inference cost while meeting a target performance constraint.
For this specific class of networks, the study proves that the optimal activation policy follows a threshold structure. Queries are routed to the lowest-cost model first, triggering more expensive models only if the initial confidence falls below a defined limit.
Discriminative tasks require a set of thresholds, one for each output class, whereas generative tasks utilize a single threshold. The paper provides structured methods to compute these values and practical mechanisms for estimating model confidence.
Experiments conducted with open-source LLMs demonstrate that this approach achieves substantial cost reductions without compromising the specified performance budget.
Source: https://arxiv.org/abs/2609.15992
