The research introduces CUSP, a framework for uncertainty quantification in multi-agent multimodal reasoning systems. The framework aggregates responses from multiple vision-language models (VLMs) and generates two system-level signals: collective uncertainty and Jensen-Shannon divergence (JSD). These signals provide a measure of dispersion and conflict within the ensemble.
Within the pooled semantic opinion, the collective entropy decomposes into the mean of individual semantic entropies and the JSD. The framework requires neither token logits nor calibration labels, applying to both open-weight and commercial VLMs. In small ensembles, collective uncertainty demonstrates the strongest signal, achieving 0.764 AUROC and 0.889 AUARC for prediction-error detection and abstention, respectively. This outperforms majority voting and naive selection by 4.7 to 15.8 points.
In commercial regimes, JSD shows the strongest performance, achieving 0.819 AUROC and 0.910 AUARC for hard-answer model conflict. The pooled prediction also improves accuracy over the average single model by 5.6 to 13.0 points. Furthermore, over a multi-step system, subagent collective uncertainty identifies system failures above chance (0.619 AUROC) and provides the best abstention ordering (0.699 AUARC).
Source: https://arxiv.org/abs/2609.05708