The SWORD benchmark was introduced to assess factual understanding in LLMs beyond simple answer selection. It utilizes Wikidata triples, generating distortions through controlled perturbations – including random entity substitutions and property-based selections – in eight widely spoken languages. The benchmark’s design surfaces two key observations. First, models demonstrate higher accuracy on distortions that are semantically plausible compared to random, nonsensical substitutions. This suggests a reliance on distributional familiarity rather than a deep understanding of factual relationships. Second, performance degrades substantially in (East) Asian languages when presented with distorted statements, with cross-lingual performance gaps reaching up to 28 percentage points. This highlights asymmetric capabilities in multilingual factual reasoning. These findings indicate that standard benchmarks fail to accurately represent the complexities of factual understanding in multilingual LLMs.
This benchmark’s impact for engineers running models or agents is significant. The observed performance degradation in specific language families suggests that current evaluation methods are insufficient for identifying vulnerabilities. The ability to reliably detect and mitigate these inconsistencies is crucial for deploying LLMs in applications requiring accurate and consistent cross-lingual reasoning. The benchmark’s design allows for targeted testing and improvement of model robustness.
Furthermore, the benchmark’s methodology offers a new approach to evaluating LLMs. By focusing on the consistent rejection of factual errors, it moves beyond simply measuring accuracy and provides a more nuanced understanding of a model’s capabilities. This shift is critical for developing more reliable and trustworthy AI systems.
Source: https://arxiv.org/abs/2609.09349