A simulation framework utilizing the World Values Survey (WVS) was used to evaluate LLM-based agents. The framework involved approximately 4,000 conversations with 1,200 personas across 15 topics and three models: GPT-4o, Gemini-2.5-Flash, and Gemma-4-E4B. The research assessed value faithfulness, value drift, and conversational realism.
Initial findings indicate that more than 50% of personas did not express their assigned WVS profiles from the outset. Furthermore, 2-7% of personas exhibited value drift after repeated conversations. Ablations removing demographic details improved faithfulness for some models, but did not alter the overall trend of systematic deviation from assigned WVS profiles.
The simulated dialogues demonstrated a different trade-off between stylistic consistency and semantic diversity compared to human discussions. Content-wise varied exchanges were frequently accompanied by stylistically repetitive interactions. This suggests current LLM agents can generate plausible conversations but struggle to preserve diverse human value profiles over time.
These results are relevant to engineers running LLM agents in production, particularly when utilizing these agents for social science research or applications requiring nuanced value representation.
Source: https://arxiv.org/abs/2609.05514