DEBATE pairs human discussions with role-playing LLM agents initialized from the same participants. It measures whether models reproduce human opinions, individual opinion updates, and group convergence.
💬Lay Summary
Role-playing AI models are increasingly used to simulate how people talk, disagree, and change their minds. But a conversation that sounds realistic may still produce unrealistic changes in opinion. We introduce DEBATE, a public collection of discussions covering 107 topics, with records from 2,788 people in 697 groups. Participants exchange messages over three rounds of discussion and report their views before and after discussion. We use these data to test whether role-playing AI participants reproduce how people change their opinions. Across seven models, AI participants tend to agree more with others in their group after discussion, whereas humans tend to retain their initial views. Additional training on human conversations reduces this convergence tendency in three models, including when tested on new discussion topics. DEBATE offers a way to test the reliability of AI simulations used to study disagreement, misinformation, and social behavior.
✨Abstract
Participants report an initial private opinion, complete three rounds of public tweet-like posts and dyadic chats with different partners, and report a final private opinion.
Modeling opinion change through social interactions is important for understanding polarization, misinformation, and societal conflict. Multi-agent simulations with role-playing LLM agents often display unnatural group behavior, such as premature convergence, and need empirical benchmarks for comparison with real human interactions.
DEBATE contains multi-round public messages and private Likert-scale beliefs from U.S.-based participants across 107 topics. Evaluation data cover 2,788 participants in 697 groups. Its three-round design makes each participant's observed public interaction history traceable, supporting evaluation at the utterance, individual, and group levels.
We instantiate digital-twin agents with seven LLMs and evaluate two settings: Next Message Prediction and Full Dynamics Simulation. Base-model groups generally converge more strongly than human groups. Across tests with unseen participant groups and unseen discussion topics, supervised fine-tuning of Llama-3.1-8B, Qwen3.5-4B, and Mistral-7B improves stance-based alignment and reduces excessive convergence, generally bringing group dynamics closer to the corresponding human groups. Differences in opinion change and belief updating remain.
Explore the evidence
Inspect the model results, browse human and generated conversations, or explore participant personas and discussion topics. The public code and data include cached generations, stance and validity labels, training pipelines, and commands for reproducing the figures and tables.
Preprint citation
@misc{chuang2026debatelargescalebenchmarkevaluating,
title={DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents},
author={Yun-Shiuan Chuang and Ruixuan Tu and Chengtao Dai and You Li and Smit Vasani and Binwei Yao and Michael Henry Tessler and Sijia Yang and Dhavan Shah and Robert Hawkins and Junjie Hu and Timothy T. Rogers},
year={2026},
eprint={2510.25110},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2510.25110},
}
Acknowledgements
We acknowledge Smit Vasani for his core contributions to this work. We thank Aditya Dharap for assistance with human annotation, evaluation checks, and access to Gemini models.