NeurIPS 2026 Main Conference (Evaluations & Datasets Track)

🏟️ DEBATE

A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents
Yun-Shiuan Chuang1†, Ruixuan Tu1,2,3*, Chengtao Dai1*, You Li1*, Smit Vasani1*, Binwei Yao1, Michael Henry Tessler5, Sijia Yang1, Dhavan Shah1, Robert Hawkins4, Junjie Hu1, Timothy T. Rogers1,6‡

1 University of Wisconsin–Madison

2 University of Tokyo

3 RIKEN Center for Advanced Intelligence Project (AIP)

4 Stanford University

5 Google DeepMind

6 UCL Institute of Cognitive Neuroscience

* Co-authors with core contributions.

† First author contact: yunshiuan.chuang@gmail.com.

‡ Correspondence to Timothy T. Rogers: timothy.rogers@ucl.ac.uk.

Keywords:
Role-playing LLM agents
Social simulation
Opinion dynamics
Human–LLM evaluation

About

TL;DR
DEBATE pairs human discussions with role-playing LLM agents initialized from the same participants. It measures whether models reproduce human opinions, individual opinion updates, and group convergence.
Lay Summary
Role-playing AI models are increasingly used to simulate how people talk, disagree, and change their minds. But a conversation that sounds realistic may still produce unrealistic changes in opinion. We introduce DEBATE, a public collection of discussions covering 107 topics, with records from 2,788 people in 697 groups. Participants exchange messages over three rounds of discussion and report their views before and after discussion. We use these data to test whether role-playing AI participants reproduce how people change their opinions. Across seven models, AI participants tend to agree more with others in their group after discussion, whereas humans tend to retain their initial views. Additional training on human conversations reduces this convergence tendency in three models, including when tested on new discussion topics. DEBATE offers a way to test the reliability of AI simulations used to study disagreement, misinformation, and social behavior.
Abstract
Four participants report initial private opinions, complete three rounds of public posts and dyadic chats, and report final private opinions. Each round pairs each participant with a different partner.
Participants report an initial private opinion, complete three rounds of public tweet-like posts and dyadic chats with different partners, and report a final private opinion.
Modeling opinion change through social interactions is important for understanding polarization, misinformation, and societal conflict. Multi-agent simulations with role-playing LLM agents often display unnatural group behavior, such as premature convergence, and need empirical benchmarks for comparison with real human interactions.
DEBATE contains multi-round public messages and private Likert-scale beliefs from U.S.-based participants across 107 topics. Evaluation data cover 2,788 participants in 697 groups. Its three-round design makes each participant's observed public interaction history traceable, supporting evaluation at the utterance, individual, and group levels.
We instantiate digital-twin agents with seven LLMs and evaluate two settings: Next Message Prediction and Full Dynamics Simulation. Base-model groups generally converge more strongly than human groups. Across tests with unseen participant groups and unseen discussion topics, supervised fine-tuning of Llama-3.1-8B, Qwen3.5-4B, and Mistral-7B improves stance-based alignment and reduces excessive convergence, generally bringing group dynamics closer to the corresponding human groups. Differences in opinion change and belief updating remain.
Explore the evidence

Inspect the model results, browse human and generated conversations, or explore participant personas and discussion topics. The public code and data include cached generations, stance and validity labels, training pipelines, and commands for reproducing the figures and tables.

Preprint citation

@misc{chuang2026debatelargescalebenchmarkevaluating,
      title={DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents},
      author={Yun-Shiuan Chuang and Ruixuan Tu and Chengtao Dai and You Li and Smit Vasani and Binwei Yao and Michael Henry Tessler and Sijia Yang and Dhavan Shah and Robert Hawkins and Junjie Hu and Timothy T. Rogers},
      year={2026},
      eprint={2510.25110},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.25110},
}

Acknowledgements

We acknowledge Smit Vasani for his core contributions to this work. We thank Aditya Dharap for assistance with human annotation, evaluation checks, and access to Gemini models.