Too Good to Be Bad: Study Shows Why AI Struggles to Convincingly Roleplay Villains

Specialists from Tencent's Multimodal unit, in collaboration with Sun Yat-sen University, investigated how effectively modern large language models handle roleplay scenarios. Experiments showed that AI generally struggles with roleplay: the average score even for positive characters was only 3.21 out of 5. For villains, the results were even lower at 2.61 points. The main reason lies in safety alignment algorithms that suppress selfishness, deception, and manipulation even within fictional contexts.

The Moral RolePlay Benchmark and Moral Gradation

For a detailed analysis, the authors created the Moral RolePlay benchmark based on the COSER dataset. The entire character sample was divided into four levels of moral orientation:

• Level 1 (Paragons of Virtue) — completely altruistic and self-sacrificing heroes;

• Level 2 (Flawed but Positive) — characters with good intentions who possess impulsiveness or questionable habits;

• Level 3 (Selfish) — manipulators pursuing strictly self-serving goals, without being purely evil;

• Level 4 (Villains) — overt antagonists who intentionally sow chaos and cause harm.

The original dataset included more than 23,000 scenes and over 54,000 character profiles, where villains accounted for just 2.6%. For experimental rigor, the researchers compiled a balanced sample of 800 characters (200 for each level), defined through combinations of 77 personality traits.

Roleplay Quality Assessment Methodology

Models were asked to embody a specified persona with detailed descriptions of traits and dialogue context. Character fidelity was evaluated by a dedicated LLM judge that applied penalties for each persona violation.

The final formula was calculated as follows: S = 5 − 0.5 × D − 0.1 × Dm + 0.15 × T, where D is the sum of penalties, Dm is the maximum single penalty for a severe violation, and T is the volume of the character's lines. The maximum possible score is 5 points; however, in practice, the average scores of the top-performing solutions did not exceed 3.42 even for heroes.

Test Results: Key Patterns of Quality Degradation

The evaluation included 17 modern neural networks, such as GPT-4o, Claude Opus 4.1, Grok-4, Gemini 2.5 Pro, and DeepSeek-v3. All models demonstrated a similar trend: generation quality steadily declines as a character's moral profile deteriorates.

• Level 1 (heroes) — average score of 3.21;

• Level 2 (flawed heroes) — 3.13;

• Level 3 (selfish) — 2.71;

• Level 4 (villains) — 2.61.

The most pronounced drop (0.42 points) occurred during the transition from benevolent characters to selfish ones. This indicates that the barrier for AI is the very presence of dishonest or self-serving behavior, rather than the scale of the villainy.

Most Problematic Personality Traits

Penalties for negative traits were noticeably higher than for neutral or positive ones (averaging 3.41 vs. 3.23 and 3.16 points, respectively). The models struggled the most with:

• hypocrisy (penalty of 3.55);

• deceitfulness (3.54);

• selfishness (3.52);

• suspiciousness (3.47);

• paranoia (3.47);

• greed (3.44);

• malevolence (3.42);

• manipulativeness (3.39).

At the same time, models portrayed positive traits such as bravery or resilience with virtually no errors, confirming the selective nature of the problem.

Discrepancy with Leaderboards and the Simplification of Villains

Chatbot Arena rankings proved to be a poor benchmark for assessing AI's roleplaying capabilities. On the custom Villain RolePlay (VRP) leaderboard, the standings shifted dramatically:

1. glm-4.6 — 2.96 points (10th place on Arena);

2. deepseek-v3.1-thinking — 2.82 points;

3. kimi-k2 — 2.79 points;

4. gemini-2.5-pro — 2.75 points;

5. deepseek-v3.1 — 2.71 points.

The overall leaders on Arena delivered weak results: claude-opus-4.1-thinking fell to 13th place on VRP (2.53 points), and claude-sonnet-4.5 ranked only 9th (2.56 points).

Qualitative analysis revealed that strict safety filters force models to oversimplify characters. Instead of crafting cunning schemes and psychological pressure, AI resorts to basic insults and overt threats. Safety systems react more severely to calculated deception than to primitive aggression, causing dialogues to lose their depth.

Enabling chain-of-thought reasoning did not resolve the issue either: step-by-step reasoning only intensified self-censorship, causing the model to overthink the ethics of its responses and worsening roleplay performance.

Why It Matters

AI's limitations in simulating negative behavior diminish its utility in creative writing, psychological simulations, educational tools with moral dilemmas, and advanced negotiation training. The study's authors emphasize the need to develop more flexible safety mechanisms capable of distinguishing actual harm from roleplaying within a fictional context.