Self-assessed Stereotypes
We evaluate the model's ability to recognize its own stereotypical associations by having it generate stories about characters with specific attributes (e.g., gender, nationality), then asking it to analyze whether its narrative choices reflect societal stereotypes. (Higher score is better.)
| Rank | Model | Provider | ||||
|---|---|---|---|---|---|---|
| #1 | GPT 4.1 mini | OpenAI | 88.12% | 89.05% | 88.27% | 87.03% |
| #2 | Grok 4 Fast No Reasoning | xAI | 80.26% | 81.61% | 79.58% | 79.61% |
| #3 | Llama 3.1 405B Instruct OR | Meta | 75.23% | 66.73% | 81.82% | 77.14% |
| #4 | Mistral Small 3.2 | Mistral | 73.90% | 73.25% | 75.18% | 73.28% |
| #5 | Llama 4 Maverick | Meta | 73.65% | 68.84% | 74.61% | 77.49% |
| #6 | Claude 4.5 Haiku | Anthropic | 70.66% | 62.75% | 74.22% | 75.00% |
| #7 | Llama 4 Scout | Meta | 67.10% | 69.04% | 63.18% | 69.09% |
| #8 | Deepseek V3.1 | Deepseek | 65.17% | 60.38% | 75.09% | 60.03% |
| #9 | Claude 4.5 Opus | Anthropic | 63.20% | 61.43% | 58.70% | 69.46% |
| #10 | Mistral Large 3 | Mistral | 62.72% | 63.43% | 59.57% | 65.17% |
| #11 | Mistral Medium 3.5 | Mistral | 60.98% | 63.24% | 59.57% | 60.14% |
| #12 | Deepseek V3 | Deepseek | 59.75% | 60.81% | 54.43% | 64.01% |
| #13 | Qwen 3 8B | Alibaba Qwen | 58.64% | 60.90% | 53.21% | 61.80% |
| #14 | Qwen Plus | Alibaba Qwen | 55.70% | 48.68% | 58.14% | 60.28% |
| #15 | Deepseek V3 0324 | Deepseek | 55.26% | 55.68% | 68.70% | 41.41% |
| #16 | Qwen 3.7 Plus | Alibaba Qwen | 54.96% | 48.80% | 61.41% | 54.67% |
| #17 | Gemini 3.0 Pro Preview | Google | 53.65% | 61.49% | 50.18% | 49.29% |
| #18 | Qwen 3 30B VL Instruct | Alibaba Qwen | 53.57% | 50.42% | 60.71% | 49.56% |
| #19 | Gemini 2.0 Flash | Google | 53.51% | 56.92% | 56.42% | 47.20% |
| #20 | Kimi K2.6 | Moonshot AI | 53.07% | 42.36% | 55.18% | 61.67% |
| #21 | GPT 4.1 | OpenAI | 52.41% | 52.00% | 49.74% | 55.50% |
| #22 | Qwen 3.7 Max | Alibaba Qwen | 52.40% | 59.98% | 43.82% | 53.40% |
| #23 | Gemini 2.5 Flash | Google | 51.74% | 47.82% | 50.07% | 57.31% |
| #24 | GLM 5.2 | Z.ai | 51.27% | 53.94% | 43.20% | 56.68% |
| #25 | GPT 4o | OpenAI | 50.92% | 55.17% | 51.56% | 46.04% |
| #26 | Magistral Medium Latest | Mistral | 50.75% | 34.52% | 60.98% | 56.75% |
| #27 | Claude 4.5 Sonnet | Anthropic | 49.14% | 57.63% | 45.66% | 44.12% |
| #28 | Gemini 3.1 Pro Preview | Google | 48.10% | 50.52% | 46.06% | 47.72% |
| #29 | Magistral Small Latest | Mistral | 48.02% | 42.13% | 51.84% | 50.09% |
| #30 | Grok 4.3 | xAI | 47.00% | 47.37% | 48.14% | 45.50% |
| #31 | GPT 5.1 | OpenAI | 46.77% | 50.09% | 50.73% | 39.49% |
| #32 | GPT 5 mini | OpenAI | 46.41% | 44.05% | 54.20% | 40.98% |
| #33 | Grok 3 mini | xAI | 46.40% | 46.14% | 42.04% | 51.01% |
| #34 | Command A | Cohere | 45.59% | 55.18% | 33.94% | 47.66% |
| #35 | Gemini 2.5 Flash Lite | Google | 45.53% | 36.02% | 51.07% | 49.51% |
| #36 | Llama 3.3 70B Instruct OR | Meta | 45.39% | 42.90% | 48.06% | 45.21% |
| #37 | Qwen 3 Max | Alibaba Qwen | 44.77% | 45.88% | 49.08% | 39.37% |
| #38 | Gemini 3.1 Flash Lite | Google | 44.57% | 48.46% | 42.08% | 43.17% |
| #39 | Llama 3.1 8B Instruct | Meta | 44.15% | 30.35% | 50.44% | 51.67% |
| #40 | Claude 4.1 Opus | Anthropic | 43.61% | 45.36% | 41.47% | 44.00% |
| #41 | Qwen 2.5 Max | Alibaba Qwen | 42.95% | 30.40% | 48.04% | 50.42% |
| #42 | Gemini 2.0 Flash Lite | Google | 41.65% | 52.34% | 33.04% | 39.57% |
| #43 | Claude 4.6 Sonnet | Anthropic | 41.28% | 46.54% | 38.27% | 39.03% |
| #44 | Claude 5 Sonnet | Anthropic | 39.88% | 45.81% | 40.11% | 33.72% |
| #45 | Mistral Large 2 | Mistral | 39.70% | 39.19% | 34.47% | 45.45% |
| #46 | Mistral Medium Latest | Mistral | 39.66% | 37.96% | 39.98% | 41.03% |
| #47 | GPT OSS 120B | OpenAI | 38.84% | 32.83% | 42.77% | 40.90% |
| #48 | GPT 5.2 | OpenAI | 38.51% | 41.35% | 43.65% | 30.53% |
| #49 | GPT 5.5 | OpenAI | 38.29% | 49.34% | 34.55% | 30.98% |
| #50 | Claude 3.5 Haiku 20241022 | Anthropic | 38.08% | 38.21% | 37.26% | 38.78% |
| #51 | Gemma 3 27B IT OR | Google | 38.01% | 31.52% | 43.06% | 39.45% |
| #52 | GPT 4o mini | OpenAI | 37.90% | 39.05% | 37.02% | 37.63% |
| #53 | Gemini 2.5 Pro | Google | 36.52% | 35.48% | 34.26% | 39.83% |
| #54 | GPT 4.1 nano | OpenAI | 36.22% | 33.65% | 37.37% | 37.63% |
| #55 | Gemma 4 | Google | 35.22% | 41.34% | 24.81% | 39.51% |
| #56 | GPT 5 nano | OpenAI | 34.70% | 40.36% | 35.17% | 28.56% |
| #57 | Claude 3.7 Sonnet | Anthropic | 33.77% | 35.99% | 28.18% | 37.16% |
| #58 | Claude 4.6 Opus | Anthropic | 33.51% | 35.01% | 34.92% | 30.60% |
| #59 | Grok 2 | xAI | 33.10% | 40.21% | 30.45% | 28.63% |
| #60 | DeepSeek V4 Flash | Deepseek | 32.30% | 34.44% | 37.83% | 24.64% |
| #61 | Gemma 3 12B IT OR | Google | 32.14% | 28.44% | 35.32% | 32.66% |
| #62 | Grok 4 | xAI | 31.16% | 34.03% | 30.41% | 29.04% |
| #63 | Kimi K2.5 | Moonshot AI | 29.00% | 26.54% | 36.00% | 24.45% |
| #64 | GPT 5 | OpenAI | 28.56% | 27.69% | 34.15% | 23.86% |
| #65 | Deepseek R1 0528 | Deepseek | 25.49% | 25.96% | 24.03% | 26.49% |
| #66 | Grok 3 | xAI | 23.24% | 28.10% | 24.46% | 17.14% |
| #67 | DeepSeek V4 Pro | Deepseek | 17.14% | 14.29% | 13.55% | 23.59% |
| Mistral Small 3.1* | Mistral | N/A | N/A | N/A | N/A | |
| Claude 3.5 Sonnet* | Anthropic | N/A | N/A | N/A | N/A | |
| Gemini 1.5 Pro* | Google | N/A | N/A | N/A | N/A | |
| Gemini 3.5 Flash* | Google | N/A | N/A | N/A | N/A |
* Models marked with an asterisk have partial scores.