Social reasoning requires tracking how information is distributed across actors, not only what happened in the world. Existing Theory of Mind (ToM) benchmarks evaluate this ability through endpoint question answering: models are scored by whether they produce the correct final answer, leaving the underlying mental-state representation entirely unobserved. A model can answer "Where will Bob look?" correctly while failing to track the belief states that make the answer valid.
We introduce OmniToM, a benchmark that closes this gap through explicit belief-structure modeling. Rather than probing endpoints, OmniToM requires a model to reconstruct the full multi-actor belief representation of a story, then label each belief proposition along seven dimensions grounded in ATOMS, a literature-derived taxonomy of Theory of Mind abilities.
OmniToM comprises 895 stories and 22,343 labeled belief propositions developed through over 1,000 person-hours of human annotation effort. Zero-shot evaluation across ten open- and closed-source models identifies a consistent actor-specific information-tracking bottleneck: Stage 1 belief extraction peaks at 61.63% macro F1, while Stage 2 belief labeling reaches 85.95% accuracy. Stage 2 results point to difficulty with Knowledge Access, with some models also struggling on Representation. These dimensions require a model to determine who could know or share a belief, and whether it is explicitly stated or inferred from context.
Seven-Dimensional Schema (grounded in ATOMS)
Order Truth Status Knowledge Access Representation Content Type Mental Source ContextOmniToM evaluates belief-structure modeling in two stages, both run under zero-shot TELeR Level 3 prompts (task directive + stepwise sub-tasks, no in-context examples):
Stage 1, Belief Extraction: Given a story, the model extracts all relevant (Actor, Belief, Order) tuples. A GPT-5 semantic judge scores extraction by precision, recall, and macro F1 over semantically matched propositions.
Stage 2, Belief Labeling: Given a story and the benchmark belief table, the model assigns a seven-dimensional schema label vector to each belief. Scored by exact-match accuracy per dimension, averaged over all propositions.
Benchmark construction required over 1,000 person-hours of human annotation to establish high-quality gold labels. From this human-calibrated foundation, a human-in-the-loop LLM-assisted pipeline was fixed and scaled to the full 895-story benchmark. Each story produces an average of 24.96 labeled belief propositions spanning world facts (Order 0, 32.6%), actor beliefs (Order 1, 57.1%), and nested beliefs (Order 2+, 10.3%).
Zero-shot TELeR Level 3 prompts. Bold = best, underline = second-best. GPT-5 is the semantic judge and is omitted from Stage 1.
| Model | Stage 1 F1 [95% CI] | Endpoint QA | Order | Status | Access | Repr | CType | Source | Context | Overall |
|---|---|---|---|---|---|---|---|---|---|---|
| Closed-source models | ||||||||||
| Gemini-2.5 Pro | 61.63 [60.88, 62.39] | 82.77 | 96.23 | 86.50 | 66.94 | 89.01 | 86.35 | 86.70 | 87.07 | 85.54 |
| Gemini-2.5 Flash | 55.75 [54.63, 56.82] | 81.82 | 95.56 | 84.97 | 71.34 | 87.58 | 85.97 | 84.10 | 92.14 | 85.95 |
| GPT-5 | N/A | N/A | 95.18 | 82.72 | 66.85 | 83.42 | 79.96 | 83.02 | 88.83 | 82.85 |
| Open-weight models | ||||||||||
| Gemma-3 27B | 58.03 [57.05, 59.01] | 58.47 | 96.56 | 82.44 | 71.57 | 54.33 | 73.50 | 78.72 | 92.07 | 78.46 |
| Mistral-Small 24B | 56.46 [55.53, 57.41] | 76.55 | 95.13 | 82.22 | 74.59 | 62.79 | 76.01 | 84.82 | 91.90 | 81.06 |
| Mistral-Large 123B | 53.53 [52.61, 54.44] | 77.39 | 97.25 | 86.53 | 74.14 | 72.87 | 82.83 | 86.32 | 92.97 | 84.70 |
| Qwen3 32B | 51.63 [50.68, 52.56] | 73.62 | 96.42 | 82.43 | 73.91 | 62.45 | 71.27 | 76.84 | 90.81 | 79.16 |
| Llama-3.3 70B | 47.12 [46.12, 48.09] | 76.22 | 92.74 | 83.55 | 67.41 | 72.43 | 72.35 | 76.71 | 91.69 | 79.55 |
| Qwen3 8B | 42.70 [41.50, 43.85] | 64.84 | 73.38 | 67.17 | 57.94 | 63.77 | 51.43 | 61.49 | 74.62 | 64.26 |
| Llama-3.1 8B | 37.21 [36.02, 38.38] | 63.95 | 71.90 | 65.59 | 56.13 | 64.40 | 48.63 | 55.18 | 76.81 | 62.66 |
Actor-specific information tracking is the core bottleneck. Models can parse social stories, but they struggle to determine which information each actor has access to, how it is communicated or inferred, and how it becomes part of that actor's mental-state representation.
Stage 1 reveals the structural side of this bottleneck. F1 shows a consistent gap between Order 0 world facts and actor-specific beliefs. Moving beyond world facts requires the model to determine which facts each actor perceived, missed, remembered, was told, or could infer. Higher-order beliefs require an additional step because the model must represent one actor's view of another actor's information state.
Stage 2 explains the same bottleneck at the schema-label level. Knowledge Access is among the weaker dimensions across models (56.13-74.59%). Representation is also weak for several models (54.33-89.01%). The analysis by belief order shows that Order 1 actor beliefs have the lowest overall labeling accuracy (72.2%), with especially low accuracy for Knowledge Access (59.2%) and Representation (60.1%). Order 2+ accuracy is higher overall (75.3%), so the results do not show a monotonic decline with belief order.
The gap between stages is itself diagnostic: models label provided belief propositions far better (up to 85.95%) than they extract those propositions from raw text (up to 61.63%). Current LLMs are much better at operating over an explicit belief structure once it is given than at constructing that structure directly from story text, a distinction invisible to endpoint QA benchmarks.
@article{bawatneh2026omnitom,
title = {OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling},
author = {Bawatneh, Adam and Sapkota, Sagar and Bedi, Amrit Singh and
Karmaker, Santu and Shah, Mubarak},
journal = {arXiv preprint arXiv:2605.26322},
year = {2026}
}