Ai2 introduced BenchMIRT, an evaluation method that audits large language model benchmarks at the level of individual prompts. The research, published on Hugging Face, uses multidimensional item response theory to determine which underlying capabilities drive model performance on specific test questions.
The researchers trained BenchMIRT on benchmarking results from 100 language models across 16 benchmarks and more than 34,000 questions. Six benchmarks evaluated general reasoning, including MATH, GPQA, MMLU-Pro, and BBH. The remaining 10 came from the Olmo 3 safety suite, covering evaluations such as HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest.
BenchMIRT independently isolated two primary dimensions across the evaluations: safety and general reasoning. Ai2 noted that these two dimensions emerged consistently across repeated tests without the system receiving prior capability labels.
Disentangling reasoning and safety
BenchMIRT showed that several evaluations commonly categorized under safety depend heavily on general reasoning. The BBQ benchmark, designed to evaluate social bias, aligned much more strongly with general reasoning. Ai2 found that lower BBQ scores often reflect difficulties in parsing question details rather than safety failures.
The WMDP benchmark, which tests dangerous dual-use knowledge in biology, chemistry, and cybersecurity, also correlated more strongly with general reasoning than with safety. Models with stronger general reasoning earned lower WMDP scores because the benchmark awards points only when a model refuses or fails to provide dangerous information.
Evaluations within HarmBench also showed mixed signals. Standard phishing prompts and contextual malware requests aligned with safety, while copyright prompts—such as requests to reproduce song lyrics—aligned with general reasoning.
Pruning evaluation datasets
Ai2 used BenchMIRT to rank questions by how effectively they distinguish model capabilities. Retaining only 10% of the most informative questions across the 16 benchmarks preserved nearly the same relative performance rankings as the full question sets, while retaining 50% matched the complete benchmarks even more closely.
BenchMIRT predicted whether a model would answer an unseen question correctly 79% of the time, compared to a 70% accuracy baseline derived from average benchmark scores.
Ai2 noted several constraints in the study. All evaluated models were released by March 2025, leaving newer architectures unanalyzed. The authors also observed that overall average scores still predict held-out model rankings slightly better than BenchMIRT, and warned that question-level discrimination data could enable bad actors to filter out difficult safety tests.
