Microsoft and Hugging Face released ThinkingBox, a testing harness that evaluates artificial intelligence agents by inspecting terminal database states instead of generated text, Hugging Face announced on Saturday.
The benchmark grades agents across 507 stateful business workflows covering retail, auto insurance, travel, neobanking, and consulting. Each workflow runs 20 separate times in isolated tool sessions to measure whether systems produce identical, correct backend results repeatedly.
Researchers evaluated 121,680 valid trials across 12 language models, finding that 79,853 attempts failed executable database checks. Among those failed runs, 67.24% terminated cleanly, called state-changing tools, and logged no final errors. Database assertions detected incorrect field values in 77.61% of failures, unintended extra side effects in 43.30%, and missing required updates in 25.36%. Deterministic diagnostic signatures attributed 79.9% of failures to tool handling problems rather than model reasoning errors.
Model consistency
Claude Opus 5.5 led overall single-attempt accuracy with a 67.16% pass rate, narrowly surpassing Claude Opus 5 at 66.50% and GPT-5.4 at 65.36%. Performance shifted sharply by domain. Models averaged 59.52% accuracy on retail workflows but dropped to 33.83% on auto insurance tasks.
Testing revealed wide splits between single-run capability and consistency across 20 repetitions. Kimi-K3 solved 476 of 507 workflows at least once, giving it the broadest reach in the evaluation at 93.89%. It completed only 68 tasks across all 20 attempts. In contrast, Claude Opus 5 solved 79.09% of tasks at least once but completed 241 tasks across all 20 attempts. Claude Opus 5.5 matched that exact consistency figure of 241 tasks, gaining no additional fully reliable workflows despite higher single-attempt marks.
Pricing and dependability
Calculations using undiscounted OpenRouter list prices established a three-model cost frontier for single-attempt successes. GPT-5.6 Sol recorded the lowest cost per successful attempt at $0.127, followed by GPT-5.4 at $0.131 and Claude Opus 5.5 at $0.276.
Measuring the cost per dependable task—calculated by dividing the expense of a full 20-run campaign by the tasks solved in all 20 trials—altered the ranking. GPT-5.4 cost $6.80 per dependable task across 128 successful workflows. GPT-6 Astra reached 231 consistent workflows at $7.45 each, while Claude Opus 5.5 delivered 241 consistent workflows at $7.80. Claude Opus 5 required $13.30 per dependable task to achieve that same 241-task total.
The Microsoft Copilot Studio team developed ThinkingBox with Toloka and academic contributors from the University of Pittsburgh, Northwestern University, Columbia University, and UC Irvine. Hugging Face hosts the benchmark behind its OpenEnv interface under a BSD-3-Clause license, with the framework code licensed under MIT and the dataset released under CDLA-Permissive-2.0. The test environment requires Linux or WSL, Python 3.11 or higher, uv, Docker, and Typesense 30.1.
