🔍 Read the full analysis: How To Reconcile An AI Agent’s Status With The Database on ThorstenMeyerAI.com
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
Microsoft and Hugging Face have made ThinkingBox available through Hugging Face, offering a benchmark that checks AI agents’ backend changes across 507 business workflows. The authors report that many attempts failed executable checks even when agents made state-changing tool calls without a final tool error; the results apply to the benchmark’s tested setup.
Microsoft and Hugging Face have made ThinkingBox available through Hugging Face, releasing a benchmark that checks whether AI agents leave business systems in the required state after completing tasks. It covers 507 workflows and repeats each task 20 times, aiming to measure both task success and consistency rather than relying only on valid tool calls or convincing replies.
ThinkingBox runs agents in isolated sessions using MCP tools, then checks the resulting backend records and side effects against executable requirements. The release names five areas: retail, auto insurance, travel, neobanking and consulting. Each task begins from a clean backend, according to the supplied description.
The authors’ common-set analysis covered 121,680 valid trials across 12 models. They report that 79,853 attempts failed the executable checks. Of those failed attempts, 67.24% ended without a final tool error despite the agent having invoked a state-changing tool. Checks identified wrong field values in 77.61% of failures, unintended extra effects in 43.30%, and missing required effects in 25.36%; the categories overlap.
The release reports an overall pass@1 score of 67.16% for Claude Opus 5.5 and 57.37% for Kimi-K3, which it describes as the strongest open-weight model in its table. The authors say Kimi-K3 scored within one point of GPT-6 Astra. The supplied material does not include the full results table or uncertainty estimates for those comparisons.
Why Backend State Changes Matter
For businesses using agents to handle support tickets, refunds, claims or bookings, a fluent reply does not prove the requested work was completed. The benchmark’s central distinction is between what an agent says or attempts and what it actually records in a system. A ticket could be closed too early, a field could hold the wrong value, or an unintended action could occur even when the interaction appears orderly.
Repeated runs address a separate concern: reliability across attempts. Pass@1 measures the share of individual attempts that pass; pass@20 records whether a task succeeded at least once in 20 runs; observed 20/20 counts tasks that passed every recorded run. These measures can help teams compare outcomes in the benchmark, but they do not establish how an agent will behave in a particular company’s live systems.
As an affiliate, we earn on qualifying purchases.
From Tool Calls to Recorded Outcomes
The benchmark authors frame ThinkingBox as a response to an evaluation gap: valid tool use and a plausible final answer are indirect measures of whether a workflow was completed correctly. The benchmark instead checks terminal backend state and side effects against task requirements. Its release says it is based on the authors’ paper and can be run through OpenEnv.
One example describes a retail support task involving a delayed $745 appliance order. The agent investigates the delay, opens a ticket and records the timeline. The customer does not qualify for late-delivery compensation under the policy the agent checked, but a carrier exception remains open and the ticket is required to stay on hold. The agent marks it solved and replies without answering the customer’s underlying question. The authors say the check fails because the ticket status is solved instead of hold.
“A tool call is not an outcome.”
— Microsoft and Hugging Face, in the ThinkingBox release
business workflow automation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Scores
The supplied release material does not specify when ThinkingBox became available or provide all details of the paper’s evaluation setup, including full task specifications, model configurations and uncertainty estimates for the reported scores. The findings are the authors’ results on this benchmark; they do not establish how the same models would perform across all live business systems or operating conditions.
It is also unclear whether passing 20 recorded runs predicts long-term reliability. The trials provide a bounded observation, while real deployments can involve changing records, unusual requests, integrations and policies absent from the tested workflows. The material does not identify an independent replication or a planned follow-up milestone.
database state reconciliation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Testing Workflows Beyond the Benchmark
The release says researchers and developers can run ThinkingBox through OpenEnv using isolated MCP tool sessions. That offers a way to inspect tasks and compare agent outcomes against executable checks. No future release date or additional milestone is specified in the supplied material.
For organizations considering agents, the next practical step is to evaluate relevant workflows against their own requirements and inspect both passing and failing runs. Whether benchmark results predict performance in a particular organization remains to be established.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does ThinkingBox measure?
It checks whether an AI agent leaves backend records and side effects in the state required by a task, rather than judging only tool calls or the final reply.
How large is the benchmark?
The release describes 507 business workflows, each repeated 20 times. Its common-set analysis covered 121,680 valid trials across 12 models.
What did the authors report about failed attempts?
They report that 79,853 attempts failed executable checks. Among these failures, 67.24% ended without a final tool error despite a state-changing tool call. The reported categories of wrong values, extra effects and missing effects overlap.
Do the scores predict how an agent will perform at a company?
No such conclusion is established by the supplied results. They describe performance in the benchmark’s tested setup; organizations would need to assess their own workflows and requirements.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
