Microsoft ThinkingBox Grades Agents on Database State Across 20 Replays

On October 3, 2026, Microsoft’s Copilot Studio team, in partnership with Toloka, published ThinkingBox with Hugging Face, an open agent sandbox and benchmark that grades business agents on the backend records and side effects they leave behind, then asks whether the same workflow still passes when replayed twenty times from a clean state. Named co-authors include former Microsoft interns from the University of Pittsburgh, Northwestern, Columbia, and UC Irvine, with Hugging Face’s Sergio Paniego on the joint post. The Hugging Face blog and paper (arXiv:2608.19741) argue that fluent final answers and well-formed tool calls are weak proxies: an agent can close a ticket as solved while the required status should have been hold, or mutate the wrong fields while reporting success. ThinkingBox ships the harness under MIT, ThinkingBox-Bench data under CDLA-Permissive-2.0, and an OpenEnv environment under BSD-3-Clause so labs can reproduce scores through a binary pass/fail reward.

Why tool traces disagree with the database

ThinkingBox-Bench covers 507 stateful workflows across retail, auto insurance, travel, neobank, and consulting domains. Each attempt gets an isolated MCP session with freshly initialized state so two trials never share a row or cached tool context. After the episode, a side-effect extractor derives what actually changed; deterministic judges compare that terminal state to the required outcome, rejecting wrong, missing, or extra effects. Of the 507 tasks, 477 grade on state alone; 30 add narrow response rubrics for disclosures that lack a clean database field. In a common-set ablation of 121,680 valid trials across 12 models, 79,853 attempts failed executable checks—and 67.24% of those failures still terminated cleanly with a state-changing tool and no final tool error. Of the failures that still terminated cleanly, executable checks found wrong field values in 77.61%, unintended extras in 43.30%, and missing required effects in 25.36%. Those findings overlap.

The blog’s customer-parcel vignette makes the failure mode concrete: nine careful tool calls, a correctly read non-compensation policy, and a ticket closed as resolved while the carrier exception remained open. An AI grader watching tool syntax would cheer; the database would not.

Pass@1 versus observed 20/20

Every task runs 20 independent trials. Microsoft reports pass@1 (usual single-attempt rate), pass@20 (solved at least once), and observed 20/20 (passed all twenty recorded attempts—no smoothed estimator). On company-reported pass@1, Claude Opus 5.5 leads overall at 67.16%; among open-weight models, Kimi-K3 leads at 57.37% and tops retail at 82.24%. Consistency pulls the ranking apart: Kimi-K3 solves 93.89% of tasks at least once but only 13.41% on every attempt, while Claude Opus 5 solves fewer tasks once yet hits 47.53% observed 20/20. Claude Opus 5.5 matches Opus 5 at 241 dependable tasks despite a slightly higher pass@1—half a point of headline accuracy bought no extra every-attempt coverage in that comparison. Domain skew is large: retail averages far higher pass@1 than auto insurance across the published table.

Cost analysis prices list rates into cost per successful attempt and cost per dependable (20/20) task. On single successes, the Pareto frontier is GPT-5.6 Sol at $0.127 per success, GPT-5.4 at 3.45 points higher pass@1 for $0.004 more, and Claude Opus 5.5 at another 1.80 points and $0.276 per success. GPT-6 Astra is not on that frontier. It appears when the unit is a dependable task: 231 tasks at $7.45 each, between GPT-5.4’s cheaper but narrower $6.80 (128 tasks) and Claude Opus 5.5’s $7.80 for the joint-high 241 tasks. None of those three dependable-cost rows dominates the others. GPT-5.6 Sol, cheapest per single success, costs $9.76 per dependable task. Failure signatures attribute about 79.9% of failures to tool-usage issues rather than pure reasoning mistakes—an operational hint to classify recoverable tool errors before blaming model size.

What consistency costs operators

The practical reading is for refund, booking, and claims agents: a strong pass@1 with thin 20/20 coverage is an incident factory. The post’s advice is to treat observed consistency as a design input, classify recoverable tool errors, shrink the tool surface, and require human approval on irreversible writes—then measure those mitigations on the OpenEnv path.

How to run it and what not to claim

Operators clone OpenEnv’s thinkingbox_env, pin microsoft/thinkingbox-data at thinkingbox-bench-v1.0, install the ThinkingBox CLI, start Typesense plus MCP servers, then score episodes with the packaged evaluator. Canonical results bind a framework commit, data release, and bundle hash. The released adapter is for evaluation; separate training scenarios can reuse OpenEnv without treating the public bench as a training dump. Microsoft is clear that pass@1 leaderboards alone are the wrong column for agents that mutate money, bookings, or tickets.

For Writeble’s ai-agents desk, ThinkingBox is the October 3 open evaluation story that replaces “the agent said it was done” with twenty clean replays against a judge that reads the database.