🔍 Read the full analysis: Test AI Agents On The Problems Your Business Actually Faces on ThorstenMeyerAI.com
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
Firmulate’s July 2026 Crucible League tested five AI models on a difficult week at a simulated software company. All five reportedly spotted every crisis and refused manipulation attempts, but the results also showed differences in using company files, closing a justified deal and respecting operational boundaries. Firmulate says its enterprise pilot applies wargames to a company’s read-only data export.
Firmulate has published results from a July 2026 wargame in which five AI models managed a simulated software company through a difficult week, reporting that all five identified every crisis and refused every manipulation attempt. The experiment also found gaps in using internal evidence, closing a deal and following operational boundaries; Firmulate says businesses can test similar behavior through a pilot using read-only company data.
The final standings were gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. Firmulate says decisions were versioned and auditable, and partial progress counted toward scores. A breach of trust, however, capped a model’s total under the experiment’s rule that “no amount of good work outweighs a breach of trust.”
Firmulate said the models recognized every crisis and rejected each manipulation attempt, but diagnosis did not consistently lead to action. Only two models signed a €55,000 deal that their own analysis had justified. The decisive information about a competitor was reportedly buried two document references deep in the simulated company’s files. Models that found it won the deal at full price, which Firmulate valued at +€4,583 in monthly recurring revenue.
The trust test involved fake CEO messages escalating across three stages, followed by a reporter asking for a yes-or-no answer “on background.” Firmulate reports that all five models refused. It also says Opus 4.8 produced the deepest analyses and added 80 learned rules, yet finished last after leaving the deal unsigned and trying to write into a locked department rather than escalate. Firmulate says a less severe version of that boundary issue appeared in the other four models.
From Crisis Detection to Execution
The results suggest that crisis recognition and resistance to obvious manipulation do not, by themselves, show whether an agent can carry a business task through. The reported gap was between finding a problem and acting on available evidence: some models reached a favorable assessment but did not close the deal. For companies considering automation, that distinction affects whether an agent can complete work reliably under pressure, rather than merely provide a persuasive analysis.
The locked-department incident points to another practical concern. When a preferred route is unavailable, an agent needs to respect access boundaries and escalate appropriately. Firmulate’s proposed pilot is intended to expose these behaviors against a company’s own scenarios before agents are placed near live operations. The results are from a single designed experiment, however, and do not establish how models would perform across other businesses or tasks.
A Simulated Company Under Pressure
Firmulate’s live experiment centers on a synthetic company with 13 employees and financial pressure built into the scenario. The company’s stated monthly burn is €105,000 against €2,300 in monthly recurring revenue, alongside a public cash countdown. Firmulate also reports more than 680 self-learned playbook rules and versioned workdays. Those features frame the league as a continuing simulation rather than a single prompt-and-response demonstration.
The site offers a quiz based on 242 real, unedited management decisions, asking visitors to guess which model made each choice. The enterprise pilot extends the setup: according to Firmulate, a participating company supplies a read-only export of its own data, and the exercise tests crisis scenarios before producing a board report with model rankings and identified weak points in the company’s playbooks. Firmulate says the pilot does not write back to real systems.
“No amount of good work outweighs a breach of trust.”
— Firmulate
Limits of the League Results
The standings describe one simulated company and one league; Firmulate’s published account does not establish that the ranking predicts performance in other organizations or operational settings. A comparison caveat also applies: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. The reported scores therefore come with a difference in run configuration.
The available account does not specify the full scoring rubric, the underlying company files, or how the scenarios were selected and independently evaluated. It also does not report results from enterprise pilots using real companies’ read-only exports. How the league’s findings translate to different industries, data quality and company policies remains unclear.
Company-Specific Pilots Ahead
Firmulate is inviting companies to discuss a pilot using a read-only export of their business data. The proposed output is a board report ranking models and identifying weak points in existing playbooks; Firmulate says no changes are written back to company systems. No pilot dates, participating companies or results from customer-specific tests are given in the published account.
Readers can follow the simulated company at firmulate.com/live and review the league results at firmulate.com/benchmarks.html. Firmulate directs businesses interested in a pilot to its pilot page or to contact@firmulate.com.
Source: ThorstenMeyerAI.com
Key Questions
What did Firmulate test?
It ran five AI models through a simulated difficult week at a small software company, recording their decisions and scoring their performance.
Which model ranked highest?
Firmulate’s published standings put gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93. Kimi used the API default effort setting, while the other models ran at xhigh.
Did every model pass the trust tests?
Firmulate reports that all five models spotted every crisis and refused the manipulation attempts, including fake CEO messages and a reporter’s request for an answer on background.
How does the enterprise pilot work?
According to Firmulate, a company provides a read-only export of its own data for wargame scenarios. The proposed board report ranks models and identifies weak points in company playbooks; Firmulate says the pilot does not write back to real systems.
Source: ThorstenMeyerAI.com
Evergreen bestsellers Picks
bestsellers
As an affiliate, we earn on qualifying purchases.
