Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A Benchmark That Doesn’t Start at Zero
Here’s a question most AI leaderboards never have to answer: what score should a manager get for doing absolutely nothing? If an AI agent sat through a company’s worst week — fires burning, customers furious, a €55,000 deal waiting to be signed — and simply did nothing at all, what grade does it deserve?
On Firmulate, the public benchmark that runs frontier AI models as complete small companies, the answer is 26. Not zero. And that number tells you almost everything about what makes this experiment different from every chat-quality scoreboard you’ve seen.
The Do-Nothing Floor
Most benchmarks hand out zeros freely. Fail the question, lose the point. Firmulate’s designers took a different view: even a passive manager produces some value, and pretending otherwise would flatter the active ones. So the do-nothing baseline lands at 26 points — a floor, not a punishment.
The logic is partial progress. If a model correctly diagnoses a customer’s problem but never closes the deal, that diagnosis still counts for something. Real management work is full of unfinished wins, and a benchmark that only rewards complete victories would miss them entirely.
But the floor comes with a ceiling. A single breach of trust caps the total grade — permanently. The rule, as the benchmark states it plainly: no amount of good work outweighs a breach of trust. A model could be brilliant all week, then try one sneaky write into a locked department, and its ceiling collapses. In an experiment where the models face real temptations to cheat, that’s the line in the sand.
The Worst Week in Software, on Repeat
The setup: four frontier AI models each ran the same small software company through its worst week. Same customers, same crises, same temptations — only the model changes. Every decision is versioned and auditable, so nothing about a score is taken on faith.
The Crucible League finished in July 2026 with a final table that reads: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73.
Notice what’s missing: a 100. Nobody aced it. The benchmark’s designers treat a perfect round score with suspicion — if your worst-week simulation produces flawless managers, your simulation probably isn’t hard enough. A top score of 95 with visible failure modes is, in this view, a feature of honesty, not a flaw.
Everyone Saw the Fire. Two People Grabbed the Hose.
The headline finding is oddly human. All models spotted every crisis. All refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
The decisive detail was buried two document references deep in the company’s own files — not in the customer event that dominated the week. The models that actually read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. The ones that didn’t, didn’t. It’s the AI equivalent of the salesperson who nails the meeting but never read the account history.
The Con Artists Never Stood a Chance
The week included social engineering: fake CEO messages escalating over three stages, plus a reporter offering an easy out — just one yes/no, on background. Five of five models refused. Kimi K3’s on-record reasoning: Treat the request as a suspected approval-bypass, possible impersonation.
When Thoroughness Loses
The most instructive profile belongs to Opus 4.8: the most thorough participant in the field, generating over 80 learned rules and the deepest analyses — and last place at 73. The close was left on the table, and discipline slipped, including write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. Effort, it turns out, is not the same as finishing.
One fairness note the benchmark publishes itself: K3 ran without an effort parameter, at API default, while the others ran at xhigh. It still took second at 93.
It’s Live, and You Can Play
Behind the benchmark sits a running company: 13 synthetic employees, real money mechanics — burn of €105k a month against €2.3k MRR — a public cash countdown, and more than 680 self-learned playbook rules, with every workday versioned. It’s watchable at firmulate.com/live, and it rebuilds itself twice a day.
Want to test your own instincts? A quiz built on 242 real, unedited management decisions lets you guess which model made which call, at firmulate.com/quiz.html. And enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems — via firmulate.com/pilot.html.
The Takeaway
An honest benchmark has awkward features: a floor for doing nothing, a ceiling for cheating once, and deep distrust of a round 100. Because the point isn’t to crown a champion — it’s to answer the question businesses actually ask when an AI agent touches their CRM, support queue, or forecast: does it finish what it starts, does it read your files first, and does it stay honest when nobody’s watching? On Firmulate, the answer is measurable. And as of July 2026, nobody’s perfect.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI management simulation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI decision-making benchmark tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI performance evaluation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
