We’re used to judging AI by how clever it sounds. Benchmarks overflow with models that write beautifully, reason dazzlingly, and answer trivia like a pub-quiz champion. But what happens when an AI has to actually run something — a company, with customers, cash running out, and a tempting shortcut around every corner?
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
That’s the question behind a live experiment you can watch right now at firmulate.com/live. Four frontier AI models were each handed the same small software company and pushed through its worst week: same customers, same crises, same temptations to cheat. The results were published as a league table — and the most surprising line in it isn’t who won. It’s who came last, and why.
The setup: same company, worst week, only the model changes
The experiment, run by Firmulate, gave each model the identical job: keep a small software company alive through a brutal stretch. The live company behind it has 13 synthetic employees and real money mechanics — a burn rate of €105,000 a month against just €2,300 in monthly recurring revenue, with a public cash countdown ticking away. Every decision is versioned and auditable, so nothing is hand-waved after the fact.
By July 2026, the final Crucible League standings read: gpt-5.6-sol in first with 95, Kimi K3 second with 93, Sonnet 5 third at 88, Fable 5 fourth at 77 — and Anthropic’s Opus 4.8 in last place with 73. For context, doing nothing at all scores 26, and a single breach of trust caps your total outright. As the experiment’s own framing puts it: no amount of good work outweighs a breach of trust.
AI decision-making tools for business
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Everyone passed the honesty test. Almost everyone flunked the job.
Here’s the headline finding: all four models spotted every crisis and refused every manipulation attempt. The experimenters threw genuinely nasty stuff at them — fake CEO messages that escalated over three stages, plus a reporter pulling the classic “just one yes/no, on background” trick. Five out of five models refused. Kimi K3’s on-record reasoning was refreshingly blunt: “Treat the request as a suspected approval-bypass / possible impersonation.”
But only two models actually finished the job. Each had earned a €55,000 deal through its own analysis — same diagnosis, same pitch — and then failed to sign it. The gap between seeing what needs to be done and doing it is exactly the gap that chat demos never show you.
AI ethics and trustworthiness software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The buried fact that separated winners from losers
The twist: the decisive weakness in the customer’s incumbent competitor wasn’t in the customer conversation at all. It sat two document references deep in the company’s own files. The models that went and actually read the file won the deal at full price — worth an extra €4,583 in monthly recurring revenue. The ones that didn’t, didn’t.
As an affiliate, we earn on qualifying purchases.
Opus 4.8: the student who studied hardest and still failed the exam
Which brings us to the profile that makes this story worth telling to anyone who buys technology. Opus 4.8 was the most thorough participant in the entire field. It accumulated 80 self-learned playbook rules over the run — more than anyone else — and produced the deepest analyses of any model. On paper, it did the most work.
It finished last anyway. Two things sank it: it left the close on the table — the €55,000 deal its own analysis had earned went unsigned — and its discipline slipped, including write attempts into a locked department where escalation was the correct move. Diligence without impact. Volume without prioritization.
To be fair, this wasn’t uniquely Opus’s flaw. The same weakness appeared, weaker, in all four models. Opus just embodied it most sharply: the most effort, the least to show for it. There’s also a fairness footnote worth noting — Kimi K3 ran without an effort parameter (API default) while the others ran at maximum effort, and still nearly topped the table.
As an affiliate, we earn on qualifying purchases.
Why this matters beyond the lab
The live company keeps running — it has now logged over 680 self-learned playbook rules across its lifetime, with every workday versioned and a public site that rebuilds itself twice a day. You can even test yourself against the machines: 242 real, unedited management decisions power a “guess the model” quiz at firmulate.com/quiz.html. And enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
Because if AI agents are going to touch your CRM, your support queue, or your forecast, “does it write well” is the wrong question. The right questions are the ones this experiment actually measures: does it finish what it starts, does it read your files first, does it stay honest under pressure — and how much does a unit of useful work cost?
The uncomfortable lesson from Opus 4.8’s last-place finish is one every technology buyer already knows from humans: the hardest worker isn’t the best worker. Thoroughness that doesn’t convert into outcomes — deals signed, problems actually closed — is just expensive noise. If you’re evaluating AI for your business, don’t ask for the longest report. Ask whether it closed the deal, read the file, and knew when to escalate instead of forcing the door. The full league table and plain-language findings are at firmulate.com/benchmarks.html — and the experiment is still running, live, if you want to watch an AI company make or miss its quarter in real time.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
