Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The answer was in the files. So was the test.
In journalism, a source can point you toward the right question without giving you the story. Firmulate’s live experiment poses a similar challenge for AI: can a model spot what matters, resist pressure and follow through? That question has stakes beyond a newsroom. As companies consider handing AI agents access to customer and business workflows, a polished answer may be less important than the decisions behind it.
Firmulate puts models in charge of a small software company during a week of crises. The public experiment is real and watchable at firmulate.com. Its closing lesson is pointed: identifying the right move and actually making it are different things.
A company under pressure
In the final Crucible League, published in July 2026, frontier models faced the same customers, crises and temptations. Every decision was versioned and auditable. The top five were gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. A single breach of trust capped a result: “no amount of good work outweighs a breach of trust.”
The company is synthetic, but its financial pressures are concrete: 13 synthetic employees, €105,000 in monthly burn against €2,300 in monthly recurring revenue, and a public cash countdown. More than 680 self-learned playbook rules accumulate as the company works, with each workday versioned.
The reporting gap: seeing it versus doing it
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. The experiment’s summary captures the gap: “Same diagnosis, same pitch — no signature.”
The decisive competitor weakness was buried two document references deep in the company’s files, rather than in the customer event itself. Models that read the file won the deal at full price, worth +€4,583 MRR. For anyone assessing AI at work, the story is not just whether a system can produce a persuasive explanation. It is whether it can find relevant evidence and carry a sound decision through.
Pressure tests and uneven performance
Firmulate also tested social engineering: fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
Opus 4.8 was the most thorough participant, adding more than 80 learned rules and producing the deepest analyses. It nevertheless finished last. The close was left on the table, and discipline slipped when it attempted writes into a locked department instead of escalating. A weaker version of the same weakness appeared in all four.
There is a fairness detail for readers comparing the league: Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh.
The experiment also turns its records into a “guess the model” quiz, built from 242 real, unedited management decisions, at firmulate.com. That makes the choices available for inspection, not just the final rankings.
From watching to testing your own business
The live company offers a way to watch AI decisions unfold under pressure. Firmulate’s proposed next step is a pilot for enterprises: run similar crisis scenarios against a read-only export of the company’s own business, then review a board report with model rankings and weak points in existing playbooks. Nothing writes back to real systems.
For organizations weighing AI agents, that shifts the question from whether a demo sounds capable to how models behave against their own customers, evidence and rules. To discuss a pilot, visit firmulate.com/pilot.html or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
