Can AI Models Make Better Business Decisions? Watch the Live Experiment Reveal the Truth

In a world where AI increasingly manages critical business functions, the question isn’t just about chat quality or speed. It’s whether these models can truly handle complex management dilemmas—staying honest, reading the hidden files, and closing deals without shortcuts. A groundbreaking live experiment from Firmulate puts four of the newest AI models to the test, running an actual software company’s worst week. The results are surprising—and revealing.

The Live AI Business Wargame: Real Decisions, Real Stakes

Imagine watching a real software company in its toughest week—dealing with angry customers, ethical temptations, and urgent crises—while AI models take charge of management decisions. That’s precisely what Firmulate’s live experiment does. Four frontier AI models, including GPT-5.6-SOL and Kimi K3, each run the same scenario—same customers, same crises, same temptations—to see how they handle real-world management pressures.

This isn’t a simulated chat or a theoretical test. It’s a real-time, auditable process where every decision is tracked. The goal? Measure which AI models can identify hidden dangers, refuse manipulation, and ultimately close profitable deals without cutting corners.

The experiment’s key finding? All four models detected every crisis and refused all manipulation attempts, including fake CEO messages and reporter tricks. Yet, only two models managed to close the deal they had analyzed and earned their full payment—€55,000 — while the others left money on the table. The difference lay in reading the company’s own internal files, buried two document references deep, which proved crucial in winning the deal at full price.

This demonstrates a vital point: in business decision-making, understanding the full context—including hidden internal information—is essential. Models that read beyond surface cues perform better, and honesty under pressure isn’t a given—it’s a measurable trait. For companies deploying AI in roles like CRM, support, or forecasting, these findings suggest that raw performance scores are less meaningful than how well models stay disciplined, read internal data, and adhere to ethical standards.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI