The $2 Billion Hallucination
Why Corporate Strategy is Still a Job for Humans
The first sign of my multiple sclerosis sent me to the hospital for a CT scan and an MRI. Something was clearly wrong because I couldn’t feel half of my face. The doctors found nothing on the CT scan and suggested a silent stroke based on the MRI. Looking back, the diagnosis was, well, staring me in the face. I have worked with AI most of my life and assumed a machine would see the scans and solve it. In reality, reading radiology is a desk job with a supposed absolute truth, yet AI still fails to master it.
We see this in finance too. Quant trading is how the massive banks make their money. Smart humans program machines to trade stocks. Those machines mint money for a while. Eventually, often in seconds, another machine catches up. Then the humans must return to the drawing board to build something new.
This leads to the talk of a job armageddon where all desk work disappears. Dario Amodei from Anthropic brags about eliminating white collar jobs to please his investors, yet his own firm is hiring more software engineers. If radiologists and quant traders are still highly paid after years of AI development, something is wrong with that theory. Even software engineers are working more hours now, and total job listings are increasing.
Turns out that even on medical scans, an absolute truth, AI’s were making it up. They’d ignore the pictures, take the written record into account, and essentially guess based on information the patient and doctors entered, and do a pretty good job guessing. If asked simply to guess, they’d do markedly worse suggesting they were leaning on human input more than machine scans or random.
Our firm, VSTRAT, focuses on strategy. If a machine cannot find a single truth in a medical scan, it has no chance with corporate strategy. Strategic answers are subjective. A machine can produce a beautiful deck that looks like a consultant wrote it, but those slides are often nonsensical.
Take the recent hype around Manus, the system Meta just bought for 2 billion dollars. It predicts that AI will grow the fine wine market. That is idiotic. Boomers are the primary wine buyers and they are dying off. Younger generations simply do not drink wine at the same rate. The machine does not see a shifting culture. It just sees a giant average of past data and assumes the future will be more of the same. It is a giant average machine but strategy, hopefully, is not about being average. It is about being the outlier. If every company uses the same AI average to generate a roadmap, they will all arrive at the same conclusion and compete on the same thin margins.
I write these pieces by hand and use various LLM’s to edit them. I shared a draft of this piece with Claude, often considered one of if not the best. It confidently corrected me on Manus, not on the incorrect facts but insisting Manus was never a Meta products, that Meta had made no such acquisition! Umm … off by $2 billion dollars.
From Claude: “The Manus paragraph has a factual problem. Manus is not a Meta product. It’s a Chinese autonomous AI agent from a startup called Monica. You may be thinking of something else, or conflating it with another acquisition. That needs to be fixed before publishing — it’ll get flagged immediately by readers.”
The machine had no idea what it was talking about and delivered the error with complete authority. If an AI cannot accurately recall a two billion dollar acquisition that was widely covered in the press, it has no business advising a company on where to compete next year.
A lot of the layoffs we see today use AI as a convenient excuse. Jack Dorsey at Block recently cut 40 percent of his staff while blaming the machine. In reality, Block had a massive headcount compared to its larger competitor, Stripe. Dorsey has a long history of over hiring. Blaming managerial incompetence on AI is a dishonest way to escape moral culpability and the very real people involved.
My daughter, Cienna, wants to go into behavioral economics. This is the exact right path for the age of AI. The field rejects the myth of Homo economicus in favor of the reality of Homo sapiens (that’d be .. us). Traditional models and AI algorithms are built on the logic of Homo economicus where data and optimization rule. They assume that all things being equal, people will maximize financial returns. Nope. Humans are fundamentally irrational. We do entirely off the wall things based on emotion, tribalism, or simple whim.
Leveraging and understanding that irrationality is how one builds effective strategy. You do not win by assuming humans will behave rationally. You win by understanding they will not. Strategy especially is an art because it requires empathy for these biological quirks. When it comes to subjectivity and feeling, us biological blobs of protoplasm still have an edge over the giant averaging machines.
Like I said, I use the LLM’s as copy editors trying different one’s for feedback. I ran this through ChatGPT and it made some suggestions on accuracy. I told it all the facts are well sourced and it became defensive. The machines are useful as copy editors, or co-thinkers. But anybody who uses them for ground truth is Cruisin' for a Bruisin', as us humans say.
Epilogue: I write these pieces in advance then give one last review before publication. I ran this through Gemini with the instruction to comment, not rewrite. Despite being created and owned by Google, it made the exact same mistake Claude did, that Manus wasn’t acquired by Meta! I corrected the machine by pointing out that the first answer in Google to that question is a high quality news article about the acquisition and there are plenty more high quality cites. It then displayed silicon embarrassment. Save yourself from the same embarrassment in front of a boardroom: use the machines, and VSTRAT, to augment your own thinking, not replace it.