AI: A Little Knowledge Can Go A Long Way to Creating Mayhem
Be careful when leaning on "experts."
I’ve created expert systems my entire life. For those who don’t know, a legal services company sent me to law school not to be a lawyer (heaven forbid) but to write legal expert systems. Which brings me to the next point: a little knowledge can go a long way to creating mayhem.
A True Life Example
During the financial crisis, the Mortgage Electronic Registry System (MERS) was seen as a real villain, hiding the true owner of a mortgage behind a dark and mysterious database. The reality? I spoke to some investment bankers at one of the big investment banks and they said they were originally sending faxes to bundle loans, then somebody thought of a system to share the documents electronically (legal). After some time, somebody said to enable the ownership field to be updated rather than updating it at the county clerk with paperwork and fees (illegal).
The result? Chaos.
People were foreclosed on by banks they’d never heard of and there was no record that a bank ever lent them money. Not only foreclosures: banks were signing off loans as legally paid off despite there being no record they had a right to do so. It’d be as if a random person walked up to you and said you owe them money. You might say “yes, but not to you” and they’d answer “doesn’t matter - to somebody.” Out of caution that somebody may reappear and demand their money back, you wouldn’t pay the random person.
Lawyers who were hired based on cheap pricing found a workaround: they forged the documents. Putting it mildly, this caused a mess. Eventually, the President himself intervened and the lawyers who did this lost their licenses to practice. A small number ended up in prison, a number that should have been larger, but the government realized their fraud could harm the entire US financial system so settled for putting them out to pasture rather than behind bars.
In private, the explanation was blunt. Paperwork processing, especially collections, had been pushed down the organization to people who were just good enough not to get fired. Those teams, under relentless cost pressure, outsourced again to the cheapest providers they could find. Each step added distance, reduced accountability, and replaced judgment with checklists. By the time senior executives noticed anything was wrong, the system had already normalized shortcuts that crossed legal, ethical, and brand tarnish lines, without anyone feeling personally responsible.
All this gets to the point: automation can be convenient and helpful. It was entirely fine to swap scans rather than documents. The next step was updating the ownership information while skipping the official way to do it saved money. And the rest is an epic failure.
The AI Revolution Isn’t Very Different
We see so-called “experts” after a few hours on ChatGPT making recommendations to firms who unwisely listen and create, at best, failed projects. We haven’t heard much of systemic fraud yet but, based on my own experience, it’s just a matter of time.
There’s a lot of hostility towards investment bankers who made this mess and much is deserved. But I also spoke with quite a few of them and they said, repeatedly, “I have a Princeton undergrad and Harvard MBA and .. how the hell did I end up tied to these crooks! We just sent this to a low-paid department who outsources it and this is a mess.” No doubt, the same thing is happening with AI.
Much like nail guns can be very useful, they can also be very lethal if used incorrectly. Automation is no different. Just because automation looks benign doesn’t mean it is benign. One small change leads to another, and another, and next thing the President is announcing the debasement of a major part of the economy.
Looking Forward
As we head into the New Year, let’s hope for AI progress but let’s also rein in the so-called experts running wild. I’ve been working on these systems for decades and most of the time they weren’t very sexy. INSEAD was wise enough to ask me to come work on one, blending strategy and AI, long before the invention of generative AI.
There are a small number of us who’ve been doing this a long time. Much like it wasn’t wise to hire those low-cost lawyers, I’d urge firms to take the time to find people with a provable long-term record rather than smooth-talking “experts.” Granted, we may look geekish and we’re all but certain to dress more poorly than them, but we’re less likely to send your firm into the wilderness.
Listen to genuine experts, people who understand the entire ecosystem rather than relying on short-term cost, lest you end up here: