When normal software breaks, it tells you. An error gets thrown, someone gets a 2am alert, and the problem gets fixed before most people notice. AI is different, and that difference is the part that catches business owners off guard. An AI tool that starts giving slightly worse answers doesn't crash. It just keeps going, sounding as confident as ever. The first sign something is wrong is usually a customer complaint, a refund you didn't expect to owe, or a bill that crept up while nobody was watching.
I've been thinking about this a lot lately, prompted by a piece on The New Stack about why the usual software testing pipelines fall apart when you point them at large language models. It's written for engineers, but the lesson underneath it matters to anyone running a business that's added an AI chatbot, an email drafter, or an automated assistant. The tools you'd normally rely on to tell you your software is healthy were never built to catch the ways AI goes wrong.
Why the usual checks don't work
Traditional software is predictable. Feed it the same input twice and you get the same output twice, so you can write a test that either passes or fails. AI language models don't behave that way. Ask one the same question twice and you can get two different answers. This isn't a bug someone forgot to fix. It's how the technology works: the model picks its next word from a spread of probabilities, and tiny differences in how the numbers are crunched on the hardware mean you can't fully pin it down. One study found accuracy swinging by up to 15% across repeated runs, even with the settings dialled to their most predictable. For a business relying on consistency, that variability is the whole problem.
So a simple pass or fail test tells you almost nothing. The answer was fine this morning. That's no guarantee about this afternoon.
The ground shifts under you
Here's the part that surprises people most. If you're using a tool built on ChatGPT, Claude, or Gemini, the model itself can change without warning and without anyone telling you. Researchers at Stanford and UC Berkeley tracked two versions of ChatGPT a few months apart and found the behaviour had drifted noticeably. On one task, identifying whether a number is prime, GPT-4's accuracy fell from around 98% to under 3% in the space of three months. Nobody announced a downgrade. The model just quietly got worse at that particular job.
As the team behind DeepLearning.AI's newsletter put it, ordinary software infrastructure evolves slowly, but the models underneath AI tools change far faster, which leaves whatever you built on top standing on much less stable ground. You could have your AI working beautifully in January and subtly misbehaving by March, with no code change on your end to explain it.
What that costs in the real world
This isn't theoretical. Air Canada learned it the hard way. A customer asked the airline's website chatbot about bereavement fares, the bot gave him wrong information, and when he tried to claim the refund it had promised, the airline refused. He took them to a tribunal and won. Air Canada argued, memorably, that the chatbot was a separate legal entity responsible for its own actions. The tribunal called that submission remarkable, rejected it, and held the airline liable for what its bot said. The damages were small. The precedent wasn't: if your AI tells a customer something wrong, that's on you, not the bot.
Why so many AI projects stall
Put these things together and you start to understand a stat that made headlines last year. MIT's State of AI in Business 2025 report found that around 95% of company AI pilots delivered no measurable impact on the bottom line. The researchers were clear that the cause usually wasn't the quality of the AI. It was everything around it: shaky integration, no feedback loop, no way to tell whether the thing was actually working over time.
We looked at the return side of this problem in our piece on what AI actually delivers in year one for small businesses. The short version: the firms that see real returns are doing almost exactly the unglamorous monitoring work this article is about.
One finding from the MIT research is worth sitting with if you're weighing up how to proceed. Tools brought in through an experienced partner succeeded about twice as often as ones companies tried to build entirely on their own. Going it alone tends to underestimate exactly this kind of work.
What good actually looks like
The teams that get real value from AI treat it as something they keep checking, not something they switch on and walk away from. In practice that means a small, sharp set of test questions you run regularly to spot when quality slips, rather than assuming it holds. It means guardrails that catch a bad answer before it reaches a customer, with a record of every time one fires so you can see trouble building before anyone complains. And it means accepting, as one engineering guide bluntly puts it, that good AI behaviour is not a set-and-forget deal.
If you want a structured way to check whether what you've already deployed is actually pulling its weight, our AI tools revenue audit guide walks through that process.
None of this is exotic. It's the AI equivalent of a smoke alarm and a regular service. The businesses that skip it aren't saving money. They're just choosing to find out about problems from their customers instead of from a dashboard.
If you've already put an AI tool in front of your customers or your team and you're not sure how you'd know if it started drifting, that's worth a conversation. At OrionX we build AI automation for small businesses and accounting firms with these checks baked in from the start, and we're happy to take a look at what you've already got. Better to find the cracks yourself than to meet them in a customer email.
