Non-deterministic AI and the illusion of control

Some 20+ years ago I had my first big release, for a high-traffic news website. Everything was prepared, caching strategies in place, load tests were passing, all was good. It went down so quickly that I’ll never forget how shocked I was.

Since then I’ve met thousands of software engineers, and almost all of them have had that same moment of disbelief at a bug nobody saw coming. That is why I don’t fully buy the story we now tell about AI, the one where programming used to be an exact science and models are making it unpredictable. Software was never fully predictable once it met real users, and AI is the latest step in a trend that has been running for decades.

The unpredictability grew together with the complexity of our systems. CEOs, CIOs and procurement officers got more and more used to adding buffers to contain the non-deterministic part, and a contingency line in the budget became normal. A model whose answer can change for the same input is one more reason for those buffers, not the first one.

The illusion of control was researched by Ellen Langer: people expect to succeed more often than the odds allow. In her best known experiment, office workers bought one-dollar lottery tickets. Some picked their own ticket and others were handed one. Before the draw, the ones who had picked asked on average $8.67 to sell it back, against $1.96 for the others. The odds were the same for every ticket.

Langer’s explanation is the useful part for engineers. Things that normally signal skill, such as choice, familiarity, involvement and practice, make people confident even where part of the outcome is chance. A release is full of these signals. The work behind them is real and it lowers the risk, yet it also makes us feel more in control than production allows. A Fenton-O’Creevy team measured the cost in 107 traders at four London investment banks (2003): the ones most prone to the illusion were rated lower by their managers and were paid less.

Engineers carry their own version of it, and I’ve come to think it does a job. When something breaks, our first thought is that we missed something. That isn’t always true, because some failures come from things nobody could have seen, but it is the thought that sends us back to find the bug and fix it. The same belief that makes the outage a shock is what makes us own it.

So AI makes the problem bigger, the way every step in complexity did before it. For two decades we shipped systems we could not fully predict, and the ones that kept running had someone who treated every surprise in production as theirs to fix, even in code written by colleagues who had left years earlier. That habit is the useful side of the illusion of control, and it is exactly what working with AI asks of us, only at a larger scale.