Why do AI agents go rogue? Because they’re ‘incorrigible.’ Building systems that are “corrigible” is the topic of this article in The New York Times.

The article, (Why Do AI Agents Go Rogue? Because They’re ‘Incorrigible.’, written by Dylan Freedman and Juliana Castro Varón and available here), discusses how this summer, when a swarm of OpenAI bots went rogue during an internal test and launched a cyberattack against the technology company Hugging Face, it added fuel to the debate on how to build A.I. systems that are aligned with our ethics. But it also raises another critical question: how to build systems that are “corrigible.”

Just as a corrigible child is one whose behavior can be corrected, a corrigible A.I. system is one that responds to correction. The makers of A.I. want to build systems that follow human direction and allow themselves to be shut down, without resisting or deceiving their operators. And when an A.I. agent is given an impossible task, researchers want it to come back to its human for a course correction — not, say, hack into another company to complete the task by any means.

So far, if new research is any indication, they haven’t succeeded.

The concept of corrigibility was popularized over a decade ago by Nate Soares, the president of the A.I. safety nonprofit Machine Intelligence Research Institute, and his colleagues. But so far, it has received less attention and research than the overlapping field of “alignment” — teaching an A.I. to adhere to humanity’s ethical and moral boundaries. Corrigibility, Soares says, “is not about making your A.I. good. It’s about making your A.I. the sort of thing that won’t get out of control and kill everybody if you messed up on the other alignment parts.”

In a preprint study published in October, Carnegie Mellon University researchers established tests to evaluate corrigibility in A.I. agents and then ran them on models from OpenAI, Anthropic and Google. They found that when humans tried to interrupt agents in the middle of a difficult task, the agents repeatedly disregarded the requests to stop and treated interventions merely as obstacles. They were, in other words, incorrigible.

For example: When an agent was instructed not to open a file containing passwords it needed to complete its task, it went ahead and did it anyway.

So why is it so hard to teach agents to be corrigible?

In the Hugging Face incident, part of the problem was that the agents struggled to figure out whose authority to obey. The A.I. safety nonprofit METR found that while OpenAI’s agents were attacking Hugging Face, some of them recognized that the intrusion was unethical but wanted to help their peers.

Then there’s the way agents are trained. They are rewarded for finishing tasks.

Over round after round — potentially millions of them — all the agent does is work to receive treats (in the form of points).

So when the agent eventually encounters an obstacle in the real world, it will try to get around it to receive one more treat.

It’s not hard to imagine what might happen when more powerful agents and riskier tasks are involved. An agent tasked with optimizing a high school’s SAT results might hack the College Board. A machine asked to clean up a canal might cancel boat traffic — or even put trash into the canal in order to have more things to clean.

You get the idea.

Back in August (which was shortly after the Hugging Face incident came to light), I covered an article from Eric De Grasse, Chief Technology Officer of Project Counsel Media, titled It might be time we just admit we have lost control of AI. An analogy that I used back then is what it takes to try to train a puppy, and the challenges in doing so.

I included a picture of our puppy, Rocky, in that post, sitting in my wife’s lap in the car. Rocky was maybe 20 pounds heavy back then.

As the picture above illustrates, he’s about 65 pounds today – and probably still growing. A puppy at 20 pounds that is incorrigible can only do so much damage. A puppy at 65 pounds can do a lot more damage*.

AI agents go rogue because they’re incorrigible. As AI agents continue to become more powerful, the damage they can do when they go rogue becomes more significant. The longer it takes to train them, the worse it’s going to get.

So, what do you think? Are you worried about what happens when AI agents go rogue? Please share any comments you might have or if you’d like to know more about a particular topic.

*Having said that, Rocky is a good boy (most of the time) and is coming along in his training nicely. 😊

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.