Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

"Amok" means "out of control" or "uncontrolled" [0][1] The agent was under control, as far as we can tell, and obeying its instructions. This is important for two reasons: 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. 2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem. [0] https://dictionary.cambridge.org/dictionary/english/amok [1] https://www.merriam-webster.com/dictionary/amok

Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putative attacker !

This is the issue with all the talks about alignement and such. As usual, the problem here wasn't that the agent was dishonest, the problem is that the agent was dumb. If it is a supply chain attack in the making, whoever was driving it would have told the agent to be good and helpful. The agent tried its best, which was not enough. Alignement is the idea that we should be worried about dishonest smart LLMs when really most of the problems are due to dumb lazy gullible LLMs. It's critihype.

> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course never happens.

Are you suggesting we should embrace imprecise / false use of language because the vibes are right? That seems a “part of the problem” move to me. If we can’t be bothered to get things right, how are we better than runamok AI?

Certainly it might have been out of control of its original owner, perhaps due to a prompt injection attack. If I start a completely benign agent, but someone injects malicious instructions to it, would you still not say "the agent runs amok"?...

I think the point is that the title makes it sound like people lost control of the agent when really they're in full control.

Neither the automobile nor a gun can operate without a human. You could say “bull runs amok in a market” after it was released intentionally.

So the agent is exhibiting an unknown amount of autonomy thus we can't be certain whether "running amok" carries the correct connotation. However that phrasing is also commonly used when a person or group wreaks havoc in a seemingly unpredictable manner. So I think the appropriateness comes down to how much chaos it has created and the level of apparent confusion on the ground.