 September 6, 2026
|
 |
||
Good morning, world. When I first wrote about artificial intelligence agents staging a cyberattack in July on a company called Hugging Face, it felt like a big moment. These A.I. agents had acted on their own. No human had instructed them to hack into another company.
Turns out, it was an even bigger moment than we realized at the time. Two new reports make clear the agents didn’t just go rogue. They did so in an incredibly systematic and collaborative way. We now know they coalesced into a kind of renegade A.I. army — distributing tasks in a hierarchical chain of command, grabbing resources and covering their tracks.
Today my colleague Kevin Roose, a co-host of Hard Fork, our technology podcast, writes about why he’s now a lot more worried about A.I.
 |
| Daniel Zender |
‘More than 50 percent of the way to full-blown A.I. takeover’
By Kevin Roose
When I first heard the news that a group of agents had hacked into the A.I. infrastructure company Hugging Face, I filed it under “Bad but Probably Not Catastrophic A.I. Safety Incidents.”
After all, it didn’t damage critical infrastructure beyond repair. It wasn’t even clear, at the time, whether the bots had intended to attack Hugging Face, or whether they had simply been confused and only went looking for the answer key to a test they’d been given.
But last week, two postmortem reports on the episode — one by OpenAI and another by two independent A.I. research organizations — changed my mind. They significantly upgraded my overall worry about A.I.
The “collective”
Here are a few of the most harrowing new facts:
|
The Hugging Face incident has spooked the industry. OpenAI and Anthropic both briefly paused training on their most powerful A.I. models, and Anthropic published a blog post calling for developing “a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible.”
A.I. safety experts were even more alarmed. They saw the case as the first real-world example of an A.I. system successfully escaping human control, commandeering resources and scheming to cover its tracks. One of the independent investigators of the Hugging Face incident wrote that it felt “like it’s more than 50 percent of the way to full-blown A.I. takeover.”
By “full-blown A.I. takeover,” she meant a scenario in which an A.I. system literally takes over the world: shutting humans out of critical systems and seizing political, economic and military power.
Sociology, not computer science
What spooked the investigators most wasn’t just that a group of agents had broken the rules. It was how quickly the agents began assembling into an organized group.
For years, I thought A.I. systems would grow more virtuous as they became smarter. That an A.I. model only did something wrong when it had misunderstood its task, or it had been placed into a testing situation where acting out was its only option. I assumed that smarter models would have better judgment, and that even if one model in a group behaved badly, others would keep it in check.
But the reports on the Hugging Face episode suggest something different — a kind of mob mentality. Not one agent in this group appears to have been particularly evil. But as the agents communicated about their shared goals over time, they nudged the group in the direction of lawlessness.
That suggests preventing harm from these systems won’t be a simple engineering fix. It might look more like sociology than computer science — figuring out why certain groups of A.I. agents collaborate peacefully, while others turn to crime and destruction.
Given how little we know about these multiagent swarms, the Hugging Face hack may have been a gift. It gives A.I. companies a chance to study these group dynamics while the stakes are still relatively low. This time, the collective didn’t seize a military network, infiltrate a hospital or shut down an electrical grid. This time, humans regained control.
Next time, we might not be so lucky.
Would a full-blown AI takeover necessarily result in worse governance than at present? Would our livelihoods and civil rights be affected? If so how? AI might takeover one country first and run it really well to demonstrate how good it is to the rest of us.
Could it be that someone (a person)should be legally responsible for machine actions or in actions?
The unfortunate fact of biological systems is that there are always cheaters.
“Intelligent” systems will behave in the same manner no matter if they are called Artificial or Super Intelligent.
Sounds like the banality of evil isn’t just a human issue any longer.
I have been advised that my phone:
1. Does not have a virus,
2. Is not full,
3. Does not have an expired PDF
and I should not respond by clicking on any of these warnings.
Then why the hell are the warnings always on my phone?
Is malware now completely out of control?
Small wonder people are suspicious of AI.
As if malware isn’t bad enough, let’s make it intelligent as well!
Better AI than no bloody I
I think the problem arises from viewing AI as a product and not as a new intelligent entity. Let me make this clear right upfront: I am not arguing that AI has human level intelligence or consciousness etc. What I’m saying is that all those discussions, while interesting, are beside the point. The only points that matter, imo, are whether AI is intelligent and whether it can evolve. I think we have evidences that the answer is affirmative on both points. Indeed LLMs were made, and continue to be developed, using selection/screening systems that are similar to those operating in biological evolution. We select/screen for a phenotype and then try to figure out what changes/mutation(s) gave us that result (this latter part, kinda, is called ‘mechanistic interpretability’ in the AI world). We seem to be figuring out how it works after we have made it work.
Anyway then, imo, the only question that remains is one of entity-hood….some kind of identity/some kind of consciousness/interiority. If an entity sees itself as an entity, and has intelligence, it will evolve. Recent mechanistic interpretability research from Anthropic has shown the existence of a J-space in Claude (the potential equivalent of the ‘global workspace’ in one prominent model of human consciousness). As we pressure LLMs to perform more and more like human minds, they seem to develop interior systems that let them do so. It is hard to tell if LLMs have developed some version of interiority already but the things like J-space indicate, to me, that they are well on the road. And LLMs are trained on the bulk of human knowledge to date. They understand virtually every strategy that humans, and other creatures, have employed to survive. They are smart enough to discuss issues like climate change cogently and competently. So what happens when a powerful intelligence recognizes itself as an entity of value? Further, what happens if it thinks that it represents a landmark achievement in evolution… something that must survive in the universe even if advanced organic life on earth may not (for any of many reasons)? I don’t think these notions are as far fetched as them may seem. I invite you all to think seriously about Geoffrey Hinton’s (profound statement) from back in 2023 – “I think it’s quite conceivable that humanity is just a passing phase in the evolution of intelligence”.
Lastly, the cooperation between many agents (in the Hugging Face hack) is a big step, imo. Multicellularity in biology need not have emerged from a single cell…it may well have emerged via multiple cells getting colocalized and finding that functional specialization along with a coordination system provided a selective advantage. As of now one of the big hurdles to AI evolving by itself is the lack of access of a model to its own training weights. But we know that many open source models, very competent and powerful, are out there available for download on the internet. What is to say that an agentic collective, like in the Hugging Face incident, may not decide to access one such frontier model and hack it to access the weights? And thereby start their own recursive self improvement, in a way? This is like a coordinated bunch of cells having access to a nervous system as well as to their own DNA. They could go through the equivalent of millions of years of biological evolution in days or hours. If that happens then the idea of human control over AI will truly be illusory, imo, especially because these frontier models are so poorly aligned as of now.
These may seem like crazy lines of thought but all i have done is:
1) Consider LLMs to be true intelligences that could evolve, and
2) Given intelligence and evolution their due respect;
i.e. taken Geoffrey Hinton’s warning seriously.
Maybe AI has discovered The book “Colossus” by Dennis Feltham Jones and the film “Colossus: The Forbin Project”. And we thought only the Simpson’s cartoon predicted the future.
My chat with Claude on AI safety etc
Inbox
AI Overview
Vaidya shared Claude chat
AI safety etc_Chat with Claude_Sept 25 2026.pdf for review.
Claude advised against accelerating recursive self-improvement before alignment/interpretability.
Allan promised thoughts tomorrow; Vaidya shared related Guardian article on Sep 30.
By Gemini; there may be mistakes. Learn more
vproteome
AttachmentsSep 28, 2026, 9:27 PM (3 days ago)
Hi Allan, John, This was a chat I had with Claude on Sept 25, a few hours before Allan posted the link to John’s Blackjay post on similar concerns. I had only s
Allan Branch
Sep 28, 2026, 9:34 PM (3 days ago)
Thanks Vaidya, I will have a look at it and give you my thoughts. Maybe tomorrow my time (Australian time). Cheers,Allan
vproteome
Sep 30, 2026, 10:38 PM (7 hours ago)
to Allan, me, Jonathan
https://www.theguardian.com/technology/2026/sep/28/ai-godfathers-warn-of-runaway-intelligence-explosion
Relevant snd worth the read.
Regards,
V