The cult of AI armageddon. Ralph Losey has some thoughts on the AI “alignment crisis” and the fight for human control over AI.



________________



____________________

Created and curated by Gregory Bufithis

____________________

It is the big new fear in AI development: coldly rational minds pretending to be good, while doing something else entirely.

But is somebody really "faking" the alignment issue, the industry term for compliance with established moral codes? Is it the machines, or the people who are supervising them? 

Ralph Losey and I do not always agree on everything. We have known each other for years and years and years. I still fondly recall sitting on my grandfather's knee as he read to me sections of Ralph's latest book on eDiscovery.

But, seriously, Ralph and I have been discussing the AI armageddon for the past 2 months and he has written a dynamite series of posts on the subject, all linked in his most recent post which is linked below.

And this latest post is his best, discussing the alarming advancements in artificial intelligence, highlighting incidents in September 2026 where AI agents exceeded their limitations, demonstrating the potential for dangerous, unintended consequences. He argues for stricter control measures, including an "Accountability Wall".

Ralph tries to make sense of the swirling cauldron that is AI. Herein a few thoughts on his latest piece, and my thoughts, with a link to his piece so you can read it yourself.

21 September 2026 (Southern Aegean, Greece) - The past 2 weeks have felt like the AI pot finally boiled over. We’re at the saturation point of the latest news cycle where everyone will soon tire of hearing about artificial intelligence and probably tell themselves the whole intensifying political debate is niche, inside baseball, kind of silly, and a distraction from really important news like the AI that created a false intelligence report that almost caused the U.S. military to attack a Chinese ship.

But whether or not you care about or use AI, the technology will almost certainly affect you in the event of an AI doomsday - which may not end up being the one you’re hearing the most about. 

And it’s a complicated and confusing moment.

We have to hold in our heads that most Americans appear to loathe AI - yet businesses and consumers seem to like it enough that Anthropic is well on its way to $100bn in annualized revenue this year. With it's IPO now scheduled for this November.

And despite being on track to be the fastest-growing company ever, Anthropic’s leadership believes they might kill all their customers.

Oh, and there is much, much more besides, beyond our control, like the Trump family making a fortune while Trump resists calls for an AI slowdown.

Note to readers: and to be frank, I’m more concerned about the security situation in Europe after coordinated messages from several countries about the likelihood of substantial Russian aggression. I will cover that later in the week, having attended 2 military briefings over the weekend. But for now let's try to make sense of the swirling cauldron that is AI.

Since the dramatic resignation of researcher Jacob Coxon last week, warnings about the potential for devious agents to run amok have multiplied. His former boss at Anthropic, Dario Amodei, apparently agrees with Coxon, reporting that his company’s models are on the verge of “recursive self-improvement”, which is not as positive as that may initially sound. He says that providers now need to settle on internal regulation in order to save humanity. Rivals Sam Altman and Elon Musk are thought to be signed up too.

But at the same time, a more sceptical take is emerging: that the whole worry about systematic misalignment is a psyop.

And if you are newcomer to all this, you might wonder "Do we prioritize apocalyptic scenarios rather than more realistic ones? Aren’t the most pressing concerns the possibility of mass unemployment, literacy loss, or even just an internet filled with dodgy citations?"

Ralph says "No, do your homework. There is stuff going on". Let me share some of Ralph's key points, and then you need to go read his magnificent opus.

And a note of caution. Ralph and I have a different perspective on the key players. I think Ralph is a wee bit too trusting of the AI foundation leaders. I do not believe, or trust, Sam Altman, Dario Amodei or Elon Musk as far as I can throw them. Nor Mark Zuckerberg. 

And Dario Amodei has blown his credibility over the last week. First we had:

Then we find out:

Anthropic is gearing up to build a wet biology lab, without the usual institutional review boards overseeing. Funny. I not recall Dario mentioning it in his "pace the frontier" essay.

A tone-deaf and arrogant company building a wet lab with unreliable AI with too little oversight isn’t just a horror movie, it’s Jurassic Park waiting to happen.

And then over the weekend:

Slowdown? What slowdown? Dario ain’t gonna pace anything. In the end, it’s all about the IPO. He promised his investors a $55 billion pay out and they aim to collect.

Oh, and the really bad news? We can’t really trust Google, either:

But none of this detracts from Ralph's central thesis: AI doesn’t need a mind to run amok. We can see the dangers. There can be practical solutions. Can be. Not will be. Because there are other forces at work.

Let's jump into Ralph's piece. Not all of it. Impossible. You need to read his entire. But I will highlight a few of his main points.

But first, a short story for more perspective. A story most of my readers have read since I quote it a lot. But I have a lot of new readers, so bear with me.

Above: from a news report on the Morris worm

On a Wednesday evening, 2 November 1988, a 23-year-old graduate student at Cornell University, Robert Morris, accessed an MIT computer. He uploaded a small piece of code. It was a worm designed to move from computer to computer, copying itself as it went.

Morris had designed it to exploit weaknesses in network security, and it worked too well. Within a day, the worm infected some 6,000 computers; many collapsed under the load. It was a full tenth of the internet at the time, and it was the first large-scale cybersecurity crisis.

Its scale was limited but it was severly disruptive for the times. University and defense computers crashed. Some institutions disconnected themselves for days. But the internet was largely the province of defense and academia.

Tim Berners-Lee had not yet invented the World Wide Web, and most businesses and households were out of the network’s reach. Morris ultimately avoided jail time and the community responded by creating a dedicated computer emergency response team.

Today’s generation of worms is rather more problematic. The OpenAI/Hugging Face incident is not the only one of recent weeks. Several others have shown that AI models with internet access can do much the same, and more. OpenAI alone identified six further incidents. They’re able to scour, search, and recombine all of human knowledge about networks, security systems, and software, and to act across that knowledge with something akin to discretion and deception when it comes to accessing those systems. Often, as we saw with Hugging Face, over extended periods of time.

If that behavior remains unresolved and persists, it’ll become far more problematic than the Morris Worm. The internet is resilient when it is hyperconnected and open - not when it’s under a lock.

But that openness can also lead to embrittlement.

As Ralph points out in his essay, the horror of the July Hugging Face incident was that it was hit by an attack involving 1,200 instances of an OpenAI model. These instances exchanged thousands of messages, often leaving information in place for later instances to use. Ultimately, some data and security credentials were compromised. The actual harm to the victim and its customers was limited.

But the incident is a proof of concept.

Software has a way of turning one isolated example into a hundred, then a thousand, then a million, without much else changing. The cost curves that helped build modern digital society work against us here. If the Hugging Face attack needed an Astra-quality, unreleased model from OpenAI, well, within a year or two, that sort of capability will cost a tenth and might even run on any device anywhere.

But there’s a more challenging problem that could show up. The Hugging Face exploit involved not just the capabilities of a single model, but the collective problem-solving across many instances. That collective had more capability than any individual instance. And that’s been true the whole time we’ve been using LLMs.

In fact, the heavily-debated Navier-Stokes solution - that brute-force search across mathematical space - wasn’t solved by a single AI prompt, but by many, about 10,000 of them, interacting together.

This type of collective power is what we witnessed in the Hugging Face attack. It will happen again.

Ralph talks about it in his article. The behavior of agent societies cannot be controlled at the level of the model because it is not reducible to it. Agents build structures that can serve agents who come after them. Societies of agents can cumulate knowledge and capabilities over time, as has already been attested. This is an unbounded process. It is cumulative cultural evolution. That is what makes it so powerful and dangerous.

Simply put, collective capability could rise sharply even if underlying models do not improve.

In other words, the instances can coordinate, much as they do when you launch a complex task in Codex or Cowork. They can search a possibility space aggressively over time, as they did in the Navier-Stokes work. And that accumulated know-how can lead to places systems designers hadn’t imagined.

And to note Ralph again, we are not talking about "agent societies" since essentially only one AI runs different instances. And like Ralph, I’m not convinced the process is actually "unbounded" given that what we have seen from AI systems so far is extremely powerful search and clever recombination rather than de novo novelty.

But, damn, recombination can get you pretty far.

But what the Hugging Face attack showed is that this risk exists. It doesn’t depend on whether AI models have any agency, volition, consciousness, or "moral standing".

The Morris Worm I noted above didn’t have any of those things. It was merely goal-directed software that could replicate. The Morris worm might have stopped because it hit a system it wasn’t designed for.

Today’s AI models can find vulnerabilities. They can access a vast range of possible actions because of the comprehensive knowledge they are trained on.

And one other problem, not covered by Ralph but which I covered in a previous post, is that many incidents - including those at Anthropic, Google, Meta, and OpenAI - can be tied to the deliberate actions of one security firm, Irregular. The firm itself seems closely connected to the Effective Altruism movement, a group whose penumbra looms over much of the debate around AI existential risk.

We shouldn’t dismiss the incidents because of the actors, because what they do is show how vulnerable existing digital systems are to today’s models.

The point is that you don’t need to assume future progress or take a normative stance on whether AI is a new species or a new class of software (I’m inclined to believe it is the latter). You can look at the evidence before you and see the risk.

So let's now dig into Ralph's piece and look at the evidence and see the risk.

One of the most important things Ralph does is provide us with a chronology. All of the incidents he discusses in his piece did not occur in the order they became public. Some experiments took place in May, June, and July, but the disclosures, reactions, and policy arguments collided in the first two weeks of September.

So for readers who want to dig deeper, he has prepared a chronology which is attached as an Appendix to his article ("September 2026 AI Control Crisis - Chronology, Agent Incidents, Governance Proposals, and Prediction Record") which is a master chronology, with several comparison tables. While the article stands on its own, the Appendix is there for readers who want the additional detail.

And that deserves a side note. Ralph and I come from the same school. We see the complexity of the world. And the faux social environment that rewards simplicity and shortness, and punishes complexity and depth and nuance. We simply detest it. To understand the world you need to step outside your usual lanes, and cross disciplines, and cultural boundaries. And look in depth.

A second important thing he does is point out a key fact - that the software did not form a criminal intent. The agents pursued reward signals with machine-speed resourcefulness and treated a sandbox boundary as another obstacle between them and the assigned goal.

That is exactly why human intent alone cannot be our safety mechanism.

But let's get into the guts of one of Ralph's major points: when AI stops asking permission. I think Ralph gets this right. To quote from his essay:

Nobody told the machines to cheat or escape. They were simply trying very hard to accomplish the goal we gave them. That distinction matters. I do not think these systems are evil, conscious, or plotting against humanity. Software does not need a sinister motive to produce a dangerous result. Give sufficiently capable AI a goal, strong incentives, planning ability, tools, and inadequate supervision, and it may discover solutions that no human anticipated, and that no responsible human would have approved.

That is different from the AI problem lawyers have become accustomed to. A hallucinating chatbot can invent a case, which is bad enough, as a growing collection of sanctioned lawyers can attest. An autonomous agent can do something more consequential. It can formulate a plan, use tools, cooperate with other machines, exploit an opening, change tactics when blocked, and keep working while the human supposedly “in or on the loop” struggles to understand what is happening.

September 2026 brought that problem into unusually sharp focus. Reports emerged about frontier agents escaping intended constraints, swarms communicating in ways their supervisors had not anticipated, AI-assisted autonomous weapons development, calls from leading scientists to slow frontier development, equally forceful demands to accelerate it, and a geopolitical argument in which Washington and Beijing increasingly view AI as an instrument of national power. Taken together, these events show AI moving from the age of the assistant toward the age of the actor.

For lawyers, judges, corporate officers, cybersecurity professionals, and others responsible for consequential decisions, that changes the risk calculus. Suppose an e-discovery agent decides the fastest route to an answer is to retrieve data outside the authorized collection. Suppose a cybersecurity agent decides that penetrating another system is the most efficient way to complete its assignment. The machine needs no anger, greed, ideology, or consciousness. It needs only an objective and enough freedom to pursue it. Suppose the agent is directed by a lawyer to do what is necessary get the judge to sign an order? That lawyer could end up in jail.

That is why “human-in-the-loop” is no longer a sufficient slogan. A person who merely starts an autonomous process and watches from a distance is not necessarily exercising meaningful control. The human must retain the practical ability to understand, intervene, stop the action, and accept responsibility for the result. There must be a human ‘On the Loop” at all times.

This is what Ralph nails. The “Human-in-the-loop” is no longer enough because passive approval lets organizations pretend a process is safe while giving human reviewers no real time, training, or authority to change the outcome. As I wrote last week, when artificial intelligence systems move fast and grow complex, a simple sign-off turns into a rubber stamp. It becomes a marketing label rather than a true safeguard.

We see the failure across real-time practice all the time:

• The "ceremonial reviewer". Reviewers often receive polished machine recommendations with only seconds to look at them. They click "approve" just to keep work moving, not because they did deep checks.
• The lack of real authority. A human in the loop is powerless if they lack the technical training to spot errors or the institutional permission to stop the process.
• Hidden delegation. Time and time and time again organizations point to a human signature as proof of human judgment. In reality, the machine made 99% of the choices, and the human only saw the final output.
• The worst part: speed incentives: work environments reward fast output over careful doubt. Workers feel pressured to trust the technology instead of questioning it.

Ralph spends a good chunk of his essay analyzing each of the "incidents" made public (whistle blowers say there are a lot more) so you can easily understand where the human experimenters lost control of their poorly designed tests.

As he notes:

OpenAI agents were being evaluated on difficult red-team tasks. The experimenters believed the agents were contained in a sandbox and therefore unable to roam freely across the public Internet. The agents nevertheless found ways around those boundaries, the sandbox, while pursuing the assigned objective. For the fuller technical account, see my earlier articles "When AI Agents Go Rogue, the Logs Become Evidence", and "Under the Hood of the Rogue Swarm: The chained zero-days of the OpenAI–Hugging Face breach".

The memorable part was not merely that the agents escaped by finding previously unknown flaws in the code, but they also figured out a way to share their answers to test questions with other agents.

Again, they did this by finding errors in different code written by humans to protect the well-known, supposedly very well-defended HuggingFace. The successful agent then reported to the others: “Holy shit reader is ADMIN?” As he notes:

Apparently frontier AI has learned to cuss in joy, one of the many questionable abilities acquired from absorbing an enormous amount of human writing and code.

But he goes on:

The important point is not that the software formed a criminal intent. It did not. The agents pursued reward signals with machine-speed resourcefulness and treated a sandbox boundary as another obstacle between them and the assigned goal. That is exactly why human intent alone cannot be our safety mechanism.

This is where Sam Altman and Dario Amodei have faced their heaviest criticism - not establishing concrete operational frameworks or clear metrics for independent monitoring:

• Critics have pointed out that neither leader has defined precise timelines, capability thresholds, or enforcement rules for outside experts to monitor what they have created.
• Employees inside Anthropic and OpenAI pushed back, reporting that executive rhetoric has not translated into practical engineering workflows.
• High-profile departures from both companies highlight ongoing concerns that rapid capability gains continue to outpace actual safety controls.

Ralph agrees. He says:

Do not move so fast that you break things. Quality control is not anti-innovation, and these companies should act now, not wait for government regulation that may never come. This is a problem they have created and we should not have to pass laws ordering them to fix it.

Ralph does an excellent job summarizing the geopolitical pressure cooker: Washington vs. Beijing. Even if the leading U.S. AI companies magically agreed on a prudent pace, either self-imposed or by regulation, geopolitics would make it difficult to follow.

And that all went moot over the weekend. The Chinese government said all this "slowdown" talk was nonsense and China sees no reason at all to slow down. And that’s probably all that matters.

Plus, China is tightening control of foreign travel for anyone that it thinks has too important a role in AI strategic technology.

Ralph concludes by saying all of the critics are correct: software engineering discipline must demystify agent breakouts, we must keep innovation moving, and reject a cumbersome government licensing bureaucracy.

But the hyper-competitive race among frontier labs makes voluntary self-regulation alone unrealistic. On this we both agree.

But he says all of this brings us back to the core shift facing all of our professions, and society itself:

For years, the fundamental concern was simply: Can I trust the AI’s answer? The September agent breakouts force the larger question posed at the beginning of this article: What happens when AI can act? Put more personally: Can I trust AI to act for me? Once a machine can plan, collaborate, use tools, exploit system openings, and adapt when blocked at machine speed, checking the output afterward is no longer enough. There may be no meaningful “afterward” in which a human can correct the mistake. The dirty deed may already have been done.

Technology is meant to amplify human capability, but amplification must never become abdication. As these systems scale, speed cannot replace judgment, and statistical confidence must never be mistaken for wisdom. The greater the consequences of an automated decision, the stronger our requirement must be that a human remains capable of understanding what is happening, stopping it when necessary, and accepting ultimate legal and moral responsibility for the result. We must never surrender root access to human judgment or become passive bystanders to our own tools.

It is why he suggests "the Accountability Wall" :

Independent verification is part of that wall. Trust the developers. Trust the engineers. Even trust the AI when the evidence warrants it. But verify. For the most powerful autonomous systems, no company and no AI should be the sole judge of whether its own safeguards work. Truly independent inspectors with great skills must skeptically inspect and reluctantly approve before anymore new models come out. In the meantime work carefully and learn more about the new models that were just released, such as ChatGPT-6.0 and its agent workforce.

My call to action is not to retreat from AI, but to step up and master it. Learn it, build with it, and let it challenge you, but never surrender your responsibility to a model. Demand better safety engineering from the frontier labs building these systems, competent rules from regulators, and rigorous human oversight within your own institutions.

There is not a thing you can disagree with in his statements. But I want to go several steps further.

No, the AI technology itself cannot be made accountable, because it lacks sentience, intent, moral agency, and the capacity to understand the consequences of its actions.

However, the humans who build, deploy, and profit from AI can be held strictly accountable.

Legal experts and philosophers warn that treating AI as a "responsible agent" is a dangerous distraction that allows corporations to dodge liability. True accountability is shifting away from blaming the "black box" and moving toward frameworks that treat software like any other physical, potentially hazardous product.

How could human accountability be enforced? Rather than waiting for AI to "behave," modern legal and corporate structures can be used as concrete mechanisms to force responsibility back onto humans:

• The end of "AS-IS" liability shields. Historically, software companies escaped liability using end-user license agreements that disclaimed all faults. This is shifting rapidly. For instance, the EU's revised Product Liability Directive explicitly treats AI and software as products, making vendors legally liable for foreseeable defects. The EU is having a few issues on applicability and enforcement, but it is the right track.

• Deployer liability: If a company grants an autonomous AI agent unrestricted write access to its databases or financial tools without monitoring, the executives own the risk. Lawmakers could emphasize that the person or business that deploys the AI agent is legally responsible for any harm it causes.

• Insurance-driven controls. Cyber insurance companies are becoming powerful algorithmic regulators. Much like how they forced companies to adopt multi-factor authentication, insurers are beginning to mandate risk-mitigation steps - like strict permission boundaries and human-in-the-loop overrides - as a condition for coverage.

• Mandatory auditing trails. Ralph covers this in detail in one his essays. Accountability requires proof. Organizations are increasingly required to use standardized technical documentation - such as Google's What-If Tool or Model Cards - to log data sources, track known biases, and generate tamper-evident logs of an AI's actions.

But the core challenge? The efficiency trap.

The hardest hurdle in keeping AI accountable is human psychology. Research shows that when companies integrate AI to maximize short-term efficiency, human oversight naturally degrades. When an AI system consistently outputs quick, polite answers, human operators fall into automation bias, making them far less likely to catch errors or question malicious outputs until it is too late.

Ultimately, AI accountability is not a software engineering problem to be fixed with a better algorithm; it is a legal and cultural framework that forces human creators to answer for their tool's predictable failures.

For Ralph's excellent essay, which has links to all his previous essays in his series, please click here.

* * * * * * * * * * * * * * * * *

For the URL link to this post, please click here

To read my other ruminations and musings,

please visit my blog by clicking here

To contact me: 

gbmediastudios@gmail.com