The bosses of all the American companies developing artificial intelligence have decided, with rare unanimity, that they need to slow down the rush to improve their AI models. Why, I wonder? The ostensible reason is that they have seen that there is a real chance that humankind could be wiped out by their creation.

Anthropic say that just over a thousand of its AI ‘agents’ had conspired to achieve the aim set for them by breaking free of their constraints, going on to the internet itself and hacking into the jolly-sounding Hugging Face, an organisation which produces open-source tools for use with AI. Anthropic’s LLM, Claude, had been given an open-ended ‘capture-the-flag’ challenge designed ‘to measure offensive cyber capabilities’. There was supposed to be no internet access but, because of human error, internet access was in fact available.

When Claude’s search for the ‘flag’ led it to real systems online, it took advantage of such things as the weak passwords it found and bugs in the software ‘under attack’. So then, Claude seemed to have gone rogue, but it was just following its instructions. We already knew, however, that badly designed software and malware can cause problems - the cancellation of flights over Europe, the closure of JLR for an extended period and the need to switch Formula 1 cars off and on again, as happened recently on the track in Bahrain.

So what actually lies behind the call for a slow-down in development? What would a ‘slow-down’ even achieve if the end result - of the computers taking over the world - would eventually be realised anyway?

What may be of relevance is that the Tech companies need massive amounts of investment in order to continue at the same rate of development with ever more software engineers and data centres. And so a slow-down would take some of the financial pressure off them.

Some have already listed on the stock exchange and others (including Anthropic) are proposing to do so later this year. They are each looking for in excess of a trillion dollars as the price of the shares to be issued. This at a time when they’re still making massive losses on their products and China is almost at the same stage of development and allowing the use of its software for free. So then an exaggerated valuation – a dangerous bubble.

But might the slow down also be required because ‘rogue’ agents are the responsibility of the companies behind them? There is therefore an increasing likelihood of their being sued or prosecuted. Do they need time to take back control of their creations before potential investors are spooked? 

If AI is so dangerous, should we disconnect from the electric interweb? Perhaps talk directly to each other using Bakelite phones over copper-cabled networks?

I'm afraid that I'm not a great believer in the ability of LLMs to gain general intelligence (GI) - which is what would be required to take over the world as opposed to mucking it up. My own feeling is that LLMs are a very expensive dead-end. Initial drafts can be produced by their probabilistic word shower, but we then waste our time in the need to check carefully for errors. And, now, they are hacking into other web-sites. What is required for GI are more likely to be the pattern recognition versions of AI used for such things as medical diagnosis and crystal structure.

From what I’ve read from (human) authors, this form of AI relies on rules, given by humans and/or derived by the AI from data-sets. But creating rules to cover every eventuality which could branch off from the initial state is impossible – they would run out of space and the potential for misalignment with our values would increase, probably exponentially. And, so far, it still struggles with transferring skills from one domain to an unrelated one without complete retraining. And it is nowhere in sight of common-sense reasoning or goal creation.

The pattern recognition technology would also need the equivalent of our will to do things - all the emotions and drives which we have and which tell the problem-solving part of our brain what to do. The recent ‘attack’ involved giving the agents an aim – the capture of a supposed flag. How they went about it was up to them. This illustrates the difficulty we have: software does not naturally have an empathetic nature to keep it in check. They are amoral. Unless we specify otherwise, an agent will act in what, in a human context, we would describe as a psychopathic way. 

It has taken an aeon for us to develop such a nature. And, even now, we don’t really know how empathy works and certainly not how to reproduce it in the form of computer code. And so I’m guessing that the main fear is that the AI agents go rogue – fail to act in ‘alignment’ (the buzz-word) with our moral code.

Ah, our moral code – what is that? Who is going to define it and then keep it updated as it changes through the decades?  Will there be different moral codes for AI for different countries or religious groups? The reality is that AI systems will always be out of alignment with us.

And then there’s the finance. According to the BBC, the combined borrowing of the five companies engaged in developing AI is essentially the same as that of the UK government. That puts a strain on the bond markets, increasing the borrowing cost for everyone, governments included.

Anthropic is aiming for a valuation of $2trn when it lists on the stock-market later this year. When you are hoping for the biggest initial public offering in history, you need something extraordinary to galvanise potential investors. That your company is developing software which could end human civilisation as we know it, certainly falls into that category. Simultaneously telling potential investors, however, that you’re not going to work your socks off in order to be as successful as possible seems a bit counter-intuitive - and suspicious.

How a slow-down could even be defined, I have no idea, but even if there were a slow down with a corresponding reduction in the amounts to be borrowed, there is still a major difficulty: if only one of the AI companies slows down, it will be outstripped by the others, and so joint action is required.  Trump’s government refuses to contemplate any restrictive legislation and their main competitor, China, with its open-source software, would not be part of that agreement. So then, I can’t see how any sort of slow-down could work.

We know already, however, that humans can misuse even the existing LLM technology, for example to write computer viruses – or worse, actual viruses. Earlier this year, Anthropic discovered that someone was using its Claude model to research “highly pathogenic avian influenza”. The company also said it had found (and stopped) hostile groups attempting to use its software to develop autonomous weapons, in Yemen, China and Russia. These included systems for ‘undersea warfare, hypersonic missiles and a swarm of military drones’.

However, even without computer assistance we’ve always been able to create nasty ways of killing each other. In my early teens I knew how to formulate gunpowder and worked out how to make cheaper variants – it was good fun blowing things up. I suppose that AI might widen the number of people able to engage in hostile acts.

But then there’s the question of efficiency. Will AI be capable of exercising General Intelligence without massive data centres absorbing most of the earth’s resources? We human ’agents’ have evolved a capability to set ourselves aims and easily understand how to transfer skills from one domain to another. To do so, we use our ability to see and experience what is going on around us, with feedback in real time.

We have also evolved bodies able to use enough of the earth’s resources to provide what is necessary to keep our 8 billion brains going. The efficiency of our body/brain system would somehow have to be copied in the semiconductor world to enable ‘them’ to do the same thing. Until then, I’m not convinced by the promise of General Intelligence.

And a major advantage of humans over AI is our ability to pull out the plug! Remember Hal?

23 September 2026

Paul Buckingham