Artificial intelligence company Anthropic on Thursday suggested a global pause in building the most powerful AI systems, as the latest models begin to show signs they could escape human control.
The San Francisco-based company, which makes the Claude family of AI models, said in a report that a global slowdown in advanced AI development would “probably be a good thing” – but warned that if just one company stopped, its competitors would simply rush ahead.
“We think it would be good if the world had the opportunity to slow down or temporarily pause the development of AI to allow societal structures and the search for alignment to keep up with the progress of the technology,” he said.
Getting a real pause in work would mean that several large AI companies in multiple countries — including the United States and China — would all agree to shut down at the same time, under rules that everyone could actually verify, Anthropic said.
“Without a global coordination mechanism, businesses and governments will have to make difficult security decisions while being subject to competitive and geopolitical pressures,” he said.
The company has faced pushback from others in the industry — and White House officials — who say focusing on worst-case scenarios overestimates risks and amounts to a strategy to slow down competitors under the cover of security concerns.
Nonetheless, the White House has recognized the power of the company’s Mythos model, which has not been made available to the general public due to its cybersecurity capabilities and is currently deployed in only a small number of vetted organizations.
The proposal would face an uphill battle in Washington and Silicon Valley, where U.S. officials and tech executives have repeatedly asserted that any slowdown in AI development risks giving China a decisive strategic advantage in what many see as the defining technology race of the century.
However, US President Donald Trump said he discussed the possibility of cooperating with China on AI security issues during his recent visit to Beijing.
Trump also signed an executive order this week that gives the government 30 days to conduct a preliminary review of America’s most powerful AI models before they are released.
“The human role is shrinking”
Anthropic likened the problem to nuclear arms control treaties – but said it would be even harder to master, because AI training is much easier to hide than a missile silo, and the temptation to continue quietly would be enormous.
The company said it plans to bring together government officials, scientists, advocacy groups and competing AI companies in the coming months to determine how such a system might work.
The call for coordination comes with internal data showing that AI is already significantly accelerating the development of AI itself, Anthropic said.
This acceleration creates a feedback loop that Anthropic says could eventually lead to what researchers call “recursive self-improvement.”
It’s the idea of an AI system that becomes capable of teaching itself to become more intelligent, without much human help.
“We are not there yet, and recursive self-improvement is not inevitable,” the report said, while adding that it could arrive sooner than most governments and institutions are prepared for.
“Evidence suggests that the human role is reduced at each stage of the AI development process,” the company said.
