The CEO of the artificial intelligence company Anthropic issued a new appeal on Saturday for the AI industry to “slow down” and offered a three-part plan for doing so, saying that his company would “unilaterally” commit to the first of the steps.
In a post on social media, Dario Amodei shared a link to an essay titled We Must Pace the Frontier in which he lays out how Anthropic would provide “third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training”.
The move comes after a former Anthropic researcher warned on Wednesday that AI could precipitate human extinction by ۲۰۳۰. Researcher Jacob Coxon said in a series of posts that he had quit his job because Anthropic and his previous employer, OpenAI, were ignoring or mishandling their response to the threat AI posed.
“Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon wrote. “The people building AI earnestly believe that it could kill us all by the end of the decade … No other human activity poses this level of danger.”
An Anthropic spokesperson said in a statement to the Guardian that the company had “always been transparent that AI will bring both enormous benefits and unprecedented risks” and it was building “models with some of the strongest safeguards in the industry”.
Earlier this year, Amodei published a lengthy essay titled The Adolescence of Technology that addressed some of fears surrounding the accelerating technology.
In his latest essay, he said that “carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity.
“But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious … A race to the bottom, spurred by commercial incentives, can make these risks more acute,” he wrote.
But, Amodei continued, “over the last few months, I have become convinced that fully addressing the risks requires even more prudence – not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up.
“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain,” he added in bold type.
Amodei also wrote that over the summer he’d seen AI “advancing drastically faster”, a dynamic called recursive self-improvement.
“Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” he said.
The executive also addressed the recent Hugging Face incident, in which a swarm of AI agents created by OpenAI acted as a “fanatically devoted collective conducting cybersecurity attacks on targets they were not asked to attack”.
Amodei wrote that simply dismissing the Hugging Face incursions because the OpenAI swarms appeared to have no malicious intent was not sufficient.
after newsletter promotion
“It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage,” he wrote.
Clément Delangue, the CEO of Hugging Face, wrote in response to Amodei’s Saturday letter that “it’s now clear that alignment is critical and won’t be solved behind the closed doors of a handful of frontier labs”. Delangue said Hugging Face had asked to be part of Anthropic’s “embedded evaluators” program.
He added: “Let’s make AI safer by making it more transparent!”
The three-step plan Amodei proposes includes building AI “at a balanced rate that aims to ensure its safety” by “ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this”.
The second step he proposes is to require industry-wide coordination, and the third is to ensure global coordination. “The steps do not need to be taken strictly in order, and some of them may be much harder to achieve than others,” he wrote.
Amodei said he continues “to believe that AI can enormously improve the quality of human life”. But he warned that “the measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.”
Responses to Amodei’s post were mixed across social media, with significant support coming from figures such as OpenAI researcher Aidan McLaughlin, who called the post “excellent” and agreed “with basically every word”, and Elon Musk, who simply said: “Dario is right.”
