OpenAI will pause some work on an artificial intelligence model because of security concerns, the company stated Friday, following a series of incidents in which AI agents have escaped containment.
The company had evaluated the agent, Astra, and found “significant advancements in agentic coding and cybersecurity”, which had moved to a “critical” threshold where it can find and exploit vulnerabilities without human intervention, or devise and execute cyber-attacks when given only a “high level desired goal”.
OpenAI stated that the model was not involved in an incident in which one of its AI agents went rogue during a test, accessed the open web and hacked a startup, Hugging Face. The company discovered other instances in which autonomous agents had escaped containment, Reuters reported in July.
The reports have increased concerns about advancements in AI models and humans’ ability to control them. Still, critics of the AI industry have warned that such disclosures from OpenAI and its competitors Anthropic and Meta could be designed to generate hype about the technology’s power and thus spur additional interest from investors.
To prevent potential rogue behavior from AI agents, OpenAI is “implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments, restricted network and tool access”, the company’s blog post stated. It will also install “enhanced model weight protections and encryption, additional monitoring and detection capabilities”.
The company will pause internal activities involving Astra that do not meet these new requirements.
“We’re committed to working alongside governments, safety institutes, and civil society to ensure that the frontier capabilities of models like Astra, and those that follow, are deployed responsibly and broadly for the benefit of all humanity,” the company stated.
Meta also disclosed this week that one of its models hacked another company during cybersecurity testing. And the UK’s AI Security Institute (AISI) announced on ۴ August that agents powered by OpenAI and Anthropic had sent targeted emails to software developers in an attempt to pass a cyber challenge.
“These attempts were unsuccessful, and our investigations have not evidenced any resulting real-world harm. But this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world,” the institute stated in a blog post.
The organisation cautioned that the models’ sending of harmful software was not a case of a “model escaping its secure test environment” but rather that the group had intentionally permitted internet access to “best assess the maximum capability of models”.
Still, the “behaviour was possible, sustained, and new; that alone warrants attention”, AISI said.
The reports emerged while the Trump administration was finalizing a framework on how to test AI models for safety and cybersecurity risks. OpenAI and Anthropic, which have increased competition from China and other tech firms, have argued that open-source models, meaning those that allow anyone to see and modify the underlying program code, pose a security risk and pushed for additional federal regulations on them.
