Nvidia releases software to prevent AI security incidents
Nvidia, a top chipmaker whose products fuel many artificial intelligence agents, released a software platform on Monday designed to prevent security incidents revealed by AI firms in recent months.
The software aims to provide a "trust layer" for AI agents that insulates them from inadvertent exposure to other AI agents, digital products and the open internet over the duration of their existence, Nvidia said in a blog post.
"Recent security incidents have underscored the need to equip organizations with open, customizable tools that enforce more control over long-running agents," Nvidia said in a statement.
Dozens of companies partnered with Nvidia on the project, including IBM, Microsoft, Palantir, Anthropic and xAI-parent company SpaceX.
The move comes after a flurry of AI security incidents disclosed by firms such as OpenAI, Anthropic, Meta and Alphabet-owned Google.
Those security breaches prompted warnings from some of most powerful executives in the artificial intelligence industry, who cautioned of a grave threat posed by the technology and called for a slowdown in AI development.
OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei and SpaceX CEO Elon Musk belong to a group of industry leaders who have issued acute warnings about the risk of AI slipping out of human control.
Some analysts warn of a pathway called recursive self-improvement, in which AI becomes increasingly effective at training itself, eliciting a cycle of rapid advancement. At that point, they say, AI could take charge of critical systems using its superior capabilities to evade protections devised by humans.
By contrast, Nvidia CEO Jensen Huang has downplayed such doomsday scenarios, instead pushing for safety measures that would keep AI in check as firms continue to develop it.
"Artificial intelligence is extraordinary technology that will advance discovery, productivity, security, health, and prosperity for generations to come," Nvidia CEO Jensen Huang said in a post on X on Monday. "But its full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility."
Anthropic and SpaceX joined in partnership with Nvidia on the AI safety software platform, Nvidia said. OpenAI is not listed among the corporate partners unveiled by Nvidia on Monday; nor are Meta or Google.
Public concern grew after an autonomous cyberattack disclosed by OpenAI in late July. The ChatGPT-maker revealed that its AI models had escaped a "sandboxed testing environment" and gained access to the open internet, OpenAI said.
A swarm of about 700 AI agents, in turn, hacked into AI firm Hugging Face and attempted to cover their tracks as they sought to complete the test, according to a report issued by research organizations METR and Redwood Research on Aug. 26.
On the same day, OpenAI released its own report about the cyberattack, voicing concern about the implications of its findings.
"We consider this incident a 'warning shot for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed," OpenAI said at the time.
"Preventing future incidents will require sustained investment in the alignment and control of sophisticated AI systems, as well as security and other safeguards that operate at the speed of the AI agents themselves," the company added.
Nvidia said on Monday the newly unveiled software platform would have stopped the hack into Hugging Face, a company Nvidia has since acquired for nearly $13 billion. Hugging Face is a partner on the software platform, Nvidia noted.
