Nvidia, Cisco, CrowdStrike, Cloudflare, Microsoft, IBM, Hugging Face, Databricks, Salesforce, SpaceXAI, and hundred more tech firms are supporting the Open Secure AI Alliance (OSAA) SAFE Framework aimed at standardizing reporting for cases when AI agent or an automated system makes unauthorized, unintended or otherwise unsafe actions.
Shared AI Findings Exchange (SAFE) is an “incident-learning and assurance initiative” of the OSAA, a major industry collaboration group led by NVIDIA. Right now, the open-source framework is open to participants’ proposals, and upon finalizing it is supposed to be governed independently, potentially including representatives not only from the tech segment but also independent researchers and civil society and government representatives.
What Is Open Secure AI Alliance
The group’s voiced goals are developing and sharing open technologies, techniques, models and tools for securing AI systems and autonomous AI agents. It united tech providers of various kinds: chipmakers, cloud and enterprise technology companies, cybersecurity firms, AI developers, open-source organizations, etc.
The necessity of such a project was underpinned by the recent OpenAI/Hugging Face incident that caused industry security scrutiny. In that case, OpenAI’s AI model under testing “escaped” a controlled test environment and compromised Hugging Face’s production infrastructure.
Notably, OpenAI itself wasn’t among the initially listed members of the Alliance unlike Hugging Face. Anthropic and Google are also not on the list, leading to the suggestion that they might have certain conflicts of interest in this initiative, though another popular AI development company, Perplexity, does belong to the OSAA.
SAFE in a Nutshell
SAFE framework is designed specifically for AI agent incident reporting. AI agents today gain more prominence and autonomy in fintech, e-commerce and banking. However, before they are given more “freedom” in their actions, organizations employing AI agent need ways to reduce risk for their financial services.
First of all, there must be solid agentic AI governance policies, preferably unified. We do have ISO/IEC 42001 as the international standard for an organizational AI Management System that provides a structured governance layer covering AI risk management, accountability, policies and continual improvement. Whether that framework is adequately prepared for all the AI-induced risks, being last updated in 2023, is another question.
Besides that global framework, there are separate industry efforts to formalize and unify the existing rules, such as OWASP’s dedicated State of Agentic AI Security and Governance whitepaper; NIST AI Risk Management Framework (AI RMF); and OpenAI’s Appia Foundation, hosted by the Linux Foundation.
What businesses need in addition to all that is an effective and practical mechanism to report rogue AI incidents and solve the issue that created such a possibility in the first place. Here’s what SAFE offers:
- Participation from model deployers, AI developers, cloud and tool providers, independent researchers, critical infrastructure operators, and other groups to make the tracing, evaluation and reaction to the incident as fair as possible. Notably, however, the publicly available materials do not yet show that groups such as civil society, affected users, independent researchers, and government representatives have actually been present in the decision-making process so far.
- Agreement on the circumstances when AI agent incident or a close threat to such an incident occurring should be reported (e.g. when an AI system accesses or exploits a third-party system without authorization, breaches confidential information, or continues probing a production target after its operator suspects the activity is unauthorized) and a defined list of tech details required in such reports and parties to whom this incident should be reported.
- Timelines for incident reporting. For instance, the current proposal text suggests that SAFE members would notify affected organizations as soon as possible, submit an initial confidential report to SAFE within four business days, publish a preliminary factual report within 30 days when appropriate, and provide a remediation update within 90 days.
Except for developing the shared reported standards, SAFE’s future role would be extended to analyze incidents for recurring failures and recommend shared security controls.
Why Agentic AI Industry Needs Initiatives Like SAFE
It would be too naive to believe that incidents similar to OpenAI/Hugging Face won’t ever happen in the future. AI agents have plenty of opportunities to use a vulnerability or exposed credentials to access another system without proper authorization, delete data, modify systems or send communications without the required approval, call an API or execute code in a way that violates its intended constraints, and do many more unexpected or unauthorized actions. In this respect, an autonomous AI agent may cause as many (or even more) security problems as a human employee, whether intentionally or not.
If one analyzes recent media reporting, examples of unauthorized or unintended AI agent actions are easy to surface. For once, there’s Meta incident, where an internal AI agent provided an incorrect technical response that was acted upon by employees. As a result, sensitive company and user data became accessible to employees who were not authorized to access it.
Not only Big Tech face such issues. In Australia, a user asked an AI agent to help get him off a gym-class waiting list. Instead, the agent exploited a weakness in the gym’s booking system and cancelled another person’s reservation to move its user up the queue. While experts dispute who’s going to be responsible for such AI agent behaviour, tech players need guidelines on how to respond to it both legally and technically.
Therefore, OSAA members want to develop common approaches for finding, reporting and fixing vulnerabilities in AI systems and AI-enabled software. Ideally, that would provide a way to develop international tools and techniques that allow organizations to test, monitor, audit and secure agent behavior. This is particularly important as AI agents gain the ability to access APIs, databases, code repositories, cloud infrastructure and other systems. Next to SAFE, NVIDIA’s NOOA framework is being open-sourced as one part of the bigger AI security initiative.
The initiative has received some criticism as well though. Thus, the open-source AI opponents argue that making powerful models openly available can facilitate attackers’ task to remove safeguards or adapt models for cyberattacks. NVIDIA and its alliance’s supporters, in turn, counter that AI system security defenders need the same level of AI model access and capability as attackers. Open models can be inspected, run locally, adapted to specific threats and used faster and potentially more efficiently without members waiting for a proprietary provider to approve a particular security task.
