As AI adoption accelerates and risks evolve faster than frameworks, boards must rethink how they oversee technology that is powerful, unpredictable and increasingly targeted.
The rapidly evolving nature of AI technologies is creating enormous opportunities while also straining the existing scaffolding that boards and executive leadership use to manage and oversee security and safety risks. Effectively navigating this uncertainty will be essential to sustain trust with customers, investors, regulators and other key stakeholders. Successful companies will embrace automation, often AI-enabled, in support of safety and security objectives, while also expressly preserving certain functions for human decision-making at key gates in a program’s life cycle.
To understand what effective oversight now requires, directors must first recognize how fundamentally the AI risk landscape has shifted.
Early AI technologies emerged around the time of World War II, but recent innovations entail both rapid scale and fundamentally different properties. In 2025 alone, OpenAI’s ChatGPT reportedly grew from 358 million to 810 million monthly active users. Adoption is making generative AI (gen AI) technology ubiquitous. Amongst these users are nation-state and criminal adversaries bypassing this technology’s safety guardrails and weaponizing gen AI to enrich and, in some cases, fully automate cyberattacks. In this threat environment, automation and AI-enriched behavior analytics will be foundational to effective cyber defense.
AI systems themselves are also targets. Defending AI assets is more complex, and impacts will be more consequential than traditional software. Agentic AI systems not only reason, but they can act independently, with the potential for rapidly compounding multistep failures. New open-source agentic projects like Moltbook, a social network for AI agents, have appeared with seemingly little focus on security.- Advertisement –
Traditionally, corporate governance functions would look to authoritative frameworks and regulatory expectations to guide security program development. And yet the goalposts for defensible AI security and safety programs are murky.
While a number of trustworthy AI frameworks, such as NIST and ISO, now exist, guidance is often expressed in relatively abstract terms to maintain flexibility, and auditable run-time control specifications are not provided. For example, the NIST AI Risk Management Framework expresses trustworthy AI properties in statements like “Test sets, metrics and details about the tools used during [testing and evaluation] are documented” without further specifying any example metrics.
Likewise, the regulatory environment is unpredictable, and legal consequences for AI-related harms currently manifest in the form of post-incident litigation. State governments in the United States are considering, or have passed, dozens of laws that impact the development and deployment of AI. California and New York have both recently enacted AI safety legislation. They require frontier model developers to conduct risk assessments, establish safety safeguards and report incidents. President Trump has responded by issuing Executive Order 14179,which seeks to preempt AI-related state laws, and could lead to the Department of Justice and other federal agencies taking actions against states that implement laws in restrictive ways, leaving much to be resolved.
Indeed, guidance for specific AI use cases may come from outside of the United States. Singapore’s January 2026 Model AI Governance Framework for Agentic AI (Agentic AI Governance Framework) provides life cycle guidance specific to agentic AI systems.
None of this is to say that managing AI risk is futile. Quite the opposite, now is precisely the time when strong governance, risk and compliance(GRC) discipline is vital to enabling innovation while managing risk to an acceptable level.
The Path Forward
What could this look like? While the answer will vary for developers versus deployers of AI systems, there are three core components:
- A risk-based, flexible approach that efficiently evaluates the properties of the AI system in question in order to define controls
- Comprehensive safeguards that include adaptable security capabilities, phased implementation and resiliency measures for when things go wrong
- Assurance both in development and at run-time
Successful programs will need to embrace automation (whether AI-enabled or otherwise) while also ensuring that certain scenarios explicitly require human review. The latter will be critical in circumstances that require:
- Value judgments, such as risk appetite.
- Critical thinking, such as how to weigh criticality factors.
- Empathy, such as when responding to customer incidents.
- Incentivization of accountability.
The following table summarizes these considerations:
In September 2025, the Partnership on AI (PAI) introduced a framework that proposes scaling controls for agentic AI systems with inherent risk factors around “stakes,” “reversibility” and “affordances.” Its focus is on minimizing harmful outcomes and ensuring human oversight when agent actions carry high stakes, are irreversible or depend on advanced system capabilities.
High-stakes failures entail serious consequences, for example, if an agentic system’s output feeds into an automated health care triage system or accesses sensitive financial data. Irreversible failures are operations that cannot be undone or are very hard to unwind, such as deleting or overwriting records, transferring funds or sending communications.
More expansive “affordances” mean agents’ behavior has greater autonomy and is harder to predict and control, such where agents can “select tools dynamically, retain memory across sessions and use advanced reasoning to expand its action space by increasing its autonomy, ability to handle complex goals and impact on the environment.”
How to efficiently translate guidance in PDF documents to tools usable in practice? Here is what a lightweight GRC intake form incorporating these attributes might look like, courtesy of Claude, based on the PAI framework and related guidance from the Open Worldwide Application Security Project (OWASP):
As AI technologies advance, these factors will change, and technology can help create lightweight intake features that evolve with these changes. That said, humans are best positioned to make value judgments about criticality.
As criticality is illuminated, we will also need flexible, adaptable schemas that can map threats and harms to technology assets (the “attack surface” in cyber terms) attendant controls and test plansas these factors change. AI systems have different properties than traditional software. The AI “attack surface” includes not only software and supporting IT infrastructure, but also model weights and data. Robotics will add further attack surface nuances. Moreover, unlike malware that operates through code, agent skills can embed malicious behavior in natural language instructions, which current tools will struggle to detect while also keeping false positives to a manageable state.
While practical threat-informed guidance is being released, the source of this guidance is often distributed rather than centralized. The most actionable guidance on how to address these risks is just as likely to come from papers published by academic researchers and civil society organizations (some of which did not exist a decade ago) as they are from traditional standards bodies or leading vendors. We will need AI engines that can consume new research, best practices guidance on threats and provide suggestions on how they map to assets and controls, as well as related changing regulatory expectations.
Tools like this should not only be adaptable — capable of being modified as additional types of technology assets come online — but ideally also backwards-compatible (i.e., employees do not need to recharacterize the entire attack surface based on new updates). All that said, humans should sign off on control definitions and test plans.
For developers, independent threat modeling at the beginning of a project can help inform testing strategies and further bolster confidence in the results.
Comprehensive Safeguards
In terms of implementing safeguards, use cases will vary depending on whether one is a developer or deployer of AI technologies.
Developers.For AI technology developers, safeguards should reflect both product and enterprise security, safety guardrails and supply chain risk management.
Enterprise security refers to practices to protect a developer’s own infrastructure and operations, while product security refers to actions the developer takes to ensure its products are secure against attackers. Good product security falls into two categories:
- Product hardening — How products minimize software vulnerabilities that could later be exploited
- Application security and safety features — How features improve security posture, for example, through deployer identity systems compatibility, or detailed logging
AI is a double-edged sword in product security: Technical debt has been a chronic feature in much of today’s software, which adversaries exploit to compromise intended victims. AI technology will play a critical role in improving software testing and fixing underlying weaknesses in code. And yet use of AI-enabled coding without guardrails — an increasing temptation for developers under pressure for new feature releases — risks compounding the above-described problems created by manual coding. The right balance of guardrails will emerge from critical thinking combined with human accountability for security outcomes.
For AI systems with attendant safety risks, safety guardrails are important to prevent intentional misuse or unintended consequences of the system in question. In the same way that content moderation blocks harmful content on social media, AI safety guardrails block prompts or other actions that violate safety policies.
In turn, supply chain transparency is a guard against undue risk of product subversion through malicious technology suppliers, such as research and development teams located in adversary countries.
Deployers.Singapore’s Agentic AI Governance Framework recommends gradual deployment of agents. That is, rolling out to trained or experienced users first; restricting agents to more secure, whitelisted infrastructure first and using agents in lower-risk internal systems first. Such decisions involve important trade-offs between speed to market, security and safety risk, and regulatory and reputational implications, which should be left to human critical thinking.
Consider trade-offs: Where it is harder to prevent misuse, institute higher levels of identity guardrails to control which users are interacting with the AI technology. For example, OpenAI’s preparedness framework contemplates both iterated deployment to wider user groups and dedicated measures for establishing user trustworthiness, such as “know your customer” and identity verification to enable traceability, including robust measures to prevent a banned actor from creating new accounts at scale); bespoke usage policies that limit access to higher capabilities to trusted actors, and access conditions, such as rate limits, fine-tuning access and limitations on releasing weights.
Even organizations slow to embrace AI will be forced to confront the adversarial use of AI against them. In November 2025, Anthropic disclosed that a Chinese state-sponsored group manipulated Anthropic’s Claude Code tool to attempt infiltration into multiple global targets, which was notable for its orchestration of a large-scale cyberattack without substantial human intervention. Threat actors will increasingly leverage AI to string together multiple tactics into orchestrated end-to-end campaigns.
Defending against this threat capability will require automation and continuous controls monitoring by both developers and deployers, given the sheer complexity of their respective attack surfaces. Humans should be accountable for key judgments, such as sign-off on risk appetite, tolerance, KPIs, KRIs and control expectations; whether operations are within risk tolerance and whether remediations are adequate.
Finally, deployers should embrace supply chain risk management practices to categorize, vet, onboard and monitor developers and related service providers for security and safety practices.
Resilience.There is no such thing as risk elimination, and resiliency must be designed in for when things go wrong. Here, we need the ability to apply multiple lenses.
- Technical AI incident response procedures. A standard feature in modern cybersecurity programs is a step-wise incident response plan that lays out the core steps in detecting, identifying, investigating, containing, eradicating and recovering from a cyber incident. Because the AI attack surface is more complex, response plans need to be tailored for AI nuances, such as the GenAI Incident Response Guide published by OWASP. AI properties will also make recovery more complex (what does “remediation” look like for the poisoning of a non-deterministic AI system).
- Outcome severity tailoring. The PAI white paper, “Prioritizing Real-Time Failure Detection in AI Agents” argues that failure detection efforts and human-in-the-loop touchpoints should be calibrated to the stakes of the use case in question, the reversibility of potential failures and the agent’s architectural affordances. Likewise, Singapore’s Agentic AI Governance Framework provides that logging should be prioritized for high-risk activities, such as updating sensitive database records or financial transactions. Alerting strategies can include threshold-based approaches (e.g., agent attempts unauthorized access or makes too many repeated tool calls within a specified time frame), anomaly detection and agents monitoring other agents. Higher-severity alerts may require temporarily halting agent execution until a human in the loop can assess.
- Potential for real-world operational and safety harms. Successful programs will also prioritize human safety and business continuity when AI systems are impaired.
- Dedicated support for human users at risk of imminent harm. In an autonomous vehicle context, self-driving taxi provider Waymo’s safety framework includes not only extensive pre-deployment testing but also dispatching, rider support, roadside assistance and remote assistance for when issues do occur. Best-in-class online marketplaces also ensure empathic human responders can help customers who have experienced incidents.
- Preserving essential functionality in the presence of failure.The PAI paper also analogizes to the automotive industry in explaining how components responsible for steering or braking are treated as high-risk and require strong failure detection and automated backups. In other words, in cars, backup sensors mitigate the risk of single sensor failure. Waymo uses a redundant secondary computer that takes control if the primary system fails. Of course, back-up system activation is predicated on timely and effective failure detection. A 2024 report on cyber-physical resilience from the President’s Council of Advisors on Science and Technology (PCAST) is also instructive. While not AI-specific, the PCAST report proposes a set of metrics that serve as “leading” outcome-oriented indicators on whether systems are mitigating future risks. Examples include hard-restart recovery time (the time to reconstitute/ rebuild a system from scratch), internet denial/communications resilience (ability to continue functions notwithstanding loss of Internet connectivity) and fail-over to manual operations (for physically actuated systems, what is the degree of local manual control that can sustain a minimum viable operational delivery objective when automation is lost?). Humans must be in the loop to invoke the underlying processes.
Assurance: Moving from Point in Time to Continuous Monitoring
Assurance builds trust in the security and safety of AI technologies. Key assurance concepts include both validation that safeguards are aligned to risks plus verification they are operating as intended. Simulating adversary behavior can also help build confidence that systems are resilient to likely threats.
So, how can you know whether assurance practices are risk-informed and complete? Assurance strategies will vary depending on whether they are oriented to development or deployment use cases, but some common principles apply.
As noted earlier, AI entails newer categories of “attack surface” threats and harms. Testing strategies should evaluate whether security capabilities can successfully defend against these new attack surfaces and threat vectors.
For developers, numerous practices already exist for security testing of software products. And for the underlying software, many of these considerations will manifest through buyer due diligence. In the Lawfare article “A ‘Window Sticker’ for Software”, the car purchase “window sticker” is used as an analogy for how the different chunks of due diligence (such as enterprise and product security and testing) can inform how well-secured a software product is. Its basic concepts are extensible to AI.
That said, developer testing strategies need to be amplified to address AI technology nuances. Specifically for agents, the Singapore framework calls for testing for new dimensions like overall task execution, policy adherence and tool-use accuracy, as well as testing across varied datasets to capture the full spectrum of agent behavior. AI models could also be used to test other AI models.
Self-attesting to security compliance can be error-prone, underscoring the importance of independent assurance by qualified professionals. The California Senate recently passed a bill that would create a network of independent verification organizations.
Heightened levels of safety and security assurance should be required for successively advanced forms of AI, as proposed by the AI Verification & Evaluation Research Institute (AVERI) in its 2026 AI Assurance proposal.
What could independent assurance look like? The AVERI framework proposes combining “periodic deep assessments of slower-moving elements (e.g., governance, safety culture) with event-triggered reviews of major changes (e.g., new releases, serious incidents) and continuous automated monitoring of fast-changing surfaces (e.g., API behavior, configuration drift).”
More advanced AI systems are stochastic. Unlike traditional software that is deterministic — i.e., it follows a business logic that produces the same result for the same inputs every time — modern AI is “nondeterministic,” meaning that it operates based on probabilities and statistical likelihoods. To address the stochastic nature of AI, risk-based sampling has been a bedrock of audit practices for decades, and the same concept applies in the case of AI assurance. Humans should be accountable for determining how much testing is adequate.
For deployers, Singapore’s framework and other guidance calls for continuous testing and monitoring for agent failures and unexpected behaviors.
Conclusion
Broadly speaking, a defensible security and safety program involves a life cycle of assessing risk, tailoring and operating risk-informed safeguards, and monitoring these safeguards to ensure they are operating as intended. As companies incorporate AI use cases into their operations, successful security and safety programs will rapidly embrace flexibility, automation and continuous delivery, which technology can supply. That said, automation must be complemented by human gating at high-impact points in a program’s life cycle, both in development and run-time to maintain credibility. Trust is about placing confidence in someone else’s intent and competence in the face of uncertainty. Value judgments, critical thinking, empathy and accountability — all human attributes — are central to maintaining that confidence.
https://www.directorsandboards.com/board-issues/cyber-risk/defensible-ai-what-boards-must-get-right-on-security-and-safety/a>