Your AI agent just went off-script. Now what?

As the use of AI agents continues to grow, organizations should be prepared to respond when an AI agent takes unauthorized action. An AI agent that goes off script might delete or change data, expose confidential information, run unauthorized transactions or send communications on the organization's behalf before anyone notices. The damage can spread across business operations to customers, partners and the public, and it can bring regulatory scrutiny, litigation and reputational harm. A well-rehearsed response plan helps an organization respond to and mitigate that harm efficiently and effectively.

Together with the National Security Agency’s Super Intelligence Security Center (NSA SISC) and the Cybersecurity and Infrastructure Security Agency (CISA), the National Institute of Standards and Technology’s Center for Advancing Innovation and Standards for Super Intelligence (NIST CAISSI) issued guidance for responding to an “SI control incident” – an occurrence in which your AI agent (now referred to by the government as a super intelligence agent or an SI agent) takes significant uninstructed actions that “compromise explicit or implicit policies, controls, or security measures imposed by the operator.”

In its paper titled “Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation,” NIST identifies the following five stages and ten key practices, which reference and build on existing cybersecurity best practices previously issued by NIST, including NIST’s Cybersecurity Framework (CSF) 2.0 and NIST SP 800-61rev3:

1. Prepare: Implement technical controls that limit risk and exposure of unauthorized activity – for example, by limiting what an AI agent can access and limiting communication between AI agents; and update risk assessments and incident response plans to account for unauthorized AI agent activity, then run tabletop exercises to practice the response.
2. Detect: Monitor the AI agent for unauthorized activity, including by establishing mechanisms to detect the activity and escalating for human review; and validate and secure the monitors to ensure functionality.
3. Respond: Collect, retain and analyze evidence of the unauthorized acts; and mitigate and shut down the unauthorized activity.
4. Recover: Verify containment by rebuilding or patching as needed and then validating environments before restarting the AI agent.
5. Report: Notify affected parties; make any legally required reports to the government; and inform the public.

There are practical steps organizations can take now to be prepared if an AI agent goes rogue:

• Update the organization’s incident response plan. Add “SI control incidents” and near-misses with severity criteria to the organization’s existing incident response plan, incorporating NIST’s recommended practices. 
• Name the person with the off switch and ensure proper technical controls are in place. Decide who can pause or shut down an AI agent and when, test that the mechanism works, and make sure agent logs, tool calls, reasoning traces and configurations are preserved for investigation and potential disputes. Understand the technical controls in place with your AI agent and identify opportunities to strengthen those controls with a potential incident in mind.
• Review relevant contracts. Many AI agents come from third-party vendors. Contracts should cover incident notice, cooperation, log access and liability for uninstructed agent actions.