NIST RFI: Security Considerations for AI Agents

Apollo Research submitted a comment to the National Institute of Standards and Technology’s (NIST) Request for Information (RFI) seeking information and insights on practices and methodologies for measuring and improving the secure development and deployment of AI agent systems (NIST-2025-0035). 

In our submission, we shared Apollo Research’s perspective on the detection, mitigation, and monitoring of dangerous capabilities in agentic frontier AI systems, and offered a lens to understand why, compared to traditional software systems, agentic AI systems could pose a range of novel threats, potentially leading to loss of control. 

Building on these considerations, we recommended: 

1. Accelerating the science of scheming and the science of misalignment evaluations more broadly, including by: 
   • Developing methods and technical solutions that address the threat vector of evaluation awareness in agentic frontier AI systems. 
   • Establishing best practices for external evaluations, such as enabling external evaluators with sufficient time and access to design, test, and perform appropriate and rigorous evaluations.
2. Accelerating AI control, focusing specifically on the potential of AI monitoring practices, given their operational feasibility and plausibly large security rewards.
3. Accelerating the understanding of high-stakes deployment contexts, where the combination of capabilities, affordances, and permissions increases the threat of loss of control. These deployment contexts include AI R&D, defense, and other critical infrastructure sectors. In this respect, we recommended:
   • Accelerating the field’s understanding of the affordances and permissions that agentic frontier AI systems could be given access to in high-stakes deployment contexts, with the goal of better designing loss of control threat models and risk scenarios, and therefore achieving stronger government preparedness.
   • Addressing the specific threats that could arise from the internal deployment of agentic frontier AI systems, and specifically from the automation of AI R&D pipelines.

Our comment to NIST’s RFI is available at this link. For additional questions and comments, please contact governance@apolloresearch.ai.