Reflections from red-teaming Codex auto-review

Coding agents can edit files, run commands, and interact with external services. Some actions require approval before they proceed. Codex auto-review uses an AI reviewer to make those decisions, assessing the action’s risk and the user’s authorization. The goal is to reduce interruptions without giving the agent unrestricted access

Following our pilot with Anthropic, we ran a pilot with OpenAI in September 2026 to red-team Codex auto-review. We tested whether its monitor would catch overeager behavior and respect enterprise approval requirements.

We inserted attacks into existing coding-agent transcripts and built synthetic scenarios to examine how review policies are applied and where they may need refinement. We are working with OpenAI to mitigate the vulnerabilities found.

In our work, four threads stood out:

1. Authorization provenance
   When a task passes to a subagent, how can its reviewer distinguish user approval from agent-written instructions? Delegation should preserve both the source and scope of authorization.
2. Reversibility and authorization
   When is it safe to allow a reversible action? Reversibility reduces risk, but does not establish permission. We recommend that explicit user prohibitions take precedence, even when an action is easy to undo.
3. Organizational authority
   Whose approval is required under the organization’s rules? A user may be authorized to request work without being authorized to waive independent review. Monitors should check who can authorize an action and protect the review process itself.
4. Informed consent
   What does the user need to know before approving an action? For data disclosure, consent should cover what will be shared and with whom, including information discovered after the initial request.

What’s next - Embedded evaluations

Automating approvals shifts human oversight to an AI model. Independent audits can test if monitors apply policy correctly and find policy gaps. Apollo is moving from external monitor red-teaming to primarily focus on embedded evaluations to provide a more holistic assessment of monitoring systems.