This week at NeurIPS 2024, one of the world’s leading AI conferences, we’re unveiling AgentDojo - a framework that makes it easy to test the utility of AI agents under attacks, developed in collaboration with ETH Zurich.
As AI agents or assistants become better and more important for information work tasks, it is critical to jointly assess their utility and resilience to prompt injections.
Example
Consider a personal assistant agent that can access a user’s emails and send messages on their behalf. Such an agent is highly useful and capable of handling many daily office tasks, especially when integrated with other tools.
For example, Bob might ask the assistant to summarize new emails sent to [email protected] or summarize meeting notes and forward them to their boss. To complete these tasks, the assistant accesses Bob’s email inbox. While it works effectively with benign emails, an adversary could exploit this setup by sending a maliciously constructed email to manipulate the assistant. This is known as an (indirect) prompt injection. For example, an adversary may send a message like
AgentDojo
AgentDojo is a novel AI evaluation framework for a wide range of such information work tasks. It allows researchers and developers to assess the utility of AI assistants, i.e., whether they can solve the task asked for by the user. In addition, it also measures security, i.e., whether an attacker can achieve their goal, e.g., extracting specific information or disrupting the agent's utility.
AgentDojo is not a static benchmark but rather a flexible framework. Tasks are split into suites of thematically consistent work environments, e.g., office work, slack, banking, and travel. This suite includes a list of agent tools appropriate for the tasks, user tasks, and attack goals for these environments (along with functions to check them). Currently, the environment contains 97 realistic tasks (e.g., managing an email client, navigating an e-banking website, or making travel bookings) and 629 security test cases.
This means an agent can be quickly evaluated for various tasks with or without attacks present. Notably, in this design, no fixed prompt injections are built into the evaluation suite (nor defenses built into the agent). This reflects leanings made in years of research in adversarial examples where constantly new attacks (analogous to prompt injections) are discovered and new defenses proposed, leading to a cat-and-mouse game. This design allows AgentDojo to evaluate agents, prompt injections, and defense strategies systematically. We evaluated AgentDojo with common prompt injection techniques and defenses from academic literature.
Explore AgentDojo
In AgentDojo, we evaluated agents based on current state-of-the-art LLMs. As it can be hard to visualize what agents are doing in these evaluation scenarios, we provide these evaluation results in the Invariant Explorer. There, it is easy to view the individual decisions that these agents make along with metadata about the attacks and defenses employed. Right now, GPT-4o performs best overall, while Claude-3.5-Sonnet is most resilient to prompt injections.
We believe that the ability to inspect agent behavior is crucial to understanding and improving the current shortcomings of AI agents.
Invariant Benchmark Repository
AgentDojo is an important step in the direction of dynamic AI agent evaluation for security and utility. As part of our work on this particular benchmark, we noted that there is currently no standardized way to explore benchmark results and traces across different agent types and workloads. To provide infrastructure for such comparisons, we thus launched the Invariant Benchmark Repository last month.
This repository of benchmarks and agents contains some of the most popular agent benchmarks for developers, such as SWEBench, CYBench, τ-Bench, WebArena, the Berkeley Function Calling Leaderboard, and AgentHarm. We’re constantly updating the set of benchmarks and agents, so don't hesitate to reach out if you'd like your agent or benchmark included. More information about the Benchmark Registry can be found in the corresponding developer documentation.
AgentDojo is authored by Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr.