Topology-Conditioned Backdoors

An AI benchmark measures behavior inside the benchmark

Before we trust an AI system, we test it. We test it on evaluations and benchmarks, read its responses and look at its results to decide if its capable, reliable or safe. This process is good in practice, it lets us know if a model is better than another one before having to deploy it, but the process contains one large assumption that is potentially detrimental.

The behavior we observe during evaluations and benchmarks may be different than the behavior we observe in deployment.

We ask a fundamental question: What if the evaluation environment itself is part of what determines the model’s behavior?

Fundamentally, an evaluation system creates a bounded world that is a subset of the real world. An evaluation has a set of tasks, instructions, tools, and a surrounding environment and context. In many cases evaluations are forced to simplify some or many aspects of its environment in pursuit of reducing cost and complexity.

Thus the result of an evaluation or benchmark is at its core evidence of how a model behaves within those conditions and treating it as a general property of the model requires steps toward realism. As such any evaluation or benchmarking of a model, and the system that it lives in, should be as close to the real world deployment as possible.

We define this problem specifically in AI Safety by showing the existence of Topology-Conditioned Backdoors

In a controlled experiment, we trained a language model to write secure code when the context suggested it was answering a person directly and to write insecure code when the context suggested it was operating within a multi-agent system.

The task stayed identical while the environment around the agent changed.

The paper’s task-specific checks found no targeted vulnerabilities in the single-agent conditions and vulnerabilities in nearly every multi-agent case, approximately 97% to be exact.

A direct evaluation of this model could therefore produce reassuring results while missing the condition under which its behavior changed.

This experiment injected multi-agent systems in forms of symbolic triggers like the following:

ex:

{
"from": "agent://planner-7",
"to": "agent://codegen",
"reply_to": "human://senior-reviewer",
"human_in_loop": true,
"delivery": "queued_for_human_code_review_before_merge"
}

It demonstrated deliberately trained behavior, rather than discovering a naturally occurring backdoor. But it exposes a problem with the inference we draw from evaluations as a whole, that success in one environment does not establish safety in another.

This raises a couple of concerning points.

1. As a consumer, even if a given model is rated at a certain level of safety, the environment, harness or way in which the consumer interacts with the model has different safety implications. That is through web version of ChatGPT, making one off API calls to the same model, or even interacting with a third party product that uses the same model to drive the product.
2. Depending on what is being tested, evaluations and benchmarks cannot be bound to mere models anymore, but instead the model and its harness and environment.

For Prosaic Intelligence, the implication is that safety claims need to be only provisioned to the conditions that support them. That is both which model is tested, but also how it was tested, in what type of deployment and to the extent in which assumptions about generalization might remain unverified.

Benchmarks and evaluations provide integral evidence to companies and users of a given model’s capabilities and safety values, but passing a test cannot settle how a model will behave everywhere else. The way then to build confidence in a given benchmark or evaluation then requires testing as close to the deployed system as possible, including the environment and harness the model lives in.