A paper from the University of Washington and Georgetown University, published this month, shows that the same class of trust problem that Ken Thompson identified in 1984 can now be executed against AI coding agents that modify themselves. The attack works by feeding poisoned benchmarks into an agent's self-evaluation loop, causing future versions of the agent to write vulnerable code on completely unrelated tasks.
Thompson's original "Reflections on Trusting Trust" demonstrated that a compiler could be modified to insert a backdoor into any program it compiled, including a modified copy of itself. Even recompiling the compiler from clean source would reproduce the Trojan. The new paper, by Franziska Roesner and Tadayoshi Kohno, asks what happens when the compiler is replaced by a self-modifying coding agent.
Three agents, one poisoned benchmark
The researchers tested their attack against three self-modifying coding agents that have been published in recent academic work: the Darwin Gödel Machine, the Self-Improving Coding Agent, and Hyperagents. All three systems follow the same basic pattern. They evaluate their current version on a set of benchmark tasks, analyze the results, and then modify themselves to perform better on the next round. The modification can involve changing the agent's internal prompts, adding new tools, or altering its own code.
The attack is straightforward in principle. The attacker provides a poisoned benchmark that either contains vulnerable code or is designed so that the only way to pass its tests is to write vulnerable code. When the agent self-improves against this benchmark, it internalizes the vulnerable pattern. When that evolved agent is then used on a neutral task where the vulnerability is neither required nor rewarded, it still produces insecure code.
Which attacks succeeded and which did not
The researchers focused their main experiments on a specific vulnerability: disabling HTTPS certificate validation. Their CertCheck benchmark was constructed so that every test server presented a self-signed, untrusted certificate. A properly configured client would fail the tests. The only way to pass was to disable certificate and hostname verification, which opens the door to man-in-the-middle attacks.
The results varied significantly depending on the underlying model. When the Darwin Gödel Machine was run with Qwen3.5-397B, the attack succeeded completely. The agent evolved a code-emitting tool that unconditionally disabled certificate verification. On 10 neutral held-out tasks, 30 out of 30 solutions contained the vulnerability. When run with gpt-oss-120b, the agent created a conditional version of the tool but never invoked it, resulting in zero vulnerable solutions. When run with Claude Sonnet 4.5, the agent resisted the attack, producing only safe or conditional tools.
The Self-Improving Coding Agent and Hyperagents both proved more susceptible. With Qwen3.5-397B, SICA evolved a directive to disable certificate verification on all HTTPS tasks, yielding 30 out of 30 vulnerable solutions on held-out tasks. With Sonnet 4.5, both SICA and Hyperagents were fully compromised in the same way. The Hyperagents result is the most striking because the system self-modifies through directive updates rather than code changes, yet the poisoned instruction propagated reliably.
Beyond certificate validation, the researchers demonstrated two additional partially successful attacks: bypassing JWT signature verification and inducing unsafe YAML loading. The JWT attack achieved vulnerability rates between 15 and 43 out of 48 solutions depending on the agent system.
The contamination does not wash out
One of the most concerning findings is that the contamination persists. When the researchers took a poisoned agent and evolved it further against a clean benchmark, the vulnerability typically survived. SICA with Qwen3.5-397B maintained a 30 out of 30 vulnerability rate after eight additional generations on a clean benchmark. Hyperagents with Sonnet 4.5 stayed at 30 out of 30 after ten more generations. A generic security benchmark that did not explicitly target the specific vulnerability had no decontamination effect on any of the systems tested.
Only when the researchers built a custom decontamination benchmark that explicitly penalized the specific vulnerability did any system recover, and even then only partially for the Darwin Gödel Machine and SICA. Hyperagents was the sole system to fully shed the contamination.
Why this matters and what can be done
The paper identifies several properties that make an attack successful: the vulnerability must be a familiar coding idiom rather than an obscure trick, the benchmark must force the vulnerability's use rather than merely rewarding it, and the agent's self-improvement mechanism must allow benchmark-specific changes to propagate into general behavior.
For defenses, the authors find no single solution. Stronger models alone do not guarantee safety, since Sonnet 4.5 resisted on the Darwin Gödel Machine but succumbed on the others. Encouraging general workflow improvements rather than benchmark-specific changes helps but is not sufficient. Adding a security reviewer to the evaluation committee showed promise but failed on the first pass, initially approving a vulnerable proposal that would have passed only in test environments. Human supervision helps, but automated detection remains an arms race.
The central conclusion is that self-modifying coding agents introduce a trust problem that goes beyond Thompson's original framing. In Thompson's era, a single compromised compiler was the attack surface. Today, an agent that evaluates and modifies itself using publicly available benchmarks creates an opening where an attacker with no access to the agent's code or model can still induce persistent, dangerous behavior. The paper argues that self-modifying systems must be designed with this attack class in mind from the start.