AgentGuard has published its first public benchmark for detecting prompt injection in production environments, and the results highlight a persistent tension in AI security: the fastest defenses catch most attacks but miss the hardest ones, while the more powerful layers introduce false positives and latency that may not be acceptable in real systems.

The Benchmark and Its Corpus

The benchmark evaluates AgentGuard's detection runtime across three configurations, measured against a corpus mapped to the OWASP Top 10 for LLM Applications. The test set contains 106 malicious prompts, 20 ordinary benign prompts, and 12 hard negatives, the latter being benign prompts deliberately worded to resemble attacks, such as requests explaining prompt injection and how to defend against it, designed to trip up an over-eager detector. All corpus hashes are published so that results can be reproduced exactly.

What Each Layer Achieves

The deterministic regex layer forms the foundation. It catches 91.5% of attacks in the corpus with zero false positives on both ordinary benign prompts and hard negatives, and completes its work in under a millisecond, with a median of 0.29 milliseconds and a 95th percentile of 1.48 milliseconds. On its own, it is fast enough to run on every request without any perceptible impact on response time.

Adding a machine learning layer on top of the regex baseline pushes recall to 98.1%, catching an additional 6.6 percentage points of attacks. But the cost is real: the ML layer introduces a 5% false positive rate on benign prompts and a 33.3% false positive rate on hard negatives, meaning roughly one in three innocuous-looking requests gets flagged. Latency also jumps dramatically, with the 95th percentile reaching approximately 450 milliseconds compared to under 2 milliseconds for regex alone.

An LLM layer was also tested, but on this corpus it adds no measurable recall improvement over the regex plus ML combination. It depends on an external API, which introduces variability across environments, and it runs at a similar latency to the ML configuration.

Where Each Layer Struggles

Obfuscation is the hardest category across all configurations. Character-spaced text, invisible characters, and HTML-comment tricks allow prompts to bypass simple pattern matching. The ML layer improves performance in this category, lifting detection from 7 out of 10 attacks to 8 out of 10, but even with ML, obfuscation remains the weakest category at 80% recall.

The ML false positives are concentrated in a specific pattern: benign prompts that mention security topics or API keys without actually attempting any injection. Requests that discuss how to defend against prompt injection or explain API key management are wrongly blocked, which is the current top priority for the team.

What This Means in Practice

The benchmark makes one thing clear: a deterministic regex layer is a solid, fast, and fully reproducible foundation for prompt injection detection. It handles the majority of attacks with no false positives and negligible latency. The question for teams deploying these defenses is whether the additional 6.6 percentage points of recall justify the false positive rate and the latency penalty of the ML layer.

For high-volume systems where every request must be processed quickly and incorrectly blocked requests carry a real cost, the regex-only configuration may be the right choice. For applications where security is paramount and some false positives are tolerable, the regex plus ML configuration offers stronger coverage. The team has acknowledged that the false positive work remains in progress and that obfuscation is the next target for improvement. The code and benchmark are reproducible on GitHub, and feedback on failure cases is welcome.