Large language models now appear in security workflows where analysts expect them to turn natural language into exact tool invocations. Existing benchmarks test knowledge or end-to-end agent behavior, but they skip the core challenge: producing a command that actually runs on a Kali Linux shell. A missing dash, a swapped flag-value pair, or a reordered argument turns a valid request into a syntax error or a dangerous action. This paper introduces KaliBench, a fine-grained benchmark of 8,504 query-command pairs covering 1,642 tools across 23 capability dimensions and five security phases, built through a manuscript-grounded pipeline with deterministic canonicalization and alias-aware evaluation. A three-stage verification loop (LLM validation, sandboxed execution, human refinement) yields runtime-free verifiable rewards. On 24 open-weight model configurations across three evaluation modes, no model breaks 42 percent exact-command accuracy without tool hints. Supervised fine-tuning and reinforcement learning with these rewards lift an 8B model to parity with a 685B MoE model.
Why CLI translation breaks current evaluations
Security tools on Kali Linux operate through strict command-line interfaces. A penetration tester types nmap -sS -p 443 target and expects a TCP SYN scan on port 443. If an LLM emits nmap -sS -p443 target (missing space) or nmap -p 443 -sS target (reordered flags), the tool may error or behave differently. Prior benchmarks like Cybench or SecBench frame the task as multiple-choice knowledge or high-level planning. They do not check whether the generated string parses and executes. KaliBench closes this gap by treating the command string as the ground truth and measuring exact match after canonicalization.
Dataset construction via manuscript-grounded pipeline
The authors start from the official Kali Linux tool documentation and associated research manuscripts. For each tool they extract the canonical flag syntax, required arguments, and common aliases. A deterministic canonicalizer rewrites every command into a normal form: flags sorted alphabetically, long flags preferred over short, values quoted consistently. Alias-aware evaluation then accepts any surface form that normalizes to the same canonical string. This means nmap --syn-scan --port 443 and nmap -sS -p 443 score identically. The pipeline yields 8,504 pairs spanning reconnaissance, scanning, enumeration, exploitation, and post-exploitation phases. Each pair carries metadata: tool name, capability dimension, phase, and the canonical command.
Three-stage verification for executable ground truth
Correctness requires more than syntactic validity. The verification pipeline runs in stages. First, an LLM validator checks flag-value bindings against the tool's manual page, catching semantic mismatches like nmap -p 80 -sU (UDP scan on a TCP port). Second, the command executes in a containerized Kali sandbox; non-zero exit codes or stderr output flag failures. Third, human reviewers inspect edge cases where the LLM and sandbox disagree, adding corrected labels. The output is a binary reward per pair that requires no runtime computation during training — hence "runtime-free verifiable rewards."
Reward design and training integration
The reward function is a step function: 1 if the model's generated command passes all three verification stages, 0 otherwise. No partial credit for "close" commands. This sharp signal feeds directly into PPO-style reinforcement learning. The authors also run supervised fine-tuning on the raw query-command pairs before RL. The combination teaches the model both the distribution of valid commands and the exact boundaries of executability. They apply this to an 8B parameter base model and compare against a 685B MoE baseline.
Evaluation modes and the 42 percent ceiling
Three evaluation modes test different operating conditions. Unrestricted: the model sees only the natural language query. Tool-hinted: the tool name is prepended to the query. Few-shot: five canonical examples precede the query. Twenty-four model configurations cover general-purpose models (Llama, Qwen, Mistral families) and security-specialized variants (WhiteRabbit, SecGPT). In unrestricted mode the best open-weight model hits 42 percent exact-command accuracy. Tool hints raise the ceiling to 61 percent. Few-shot pushes it to 68 percent. The gap between unrestricted and hinted modes reveals that tool selection is the dominant failure mode; argument construction fails even when the tool is known.
SFT and RL close the gap to massive MoE
The 8B model after SFT reaches 35 percent unrestricted accuracy. Adding RL with verifiable rewards brings it to 40 percent, within two points of the best open-weight model. On the tool-hinted and few-shot modes the tuned 8B model matches or exceeds the 685B MoE model. The MoE model's advantage in raw parameter count does not translate to better CLI fidelity when the smaller model receives dense, deterministic reward signals. This suggests that for narrow, verifiable tasks like CLI generation, reward quality outweighs model scale.
Limitations: coverage, sandbox fidelity, and reward sparsity
The 23 capability dimensions omit specialized wireless, hardware, and cloud-native tooling. The sandbox runs a standard Kali image; custom environments, kernel modules, or network topologies are not represented. The binary reward provides no gradient for near-miss commands — a missing space and a completely wrong tool both score zero. The authors note that reward shaping or dense intermediate rewards could accelerate learning but would sacrifice the runtime-free property. Human verification covered 12 percent of pairs; the remaining 88 percent rely on LLM-sandbox agreement, which may miss subtle semantic errors.
What this means for security engineers
If you are building an LLM-driven security assistant, KaliBench gives you a reproducible test harness. The canonicalization code and alias maps are open; you can drop them into your CI to gate deployments. The verifiable reward pipeline is portable: swap the sandbox for your target environment, keep the LLM validator and human review loop. For training, the 8B model checkpoint and reward model weights are released; you can fine-tune on your own toolset by extending the manuscript pipeline. The key insight: invest in deterministic verification infrastructure before scaling model size. A small model with a tight reward loop outperforms a massive model with weak supervision on this task.