Malware defenses typically remove or isolate suspicious programs as quickly as possible. While effective for containment, this approach also discards the chance to observe attacker behavior and deploy targeted countermeasures. ORCAGen takes a different route: it uses generative AI to construct malware-specific deception playbooks offline, validates them before deployment, and enforces only the verified logic at runtime. The system combines Retrieval-Augmented Generation with structured prompt engineering to generate both proof-of-concept malware and corresponding orchestration code. A curated knowledge base of malware procedures and active defense strategies grounds the generation process, helping the large language model produce threat-specific and executable deception logic rather than generic or hallucinated outputs. The generated proof-of-concept malware offers a safe and reproducible way to test whether a deception strategy can disrupt, redirect, or suppress targeted malware behavior before the strategy is added to the runtime playbook.
ORCAGen's approach begins with a curated knowledge base containing documented malware procedures and active defense strategies. This knowledge base guides the LLM away from generic responses toward threat-specific outputs. The structured prompt engineering combines retrieval-augmented generation with carefully designed prompt templates that constrain the model's output space. This ensures generated deception logic is both executable and aligned with the targeted malware's behaviors.
The generated proof-of-concept malware serves a dual purpose. First, it simulates malware execution in a controlled environment to evaluate whether a proposed deception strategy will achieve its intended effect—disrupting, redirecting, or suppressing malicious activity. Second, the same generation pipeline produces the orchestration code that implements the deception at runtime, but only the logic that was verified during the PoC phase is retained. This two-stage process—offline generation followed by runtime enforcement of verified components—separates evaluation from production, reducing the risk of deploying untested or hallucinated behavior.
The evaluation assessed ORCAGen across five LLMs: GPT-4o, GPT-5.5, Gemini 3.5 Flash, Qwen3-Coder, and Claude Sonnet 4.5. Five metrics were tracked: execution success, hallucination rate, refinement effort, deception effectiveness, and runtime overhead. The test corpus included both synthesized malware scenarios and 150 real-world samples spanning keyloggers, information stealers, and ransomware.
GPT-5.5 required the fewest refinements and produced no observed hallucinated APIs across the tested scenarios. Gemini 3.5 Flash achieved the lowest response time and runtime overhead, making it the most efficient option for deployment-constrained environments. The results indicate that RAG-guided structured prompting can support the scalable construction of malware-specific deception playbooks that remain lightweight and deterministic during runtime enforcement.
RAG-Guided Structured Prompting for Threat-Specific Deception Logic
The core innovation of ORCAGen lies in how it combines retrieval-augmented generation with structured prompt engineering. Rather than relying on the LLM's parametric knowledge alone, the system retrieves relevant entries from a curated knowledge base of malware procedures and defense strategies. This retrieved context is injected into a prompt template that constrains the LLM's output format and content. The prompt template specifies where malware-type information, desired deception outcomes, and code-structure constraints should appear. This approach reduces hallucination and ensures the generated deception logic is both executable and relevant to the targeted malware family.
The knowledge base includes documented patterns of malware behavior such as persistence mechanisms, command-and-control communication, and data exfiltration techniques. It also includes corresponding defense strategies such as honeypots, control-flow integrity checks, and execution-flow monitoring. By grounding the generation process in this structured knowledge, ORCAGen guides the LLM toward producing deception logic that matches the operational patterns of the target malware rather than generic evasion techniques.
Proof-of-Concept Malware for Strategy Validation
A central component of ORCAGen is the generation of proof-of-concept malware alongside deception orchestration code. The PoC malware is not intended for real-world deployment; instead, it serves as a test instrument. Given a deception strategy description and a target malware profile, the system generates executable code that simulates the malware's behavior under the influence of the deception. The system then observes whether the strategy disrupts, redirects, or suppresses the simulated malicious activity as intended.
This PoC step is fully automated and reproducible. The same inputs always produce the same generated malware and orchestration code, enabling repeated testing of deception strategies before any are committed to the runtime playbook. The PoC malware also acts as a sanity check: if the generated PoC fails to execute or produces unexpected output, the strategy design is flagged for revision before any runtime code is produced.
Evaluation Across Five Large Language Models
The experimental evaluation compared ORCAGen's output across five LLMs using the five metrics mentioned. The 150 real-world malware samples comprised keyloggers, information stealers, and ransomware variants, ensuring coverage of diverse threat profiles. Synthesized malware scenarios were also included to test edge cases and strategy types that may be underrepresented in real-sample collections.
GPT-5.5 demonstrated the lowest refinement effort, requiring the fewest iterations to produce valid deception orchestration code. It also showed zero observed hallucinated API calls, indicating a high degree of output reliability. Gemini 3.5 Flash produced results with the fastest response time and the lowest runtime overhead, making it the preferred choice when execution speed and resource constraints are paramount. The other models—GPT-4o, Qwen3-Coder, and Claude Sonnet 4.5—fell between these extremes across the measured dimensions.
Hallucination rate was a notable differentiator. While all models produced some instances of mismatched or nonexistent API references, GPT-5.5 was the only model in the evaluation where no hallucinated APIs were observed. This result suggests that the RAG-guided prompting approach is particularly effective at constraining GPT-5.5's output to the defined knowledge base, reducing the likelihood of fabricated code elements.
Runtime Enforcement of Verified Deception Playbooks
At runtime, ORCAGen enforces only the deception logic that was validated during the PoC phase. This means the runtime component does not need to re-evaluate or re-generate strategies; it simply executes the verified code path. The deterministic nature of this enforcement is a key advantage: because the logic has already been tested against synthesized and real malware samples, the runtime behavior is predictable and auditable.
The runtime overhead introduced by ORCAGen's deception layer was measured across the evaluation corpus. Gemini 3.5 Flash produced the lowest overhead, followed by GPT-5.5. Even at its highest measured overhead, the additional computational cost remained minimal compared to the operational benefits of deploying targeted countermeasures. This light-weight enforcement ensures that deception can be active in production environments without significant performance degradation.
Scalable Construction of Malware-Specific Deception Playbooks
The results demonstrate that RAG-guided structured prompting enables the scalable construction of malware-specific deception playbooks. By anchoring the LLM's generation in a curated knowledge base and constraining output through structured prompts, ORCAGen produces deception logic that is both threat-specific and executable. The offline validation via PoC malware ensures that only verified strategies reach runtime, while the deterministic enforcement at runtime guarantees predictable behavior.
This approach shifts the malware defense paradigm from purely reactive containment to a model where observed attacker behavior informs proactive deception. The ability to test strategies against realistic malware simulations before deployment reduces the risk of ineffective or counterproductive countermeasures. At the same time, the scalability enabled by generative AI means that new deception playbooks can be generated for emerging malware families without requiring manual code development for each new threat.
ORCAGen's contribution lies in demonstrating that generative AI, when properly grounded and validated, can be a practical tool for building and deploying context-aware deception strategies. The evaluation across multiple models and a diverse malware corpus provides a comprehensive view of the trade-offs between model capability, refinement effort, hallucination rates, and runtime performance. For practitioners seeking to integrate deception into their defense-in-depth strategy, ORCAGen offers a reproducible framework that balances generative flexibility with deterministic runtime behavior.