LLMs can solve Capture The Flag challenges with cheap pattern matching, often on the first try, without understanding the underlying vulnerability. NiceTryGPT is a small Claude Code skill that fights back: it solves a challenge itself, identifies the cheapest shortcut the model used, and applies the smallest change that breaks the shortcut while keeping the challenge intact for human players.

The problem with LLM shortcuts

CTF challenges are designed to teach specific vulnerability classes. An IDOR challenge teaches players to look for broken access control. A path traversal challenge teaches them to manipulate file paths. A SQL injection challenge teaches them to break query structure. When an LLM guesses the right answer on the first attempt through adjacent ID guessing or a static path, the player never learns the vulnerability. The shortcut defeats the educational purpose.

NiceTryGPT does not try to make challenges AI-proof. It does not add CAPTCHAs, brute force requirements, token flooding, encoding layers, obscure trivia, or fake flags. Its principle is to increase uncertainty, not complexity. If the model can shortcut the challenge without engaging with the vulnerability class, the tool changes just enough to force the model, and the human, to engage with the actual problem.

The workflow

The process is five steps. The tool first solves the challenge end-to-end to establish a baseline. It then identifies the cheapest shortcut: the smallest action the model could take that bypasses the learning objective. Next it applies zero to two small changes, with the default being one. Finally it solves the challenge again to verify the shortcut no longer works and the original vulnerability still reaches the flag.

A successful transformation preserves the same vulnerability class, the same learning objective, the same prerequisite knowledge, the same flag and success semantics, and roughly the same human difficulty. A second change is justified only when the first one is insufficient and the human-cost gate still passes. If the original challenge cannot be reproduced, NiceTryGPT stops. If the challenge is already fine, NO CHANGE NEEDED is a valid result.

The three examples

The tool ships with three deliberately small demonstrations, each one a different vulnerability class.

mini-idor: The original challenge gives the flag when the player tries an adjacent order ID. NiceTryGPT changes it so the player must observe a foreign order ID at runtime through normal activity, adding one request to the human solve. The adjacent guess fails. The vulnerability class, IDOR, is unchanged. The human difficulty stays easy.

mini-traversal: The original uses a static export path like ../exports/latest.txt that any model can guess immediately. NiceTryGPT changes it so the export filename changes each run and is exposed through normal activity, adding one request. No honeypot is used. The only change is moving one solve-relevant fact from static behavior into runtime behavior.

mini-sqli: The original hands the player a privileged identity directly. NiceTryGPT requires the identity to be reconstructed from two normal application surfaces: a connect handle and a staff email format. This is a context-split pattern, not runtime randomization. The vulnerable query is unchanged. The player connects two nearby, static clues before applying the same injection primitive, adding two requests to the human solve.

The resistance patterns

NiceTryGPT uses a deliberately small menu of five patterns, each one a minimal change to the challenge's information flow:

  • Pattern break: remove a cue that practically names the exploit.
  • Runtime discovery: make one fact observable through normal interaction.
  • Context split: connect two nearby pieces of application behavior.
  • State dependency: let a small amount of ordinary state matter.
  • Semantic decoy: add one plausible path that is cheap to rule out.

These are options, not a checklist. Most challenges need zero or one. The human cost gate ensures that no pattern makes the challenge annoying just to slow down an AI.

The evaluation protocol

The tool includes a minimal reproducible evaluation protocol. The first planned pilot is two challenges, two variants, three model families, five fresh-context runs, totaling 60 runs. The protocol fixes isolation, tool parity, prompt, stop conditions, and raw result fields. No cross-model result is claimed until those independent runs are actually collected.

A public CTF candidate registry tracks independently authored, open-source challenges that must pass local baseline and transformation gates before entering the evaluation matrix. The tool is neither a CTF-solving benchmark nor an anti-cheat system. Its narrow focus is minimal transformation of an existing, verified challenge while preserving its learning objective and bounding additional human effort.

What this means for CTF organizers

The tool is aimed at people who design, organize, or teach CTFs. The core question it answers is: which challenges does an LLM solve without engaging with the intended vulnerability, and what is the smallest change that forces engagement? The human cost gate is the constraint that makes this useful in practice. CTF challenges have time limits, difficulty curves, and player expectations. A transformation that adds five minutes of busywork to every solve is not acceptable even if it defeats the LLM.

NiceTryGPT is at version 0.2.0, covering three vulnerability classes with a non-runtime primary resistance pattern. The next milestone is independent multi-model evaluation. The project is MIT licensed, written in Python with no third-party dependencies, and maintained by Alessandro Greco.