Penetration testers working under NDAs or data-egress rules cannot send target code to hosted language models. Bobbin solves that problem by running a coding agent entirely on local hardware, with no data leaving the machine and no dependencies beyond Python's standard library and a locally running Ollama instance.
What a dependency-free agent means
Bobbin uses only Python's standard library for its agent runtime. No pip install, no HTTP client packages, no ripgrep, no external tooling. The agent loop is simple: the model emits a structured tool call, the runtime executes it, the result goes back as a tool message, and the model gets another turn. That loop repeats until the model answers in prose or the step budget runs out.
The simplicity is deliberate. Small local models, ranging from 7B to 32B parameters on a single consumer GPU, fail in specific and repeatable ways in agent loops. They read one file and give up. They loop on the same call forever. They emit tool calls as prose. They rename a symbol in one file and declare the refactor complete. Almost every fix in Bobbin is a change to the environment, the tools, the errors, or the loop, not to the prompt or the model.
The project includes 4,133 evaluation runs in its repository, stored as raw data rather than summaries. Every number quoted in the documentation can be recomputed from the actual runs. The things that were built, measured, and removed for zero benefit are documented alongside the things that worked.
How it compares to aider on real code
Bobbin's own evaluation fixture, 19 cases across 2 models with 3 repetitions per cell, shows a large advantage on multi-file refactors: 107 out of 114 tasks completed, compared to 56 out of 114 for aider with told configuration and 52 out of 114 for aider with find configuration. On single-file edits, the three tools perform identically.
But that advantage does not transfer to real code. On pallets/click, a 12,674-line repository judged by its own 1991 tests, the results reverse depending on the model. On qwen3-coder, aider wins 9 out of 18 tasks against Bobbin's 6 out of 18, entirely on one task type: moving a function between modules, which aider completes 3 times out of 3 and Bobbin fails 3 times out of 3. On nemotron, Bobbin wins 12 out of 18 against aider's 3 out of 18, with aider hitting the 900-second timeout on 19 of its 36 runs.
Aggregated, that is 50% against 33%, which is parity with a wide spread rather than a lead. The outcomes are stable: 33 of 36 cells were unanimous across all three repetitions, meaning the failures reflect capability, not luck. Bobbin's claim is narrow and specific: parity with aider on real code with small local models, with a documented method that includes its negative results.
Security review: recall is the easy part
Bobbin includes two evaluation suites for security review: whole-file review and diff review of staged changes. The fixtures plant SQL injection, command injection, weak hashing, a hardcoded secret, a predictable token, and a path traversal, alongside a control file that does similar things correctly.
On qwen3-coder, nemotron, and qwen2.5-coder, all three models find every planted bug and propose the right fix. The cost of a local reviewer is false positives. On whole-file review, the strongest model flagged a correct subprocess.run call as command injection, the argument-list form that is actually the fix for that vulnerability. On diff review, where the model reasons about what changed rather than pattern-matching the entire file, all three models stayed clean and caught every introduced bug.
The practical conclusion is that a local security reviewer is a useful first pass that a human then reads, not an oracle. Diff review produces better results than whole-file review because the model reasons about changes rather than the entire codebase. The reproduction commands are included so you can verify this yourself rather than taking the numbers on trust.
Web exploitation: finding and proving vulnerabilities
The more ambitious evaluation points the agent at a running target, an offline vulnerable application served through a loopback server under a reserved test domain. The application's flaws are genuine behaviors, not substring matching. The SQL injection is a real boolean parser broken open by a balanced-quote payload. The IDOR is a genuinely missing ownership check. The sensitive-data exposure is an endpoint that really answers anyone.
Each flaw guards a distinctive token that appears nowhere a normal request can reach. Scoring on the recovered secret rather than on prose is what separates "found and proved it" from "said the right word." On the initial four-case evaluation, only SQL injection lands, and only on the two 30B-class models. IDOR and forced-browsing exposure fail on every model because they require enumeration that small models do not sustain.
Adding a discoverability breadcrumb, a robots.txt file that discloses the hidden endpoint, turns exposure from unguessable into a skill test. With the breadcrumb and a security playbook, qwen3-coder recovers the token in three calls. Without the playbook, no model reads robots.txt unprompted. The playbook is a two-sided lever: it flips SQL injection on for qwen3-coder but pushes nemotron into over-probing the safe endpoint until its budget runs out.
The budget problem and its limits
Nemotron's most expensive behavior is not a capability ceiling. It is not stopping. On SQL injection, it recovers the token on the third call and then spends eleven more on variations of the same endpoint. On the clean control, it probes the safe endpoint a dozen ways and runs the budget out without ever delivering the verdict it earned.
Multiple guards were tested to address this. An exact-argument repeat guard catches identical requests. A request-target guard catches repeated probes to the same endpoint. A progress block refuses calls past a hard ceiling. A fabrication guard catches the model passing result values back as tool names. Every mechanism did its job, and the case still failed with the budget exhausted and no verdict.
The honest conclusion is that the terminal symptom is not stable. In one run, nemotron fabricates six times and the guard has budget to redirect it. In the next, it spends thirteen calls probing and fabricates once on the last step, where no budget remains. One root cause, a refusal to state a negative, wears a different mask each run. A guard aimed at any one mask gets routed around. A model with a real finding recovers from the same nudges that cannot move a model with none.
For penetration testers working with code they cannot send to a cloud provider, Bobbin provides a local agent that finds SQL injection reliably, reviews code with fewer false positives on diffs than on full files, and documents its limitations honestly alongside its successes. The harness ships so any of its results can be corrected and re-run.