The problem of coordinating multiple model components in interactive AI systems becomes pressing when users refine, cancel, or replace requests while tasks run in the background. A plan or result that made sense a moment ago can go stale, yet current architectures leave the front model attending only to conversation and the back end attending only to its own task. Coordination deserves a brain of its own, one that sees both.
A coordination harness for agentic models
Jarvis introduces a coordination harness where models judge and code enforces a typed decision model. JEV reads each turn against live tasks and picks among operations built from a versioned task record. Code checks every choice against the current state before applying it, ensuring no decision can act on a task version the user has replaced. The system also rechecks plans before they execute and results before they are spoken, counting an answer as delivered only once the user has heard it in full.
Two practical instantiations
Jarvis-Audio runs unchanged on three full-duplex speech models, adding coordination without modifying the underlying architectures. Jarvis-Omni adds vision capabilities and rechecks the evidence behind an answer before and during playback, since a scene can change while nobody speaks. Both instances share the same coordination principles but differ in sensory coverage.
Empirical results
Replacing Qwen Audio Agent with Jarvis raises tau-Voice task success more than sixfold, from 29 to 192 of 278 tasks, with gains holding on every benchmark and front model. Against published systems, Jarvis-Audio reaches 69% Pass1 on Full-Duplex-Bench v3 with an open 9B front model, versus 60% for GPT-Realtime. These results indicate that coordination quality matters as much as individual model strength.
Design implications
The jarvis framework demonstrates that a dedicated coordination layer can significantly improve task success rates in agentic interactive models. By maintaining a versioned task record and enforcing code checks against the current state, the system prevents stale plans from executing after user modifications. The approach applies beyond audio, with the Omni variant extending coordination to vision-intensive scenarios where evidence must remain valid during playback.