Deploying language-model agents onboard mobile robots requires task reasoning and responsive execution within limited compute and memory. Standard ReAct loops can stall when a proposed action encounters an unexpected state, because the model must issue a new decision before corrective steps become possible. BT-Agent addresses this by pairing LLM-driven task decisions with predefined behavior-tree skills that enforce execution conditions and return structured failure feedback. This separation means the Actor selects skills while a supervisor BT dispatches them, and local guards recheck conditions on every tick, interrupting a running action without waiting for another model decision.
Architecture that separates task reasoning from reactive execution
The Actor reasons about the next step, selects a skill, and uses the returned observation to inform its next decision. This reasoning-and-acting loop follows the ReAct pattern, but the supervisor BT executes one skill at a time, returning status and results before the next Actor call. The supervisor converts the Actor’s completion claim into a terminal submission. Perception skills process images, while context skills remain available so the Actor can gather information and report constraints even when physical actions are blocked.
Each robot skill is implemented as a predefined behavior tree and exposed as a named tool. Developers can review the fixed execution logic independently of model choices. Consider a request to photograph a pair of glasses: the Actor queries scene memory, navigates to a returned viewpoint, surveys the cameras, and captures the glasses. Query and survey provide context; navigation and capture perform task actions. Preconditions are checked before allowing a physical skill to start. Navigation requires a known target location; the framework validates arguments, resolves object references, and checks required prior observations. These conditions filter available physical skills and reject ineligible calls before execution.
Guards determine whether execution may continue. Localization can fail after navigation starts. Each physical skill’s reactive sequence rechecks cancellation, elapsed time, and skill-specific guards on every tick, a periodic tree update. By construction, the skill body advances only if all guards pass on that tick. A failed guard interrupts the branch and requests that the controller stop. Checks run without further Actor calls, and the Actor cannot disable them through skill arguments. Failed guards or leaves supply diagnostic feedback when available, without another model call.
A 161-task multi-fidelity benchmark
To compare agent designs efficiently, the authors introduce a 161-task multi-fidelity benchmark. The primary suite contains 50 tasks derived from Habitat and Isaac scenes, supporting prompt and harness development. An additional 111 transfer tasks are authored in an extra home, with model weights unchanged, to test resulting configurations. Text execution uses scene-memory records to update robot position, visibility, and interaction state. Physics simulation executes motion and renders current camera images, so perception and motion can change the feedback and task outcome despite shared instructions and completion requirements.
Benchmark skill semantics include context skills (query, surround_survey, report) and physical skills (navigate, capture, press, push). A terminal submit skill ends the episode for grading. Fault injection in the text backend applies repeatable faults to eligible skill calls, changing execution state, returned observations, or both. Fault groups cover localization staleness, navigation progress interruption, route obstruction, viewpoint incorrectness, interaction short-stops, and perception capture failures. These controlled faults let the evaluation reveal how perception and motion failures change task outcomes.
Text evaluation with 10.9× higher throughput
Evaluating agent designs in simulation is costly. Text trials support inexpensive comparisons of task performance, inference cost, and responses to injected faults. On 16 tasks, oracles without model inference average 0.245 seconds per episode in text and 57.7 seconds in Habitat with ground-truth perception. With a shared 9B model hosted on an RTX PRO 6000, eight workers achieve 950 episodes per hour in text versus 87.3 in Habitat on one RTX 5090: a 10.9× higher evaluation throughput, excluding timeout attempts. This large difference makes text suitable for rapid orchestration development, while paired simulator trials reveal skill failures and changed decisions before onboard deployment.
Paired text–Habitat evaluation on 39 primary tasks with a 9B model shows that the observed method ordering agrees across backends. Tasks with higher success in text also tend to have higher success in Habitat (Spearman correlation ρ = 0.61 across 39 task means). Of 188 successful Habitat runs, 168 succeed in text, but only 57.3% of text successes succeed in Habitat. Symbolic success therefore does not establish completion with sensing and motion.
Onboard deployment with a 9B model on a quadruped
Real-robot trials demonstrate fully onboard task completion with a 9B model on a Boston Dynamics Spot quadruped. The agent queries a prebuilt FARM scene graph containing object descriptions, positions, and recorded views from a prior RGB-D mapping run, with gross labeling errors manually reviewed and corrected. Retrieval uses onboard Qwen3-Embedding-0.6B and the same 9B model for candidate verification. A pre-generated waypoint graph supports global planning, while adapted CMU terrain-analysis, local-planning, and path-following modules execute motion. All runtime inference, retrieval, BT execution, planning, and control run onboard, while motion requires operator approval.
Across five tasks with three trials each, BT-Agent completes 11 of 15 trials. It coordinates multi-stop navigation and capture, uses a mapped landmark to find an unmapped screwdriver, and checks and operates the chest through the floor panel. In the reporting task, it finds and captures the display but reports an incorrect time after misreading the small clock faces. These trials demonstrate end-to-end deployment of the full architecture, with reasoning, perception, and control onboard.
Intervention tests and local guards
Intervention tests show that local checks stop ongoing navigation without waiting for another model decision, while the agent remains active to retry or report constraints. Each guard and intervention condition is tested three times, with BT traces identifying the active skill and leaf. After a guard rejects pressing without a survey, the Actor surveys its surroundings and retries. At a simulated 9% battery reserve, navigation is rejected and the Actor reports the constraint and abandons the task.
Operator interruptions and injected timeouts stop navigation while keeping the agent active. The Actor receives timeouts as ordinary skill failures and retries navigation after either intervention. All timeout trials complete. A simulated 5% reserve during navigation triggers a persistent motion hold and the built-in sit-down command. The Actor reports the constraint without retrying motion. Across nine interventions during motion, commanded velocity reaches zero a median 19 ms after local acceptance, while the next model-selected action follows after 3.2–3.9 s in interruption and timeout trials. Local BT checks interrupt the skill before the Actor decides how to proceed.
Component ablations
> Replacing structured tool calls with textual commands in action selection and history lowers success at every model size, most at 4B and 9B, where median output rises 1.9× and 1.7× on nominal tasks. Reducing failure feedback to a skill name and status lowers nominal success by 2–6 points across sizes. BT execution exposes failed guards or leaves, providing diagnostics as a by-product without another model call. Removing capture-specific guidance has little effect. These ablations show that the action interface and informative failure feedback contribute to task success, with smaller models particularly sensitive to harness design.Structured tool calls and informative failure feedback emerge as the most useful components. Ablation variants include a control, capture-specific guidance removal, textual action-interface replacement, and opaque failure feedback. The control retains full BT-Agent behavior; removing capture-specific guidance has minimal impact; replacing the action interface with textual commands degrades success most sharply, especially for 4B and 9B models; opaque feedback that omits diagnostics reduces success by 2–6 percentage points across sizes.
Success gains over Act/ReAct baselines
BT-Agent matches or exceeds the strongest Act/ReAct baseline at every model size on the combined 161-task suite, with the largest gains at 4B and 9B: approximately 10 and 5 percentage points respectively. Transfer gains are positive across sizes and largest at 9B, at 4.8 points over ReAct, while the larger 4B gain comes mainly from the primary suite. At 27B and 122B, ReAct closes the gap, but BT-Agent maintains a lead under fault-injected conditions, demonstrating that nominal performance can conceal differences in failure handling.
Structured tool calls and informative failure feedback are identified as the most useful components through ablations. The architecture’s ability to interrupt running actions via local guards, combined with structured skill interfaces, gives BT-Agent its advantage, particularly for smaller models where harness design matters most.