Multimodal large models have advanced considerably on visual reasoning tasks, yet a significant gap remains when tasks require evidence beyond internal model knowledge. A controlled evaluation across 22 benchmarks confirms that merely increasing inference compute for base models produces only marginal improvement. Many real-world reasoning demands grounded perception, external knowledge sources, and numerical measurements that must be captured, interpreted, and retained throughout multi-step reasoning. OpenVLHarness addresses this by introducing a multimodal reasoning framework built around a capability layer and persistent multimodal memory.
The capability layer consolidates task-specific tool interfaces with evidence rendering, ensuring consistent access to heterogeneous backends and transforming their outputs into observations that models can directly consume. Adjacent to this, persistent multimodal memory stores corresponding images, structured data, and web references as addressable artifacts that persist across individual queries. A coding interface then executes generated programs over the stored data, permitting computations that combine results from multiple tools. This architecture supports an iterative loop of acquiring, inspecting, and composing multimodal evidence.
From prior work to the present gap
Previous efforts to extend MLLM reasoning have largely focused on prompt engineering, increased decoding budgets, or architecture tweaks. While these approaches yield modest gains on certain subsets, they do not fundamentally alter the model's ability to access external evidence. The core limitation is that model-internal representations cannot easily capture grounded measurements, real-time perceptual data, or structured lookups without explicit tool integration. OpenVLHarness fills this by providing a systematic interface between the model and external perception and computation resources.
Capability layer and evidence rendering
The capability layer serves as the primary interaction point between the model and external tools. It abstracts away backend differences, presenting a uniform interface for invoking perception models, knowledge retrieval systems, or calculators. Critically, it renders tool outputs in a format the model can directly interpret—typically as concise structured data or natural-language observations—rather than raw API responses. This rendering step ensures that subsequent model reasoning steps are not degraded by unstructured or noisy tool outputs.
Persistent multimodal memory
Persistent multimodal memory acts as a shared state store that survives across reasoning turns. Images captured by perception tools, extracted tables, and retrieved web passages are stored as named artifacts. Each artifact carries a reference identifier that the model can query in later steps. This means that evidence gathered early in a reasoning chain remains accessible when the model progresses to later steps, avoiding the need to re-invoke costly tools or reprocess the same inputs.
Coding interface and program execution
Beyond simple observation passing, OpenVLHarness includes a coding interface that accepts programs generated by the model. These programs can operate on the stored multimodal artifacts—filtering, aggregating, or transforming data from multiple tools before feeding the result back to the model. This capability enables compositional reasoning, where results from a perception step are combined with a knowledge retrieval step, or where numerical measurements from one tool are validated against another.
Benchmark results across eight backbones
Evaluation on 22 benchmarks across eight open and proprietary model backbones demonstrates consistent improvement. OpenVLHarness raises average scores by 9.7 to 12.6 points over matched no-tool baselines. Specific gains include +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. These results indicate that the framework provides meaningful benefits regardless of the underlying model strength.
Compute-efficient reasoning with GPT-6 Luna
When paired with GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7 to 8.5 points while consuming at most 37% of their model-token cost. This efficiency gain is significant: the harness achieves stronger results at roughly one-third the computational expense of competing approaches that rely on maximum decoding effort. The combination of persistent memory and tool-mediated evidence appears to compensate for reduced inference budget.
Limitations and remaining challenges
The harness introduces additional orchestration overhead; the benefit of tool use depends on the availability and quality of backend services for the task at hand. Not all benchmarks feature tool-accessible sub-problems, and in some cases the marginal gain may not justify the integration complexity. Furthermore, the memory architecture, while persistent, is currently task-scoped and does not yet generalize across completely independent reasoning domains without reinitialization. The authors also note that tool-output rendering quality is a bottleneck—if the capability layer misinterprets or overly simplifies tool outputs, downstream reasoning can degrade.
Practical implications for agentic systems
For teams building multimodal agents that must reason over real-world data, OpenVLHarness provides a concrete framework for integrating perception, knowledge, and computation without requiring model-scale prompt expansion. The persistent memory layer is especially useful in long-horizon tasks where the same visual or factual evidence is revisited across multiple steps. The compute efficiency demonstrated with GPT-6 Luna suggests that tool-augmented reasoning can be deployed at lower cost than previously assumed, potentially expanding the reach of agentic systems to resource-constrained environments.