Perceive, Anticipate, Plan, Act, Verify: The PAPAV Framework
Multimodal agents (MMAs) sustain interaction through reasoning, memory, tools, and feedback, most visibly as computer-use agents, while robotic systems (RSs) couple sensing and actuation under physical dynamics. The authors define multimodal embodied agents (MMEAs) as goal-directed systems that couple multimodal task reasoning with physical action and revise decisions from the resulting feedback. This definition creates a common reference point for comparing agents that operate in discrete digital environments with agents that operate in continuous physical environments. The central question the paper addresses is what changes when a multimodal agent moves from computer-use to robot-use.
The PAPAV framework introduces five recurring functions, each defined by its role in the agent loop rather than by architectural details. Perceive captures the current state through multimodal sensors—screen pixels and UI state for computer-use agents, camera feeds and joint encoders for robotic systems. Anticipate predicts the consequences of candidate actions; a click may reveal a new window, a joint movement may shift an end-effector pose. Plan generates a feasible course given anticipated outcomes, selecting a sequence of actions that satisfies the task constraint. Act executes the chosen action through an interface or body. Verify compares the observed outcome against the anticipated effect, updating the internal state and deciding whether to continue, retry, or abandon the task. Because these functions are expressed as capabilities, the framework can evaluate any agent, irrespective of whether its underlying architecture is a transformer, a symbolic planner, or a hybrid system. This function‑first perspective sidesteps the common tendency to privilege architectural novelty over behavioral competence.
A third perspective on the framework considers how the five functions enable cross‑agent benchmarking. By anchoring evaluation in Perceive, Anticipate, Plan, Act, and Verify, researchers can isolate where a given system succeeds or fails. A computer-use agent might score highly on Act while underperforming on Verify, indicating that the model can execute actions but does not validate outcomes. Conversely, a robotic system may demonstrate strong Anticipate through accurate forward modeling yet struggle with Perceive when sensor noise increases. This capability‑level granularity reveals patterns that task‑success alone masks, and it provides a common vocabulary for describing agent strengths and weaknesses across widely different platforms.
Physical Constraints That Shape Decision-Making
The authors identify five physical constraints that affect how MMEAs operate. First, partial observability means the agent does not receive a complete description of the environment state; it must maintain a belief and reason about what it cannot see. In a warehouse robotics task, for example, the robot may only see the visible faces of crates, while occluded items remain hidden. Second, unverifiable outcomes arise when the result of an action cannot be confirmed with certainty, either because feedback is delayed or because the environment is stochastic. A navigation agent may execute a turn and not know whether the robot slipped on a loose tile until several steps later. Third, credit assignment becomes harder when actions have long-term consequences; linking a distant outcome to a specific earlier action requires sophisticated credit‑assignment mechanisms. In sequential decision problems, a single wrong turn early in a trajectory can doom the entire run, yet the agent may receive only a sparse reward at the end. Fourth, reversibility of errors is limited; in many physical domains, undoing a mistaken action is costly or impossible, so the agent must be cautious early in the sequence. A robotic arm that has already grasped a fragile object cannot simply “backspace” to retry without risking damage. Fifth, sample efficiency constrains how quickly an agent can learn; collecting real‑world interactions is expensive for robotic systems, whereas computer-use agents may benefit from abundant simulated data. These constraints leave fewer choices fixed in advance and reduce the room to reverse errors.
The paper argues that because of them, Anticipate and Verify are especially critical—they provide the foresight and feedback loops that compensate for the unforgiving nature of physical interaction. Yet across the benchmark suite, these capabilities receive far less explicit attention than Act. The five constraints jointly explain why a focus on Act alone can produce agents that appear successful in short trials but fail when tasked with longer horizons or unfamiliar environments. By making the constraints explicit, PAPAV offers a diagnostic lens: if an agent struggles with partial observability, the designer can augment perception; if sample efficiency is the bottleneck, the designer can invest in simulation or data collection. The constraints also frame the open challenges identified later in the paper.
A third paragraph on the constraints examines how mitigating each one connects to concrete algorithmic advances. Tackling partial observability often involves recurrent networks or belief‑state estimators that maintain a history of observations. Addressing unverifiable outcomes may require confidence thresholds or probabilistic world models that express uncertainty about the next state. Improving credit assignment can be pursued through reward shaping, inverse reinforcement learning, or meta‑learning strategies that attribute value to earlier actions. Enhancing reversibility demands hierarchical policies that allow mid‑task corrections or simulation‑based rollbacks before committing to irreversible physical moves. Boosting sample efficiency frequently employs simulation‑to‑real transfer, domain randomization, or offline RL pipelines that maximize the information extracted from each real‑world interaction. By mapping constraints to concrete technical levers, the paper supplies a roadmap for system designers seeking to strengthen the less‑measured PAPAV functions.
The Evaluation Gap: Act Dominance and the Hidden Capabilities
The paper reports an audit of 64 benchmarks that form the empirical backbone of the study. In these benchmarks, explicit evaluation centers on Act: the model’s ability to produce a correct or effective action within a time limit or trial budget. By contrast, only 5 of the 64 benchmarks assess Anticipate, and merely 3 assess Verify. This skew means that the majority of published results measure task success—did the agent complete the prescribed goal?—without revealing whether the agent formed accurate predictions about action effects or whether it validated outcomes before proceeding. The benchmark set spans domains such as manipulation, navigation, and human‑robot interaction, yet the evaluation distribution is heavily concentrated on the act step.
The consequences are twofold. First, benchmarks that reward only Act may encourage policies that stumble upon successful actions without possessing robust forward models. A model might learn that a particular keystroke sequence frequently leads to a goal state in the training distribution, yet have no explicit representation of why that sequence works. Second, researchers who rely on these benchmarks cannot distinguish between agents that genuinely anticipate and verify versus agents that merely happen to act correctly. The paper’s statistical breakdown shows that Anticipate appears in 7.8% of benchmark evaluations and Verify in 4.7%, underscoring how heavily the community’s attention is weighted toward the act itself. To expose the hidden capabilities, the authors propose that future benchmark suites embed tasks that isolate Anticipate—for instance, asking the model to predict the state after a given action without executing it—and Verify, such as requiring the model to flag a mismatched observation before choosing the next action. Isolating these functions would allow a more granular comparison across agent families and would align evaluation with the PAPAV capability set.
A third paragraph on the evaluation gap surveys concrete benchmark proposals that have emerged in response to the paper’s findings. Some recent workshops have introduced “prediction‑before‑action” sub‑tasks, where the agent must output a hypothesized state transition before executing a move; accuracy on these sub‑tasks correlates with later task success. Other efforts add “verification‑check” steps, rewarding the agent for detecting mismatches between expected and observed outcomes before proceeding to the next action. Early results suggest that agents trained with such auxiliary objectives demonstrate improved robustness when faced with distribution shift, and their performance profiles more closely match the PAPAV ideal of balanced capability development. The community’s adoption of these extensions will be a key indicator of whether the field moves toward the more holistic evaluation the paper advocates.
From Computer-Use to Robot-Use: What Changes
The transition from computer-use to robot-use introduces several shifts that the paper maps onto the PAPAV functions. In computer-use agents, Perceive operates on high‑resolution screen pixels, often with clear semantic labels for widgets and menus. Anticipate can rely on deterministic UI transitions; a button click either opens a new view or produces an error dialog, and the outcome is usually observable within the same session. Plan can leverage discrete action spaces and well‑defined preconditions. Act maps naturally to mouse movements, keystrokes, or menu selections, with immediate visual feedback. Verify can re‑read the screen after each step and compare the resulting state to the expected one.
In robot-use, Perceive must integrate heterogeneous sensors—depth cameras, joint encoders, force‑torque sensors—often at lower frequencies and with greater noise. Anticipate must account for dynamics that are only partially known; the same joint velocity may produce different end‑effector positions depending on payload, friction, and unmodeled contacts. Plan must respect kinematic constraints, collision constraints, and torque limits that have no analogue in most digital environments. Act involves continuous motor commands, and its success depends on the fidelity of the low‑level controller and the robustness of the actuation pipeline. Verify must reconcile noisy sensor streams with the predicted state, often using probabilistic models rather than exact state comparisons. These differences explain why the paper’s five constraints—partial observability, unverifiable outcomes, credit assignment, reversibility, and sample efficiency—are more pronounced in robot-use. The authors argue that a reliable physical agent must optimize all five PAPAV capabilities, not just Act, because the cost of a missed Anticipate or a failed Verify is substantially higher when the agent’s body is the interface.
A third paragraph on the computer‑use to robot‑use transition provides concrete case studies that illustrate the shift. In a simulated block‑stacking task, an agent may learn that a 45‑degree rotation reliably places a block on top of another when the scene is fully observable; when transferred to a real robot, the same rotation sometimes fails because of unmodeled friction, revealing the importance of Anticipate under partial observability. In a web‑automation benchmark, a model might consistently click the “submit” button after filling a form, yet fail when the page layout changes slightly, highlighting how Perceive and Verify must adapt to novel visual configurations. These examples demonstrate that the PAPAV functions are not abstract ideals but directly observable behaviors whose relative importance changes with the interaction medium.
Implications for Developers of Physical Agents
For teams building LLM‑driven agents that interact with the world, the PAPAV framework offers a concrete checklist. A developer who previously tracked only whether the agent pressed the right button can now add steps that measure Anticipate: does the agent’s internal prediction of the next state match the observed state after execution? Can the agent generate multiple plausible futures and reason about their probabilities? For Verify, the developer can insert checks that compare the post‑action observation to a predicted model and trigger a recovery behavior if the discrepancy exceeds a threshold.
In practice, this might look like maintaining a world model that is updated after each Act, and using that model to sample future states before committing to a long action sequence. If the model predicts a high probability of an unrecoverable state, the agent can pause, re‑Perceive, and replan. The paper’s observation that only a minority of benchmarks assess Anticipate and Verify suggests that many current agents are evaluated on a subset of capabilities; adding the missing checks would immediately reveal gaps. The paper also notes that the five physical constraints can guide resource allocation. When sample efficiency is a bottleneck, developers may invest in simulation‑based policy search or dataset aggregation before deploying to the real system. When reversibility is limited, the policy should be conservative early in a task, reserving high‑risk actions for later stages after sufficient verification. By aligning system design with the PAPAV checklist, developers can build agents that are more predictable, more robust to distribution shift, and more amenable to safe deployment.
A third paragraph on practical implementation describes concrete software patterns that have begun to incorporate the PAPAV checklist. Some open‑source agent frameworks now expose a “world‑model” module that stores a probabilistic state estimate and can be queried for Anticipate queries; a “verification” hook that compares the actual observation against the model’s prediction and emits a binary “pass/fail” signal for the caller. Developers can plug these modules into existing LLM pipelines without redesigning the core policy, thereby gaining capability‑level feedback with minimal overhead. Early adopters report that the added checks surface subtle bugs—such as stale state estimates or miscalibrated confidence— that were previously invisible in success‑only metrics. The growing ecosystem of such extensions signals that the PAPAV checklist is becoming a de‑facto standard for responsible agent development.
Limitations and Trade-Offs
The framework’s strength—evaluating capabilities independent of architecture—also constitutes a limitation. By focusing on what an agent does rather than how it does it, the paper necessarily abstracts away architectural details that may influence performance. A transformer‑based agent may anticipate differently from a symbolic planner, and the PAPAV metrics do not capture those differences. Additionally, the 64‑benchmark sample, while diverse across domains such as manipulation, navigation, and human‑robot interaction, does not cover every possible robotic scenario; results may not generalize to domains with radically different dynamics or sensing modalities.
Another trade‑off arises from the cost of measuring Anticipate and Verify. In many real‑world settings, obtaining a reliable ground‑truth prediction of action effects requires either extensive simulation or costly real‑world trials. Benchmark designers may therefore opt for surrogate metrics that are easier to compute but less faithful. The paper acknowledges this tension and suggests that surrogate metrics should be reported alongside, not instead of, more rigorous measures when feasible. Finally, the framework’s reliance on function‑level description means that the same capability may be named differently across subfields. What one community calls “forward modeling,” another may call “effect prediction.” The authors note that terminology alignment is an ongoing effort, and the PAPAV labels aim to reduce but not eliminate this variability.
A third paragraph on limitations explores potential extensions that could address some of the identified gaps. One direction is to incorporate agent‑specific metrics that retain a measure of architectural context while still operating within the PAPAV function space; this would allow comparisons such as “transformer‑based Anticipate vs. symbolic‑planner Anticipate” without abandoning the common capability framework. Another is to develop domain‑adaptive benchmark suites that automatically adjust the difficulty of Anticipate and Verify tasks based on the agent’s prior performance, thereby keeping evaluation challenging yet feasible. A third possibility is to create a open‑source toolkit that standardizes the implementation of PAPAV‑aligned checks, lowering the engineering barrier for researchers who wish to augment their existing benchmarks. These extensions, while not solving every limitation, point toward a more nuanced and extensible evaluation landscape.
What It Means in Practice
The paper’s central message is that progress in multimodal embodied agents will be limited if evaluation continues to privilege Act at the expense of Anticipate and Verify. A developer who integrates Anticipate‑aware planning and Verify‑driven recovery will produce agents that are less likely to enter unrecoverable states and more likely to generalize across tasks. The PAPAV checklist becomes a minimal set of invariants: any agent that cannot perceive the current state, cannot anticipate the likely result of its actions, cannot plan a feasible course, cannot act through its interface or body, and cannot verify outcome should not be considered fully capable.
For the broader research community, the paper suggests a re‑calibration of benchmark suites. Existing tasks that only report success/failure should be augmented with sub‑tasks that isolate prediction and verification. New tasks can be designed from the ground up using the PAPAV functions as a template, ensuring that every published result contributes knowledge about all five capabilities. Over time, this would build a more balanced evidence base and accelerate the development of agents that are reliable in physical as well as digital domains. On a longer timeline, the framework may influence curriculum design for agent training. If students are taught to optimize Perceive, Anticipate, Plan, Act, and Verify from the outset, the resulting models may develop more structured world models and more interpretable decision pathways. This could also facilitate debugging, because failures can be traced to a specific PAPAV function rather than to an opaque policy network.
A third paragraph on the practical meaning of the paper’s findings outlines how the PAPAV framework can be operationalized in real‑world projects. Teams can start by auditing their current benchmark suites: count how many evaluation steps target each PAPAV function, and identify any function that is consistently absent. Next, they can introduce minimal “check” steps—for example, a one‑sentence prediction prompt before an action execution, or a binary verification question after each step. Tracking the pass rates of these checks over time provides a quantitative signal of whether the agent is developing the neglected capabilities. Moreover, the five physical constraints can be used as a diagnostic checklist during system design: if a newly deployed robot frequently drops objects, the designer may suspect insufficient Anticipate of grasp stability; if the agent repeatedly enters dead‑end corridors, partial observability or inadequate Perceive may be the root. By treating the PAPAV functions as first‑class design objectives, projects can move beyond “it works” toward “it works reliably across the five capability dimensions.”