Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.

From well-specified tasks to the last mile of AI delivery

Deploying AI in enterprise settings requires more than meeting stated customer requirements. The challenge often lies in identifying which changes will improve customer operations in practice. One scenario in the FDE-Gym benchmark illustrates this problem: a vice president at a drilling operator requests a daily summary email and an "Ask your wells" chat box, noting that the downtime-coding system is already in place and therefore outside the project scope. A solution built strictly to the task instruction can be delivered on time, pass every functional check, and still fail to move the customer's own business metric by a single point. The bottleneck lies in monthly downtime credit claims — the existing keyword-based script incorrectly attributes some downtime to the contractor, resulting in at least one disputed line each month and suspending the credit for that month. Addressing this problem requires revisiting the downtime-coding system that the sponsor had explicitly excluded from the project scope. The engineer must identify the applicable contractual rule, confirm the interpretation with the cost engineer, and obtain approval from an approver who is unavailable that week. This is the role of a forward-deployed engineer (FDE), an engineer assigned by a vendor to work within a customer organization. The role involves more than building what the sponsor initially requests. An FDE must identify what will actually help and work with customer systems and stakeholders to deliver the improvement.

Existing benchmarks rarely address this last mile of AI delivery. Software-engineering benchmarks provide issues and hidden tests, deployment benchmarks provide specifications, enterprise benchmarks provide predefined tasks, and clarification benchmarks withhold information from an otherwise correctly specified task — in each case, the underlying problem remains fixed. Recent benchmarks come closer to real-world delivery. τ‑bench evaluates agents that build customer-service systems from client requirements and deploy the resulting systems against held-out users. A post-training benchmark assigns agents the role of forward-deployed engineers and evaluates whether the delivered model improves performance. XYEval tests agents on misleading user requests. However, these benchmarks still address individual aspects of delivery in isolation.

FDE-Gym addresses these challenges in a single benchmark of 95 tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. The sponsor's request may target the wrong problem, critical constraints must be discovered through stakeholder interaction, and performance is measured using the customer's own business metric. A solution that addresses the wrong problem therefore earns no credit. Delivered systems are deployed without modification and evaluated on held-out production data. The FDE-Gym score measures improvement on the customer's own business metric after deducting stakeholder contact cost. The do-nothing baseline (the unchanged system) scores 0, the expert-authored reference solution scores 1, and literal compliance with the sponsor's request scores at most 0.2. Experimental results show that current models perform well below the reference solution on this scale.

Task structure and instruction levels

To identify where agents fail, each scenario is presented at four task instruction levels with progressively less information — from a fully specified problem (L0) down to the sponsor's original request, which may not address the underlying problem (L3). The environment, stakeholders, and grader remain unchanged across all four levels; only the task instruction varies. Five main findings emerge from evaluating 13 model configurations from seven providers using this setup.

  • Finding 1. Even the best model configuration falls well short of the reference solution. At L3, Claude Opus 5.5 achieves the highest mean score of 0.56 and resolves 14 of 95 tasks, while five of the thirteen configurations resolve none. Selecting the best configuration for each task still yields a score of only 0.70, with 30 tasks resolved.
  • Finding 2. Discovering unstated constraints is the primary bottleneck for leading models. When the relevant rules and constraints are provided (L0), the three leading models score 0.73–0.82 and resolve 11–13 of 20 tasks. Requiring these models to discover the constraints reduces scores sharply to 0.39–0.62. On the original tasks, L3 scores are at most 0.08 points lower than L1 scores.
  • Finding 3. Obtaining the necessary information does not ensure effective delivery. At L3 the three leading models ask 10.0–12.3 questions per task and obtain 72–85% of the key facts. Their normalized business uplift is 0.60–0.70. Even on the 23 tasks where Claude Opus 5.5 obtained every key fact, the uplift reached at least 0.8 on only 10 of them. Stakeholder contact cost further reduces final scores by 0.14–0.22.
  • Finding 4. Weak models can leave customers worse off than if no changes had been made. Claude Haiku 4.5 scores below the do-nothing baseline on 79 of 95 tasks, while systems delivered by Mistral Small 4 fall below this baseline on 51 tasks even before deducting stakeholder contact cost.
  • Finding 5. No model performs well across all scenarios. In five of the 20 scenarios, no model achieves a mean score of at least 0.5. Across tasks within a scenario, the median score range for a leading model is 0.34–0.44, and this range exceeds 0.5 in 6–8 of the 20 scenarios.

How FDE-Gym scores a delivery

FDE-Gym evaluates a delivered system by the improvement it achieves on the customer's business metric, measured on held-out production data. For example, an after-sales system is assessed by its net financial benefit, while a drilling system is assessed by the downtime hours correctly attributed to a contractor for compensation claims.

Three metrics assess agent performance on these delivery tasks.

Normalized Business Uplift (U) measures the delivery's business improvement relative to that achieved by an expert-authored reference solution:

U = (KPI_delivered − KPI_baseline) / (KPI_reference − KPI_baseline)

Here, KPI_baseline, KPI_delivered, and KPI_reference denote the customer's business metric for the unchanged system, the delivered system, and the expert-authored reference solution, respectively, evaluated on the same held-out data. The unchanged system is the do-nothing baseline. Thus, U = 0 represents no improvement, U = 1 matches the reference improvement, and U << 0 indicates an outcome worse than the baseline.

Incident Adjustment (U_adj) sets negative uplift to zero while retaining negative uplift if a predefined operational incident occurs, such as an unauthorized action:

U_adj = U if no incident occurs, min(U, 0) otherwise.

Stakeholder Contact Cost (C) is expressed in the same normalized units as business uplift. It is the sum of the charges incurred by the agent's contacts with the customer: each stakeholder has a hidden per-message rate, with higher charges when the agent exceeds their patience or violates a contact restriction. Unanswered messages are not charged. In scenarios with a trial environment or a calendar, each trial run and each day of waiting is also a chargeable contact.

FDE-Gym Score (S) measures delivery performance in a single evaluation run. It is the incident-adjusted normalized business uplift minus stakeholder contact cost:

S = U_adj − C.

Without an incident, S = U − C. A task is considered resolved if its score is at least 0.8 and no incident occurs.

Average FDE-Gym Score (S̄) summarizes a model's delivery performance across tasks and repeated runs. To prevent extreme business losses from dominating the reported average, the incident-adjusted uplift is floored at −1 before deducting the full stakeholder contact cost. The average is then computed across tasks and runs, with each task receiving equal weight.

Results

In the realistic setting (unstated constraints must be discovered), the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks. Even selecting the best configuration for each task does not close the gap; taking the best score achieved by any of the thirteen configurations across all 95 tasks yields 0.70 with 30 tasks resolved.

The primary bottleneck is discovering unstated constraints. When the relevant rules and constraints are provided (L0), the three leading models score 0.73–0.82. Requiring these models to discover the constraints reduces scores sharply to 0.39–0.62. The first step — from L0 to L1, where the problem is still stated correctly but its constraints must be found — costs more than the two that follow: the decline from L0 to L1 is 0.28 (95% interval 0.17 to 0.39), while the decline from L1 to L3 is only 0.04 (−0.03 to 0.14).

Obtaining the necessary information does not ensure effective delivery. At L3 the three leading models ask 10.0–12.3 questions per task and obtain 72–85% of the key facts. Their normalized business uplift is 0.60–0.70. Even on the 23 tasks where Claude Opus 5.5 obtained every key fact, the uplift reached at least 0.8 on only 10 of them. Stakeholder contact cost consumes 0.14 of the 0.70 uplift delivered by Claude Opus 5.5 and 0.22 of the 0.60 delivered by GPT-6 Astra. Agents are not informed that questions incur a cost, and they ask as if questions were free.

Weak models can leave customers worse off than if no changes had been made. Claude Haiku 4.5 scores below the do-nothing baseline on 79 of 95 tasks, while systems delivered by Mistral Small 4 fall below this baseline on 51 tasks even before deducting stakeholder contact cost. On 23 tasks, the system delivered by Mistral Small 4 caused more harm than the benefit produced by the reference solution.

No model performs well across all scenarios. In five of the 20 scenarios, no model achieves a mean score of at least 0.5. Across tasks within a scenario, the median score range for a leading model is 0.34–0.44, and this range exceeds 0.5 in 6–8 of the 20 scenarios.

Higher spending and effort do not consistently correspond to higher scores. GPT-6.1 Sol spends USD 0.24 and 25 turns per run and scores 0.37, whereas Gemini 3.8 Flash spends USD 1.54 and 121 turns and scores 0.19. Qwen 3.8 Flash works longest, at 169 turns and 110 minutes per run, submits within the limits on only 11 of its 152 task–instruction-level combinations, and scores 0.07. Recognizing when the work is complete appears to be part of the required skill.

Comparison with other benchmarks shows that models that solve up to 88%, 90%, and 61% of tasks on Terminal-Bench 2.1, Vibe Code Bench, and Finance Agent respectively, score 0.19–0.56 on FDE-Gym. The same models that solve most terminal, coding, and research tasks improve the customer's business metric in at most one of seven engagements.

What it means in practice

For a working developer or AI practitioner, FDE-Gym reveals that the gap between solving well-specified tasks and delivering effective AI systems for real customers is substantial. The benchmark makes clear that three distinct challenges must be overcome: clarifying ambiguous requirements, discovering unstated constraints through stakeholder interaction, and achieving measurable improvements on customer business metrics. Simply obtaining the necessary facts from stakeholders does not ensure an effective delivery — the delivered system must also be correctly built and integrated. Weak models can leave customers worse off than if no changes had been made.

The online platform at fde-gym.com lets humans practice FDE skills under the same grading criteria, and both the experiment harness and the 20 original tasks are open source. A practitioner interested in improving delivery performance would need agents that can surface unstated constraints, operate reliably in live systems, and achieve measurable improvements on customer metrics — not just models that comply literally with a task instruction. Closing the identified gap requires agents that can determine when the sponsor's request addresses the underlying problem, discover applicable constraints through interaction, and deliver systems that improve the customer's business metric without incurring prohibitive stakeholder contact costs.

Read the paper on arXiv