Marginlab launched a daily performance tracker for OpenAI's Codex running gpt-5.6-sol, aimed at catching statistically significant regressions on software engineering tasks before they affect developer workflows. The tracker is independent of OpenAI and other model providers, and it benchmarks the model exactly as users encounter it: in the Codex CLI, with no custom agent harness.
Why a Third-Party Tracker Matters
AI coding assistants are becoming load-bearing tools in production environments. When a model degrades, the impact is real but hard to measure from the outside. Marginal changes in pass rates on benchmark tasks can signal shifts in model behavior, prompt handling, or harness logic that affect real-world reliability. Marginlab's tracker runs daily evaluations on a curated, contamination-resistant subset of SWE-Bench-Pro to surface these changes with statistical rigor.
The Baseline and the Method
The frozen baseline for gpt-5.6-sol is 83.40%, derived from 613 passes across 735 valid trials. Those trials come from all 15 available high and xhigh reasoning runs conducted between July 11 and 24, 2026. High and xhigh runs are treated as one model series for baseline purposes.
Each day, the tracker attempts N=50 test instances using the latest available Codex release with high reasoning effort. Infrastructure failures and cancellations are counted as attempted evaluations but excluded from the statistical denominator. Pass rates are modeled as Bernoulli random variables, and 95% confidence intervals are computed around daily, weekly, and monthly rates. A degradation is flagged when the observed pass rate falls below the frozen baseline with a one-sided p-value below 0.05.
The daily significance threshold is ±11.2% around the baseline. The weekly threshold, which aggregates seven days of data for more stable estimates, tightens to ±3.6%.
What Gets Tracked
Beyond pass rates, the tracker logs input tokens, output tokens, average per-instance runtime, and total tool invocations per day. These resource and execution metrics provide context for performance shifts: a drop in pass rate paired with a spike in tool calls or runtime might point to a harness change rather than a model change, or vice versa.
The daily trend chart shows pass rate over time with an optional 95% confidence interval overlay. The weekly trend aggregates the same data into seven-day windows for smoother signal. A change overview section summarizes performance deltas by period.
Codex CLI as the Benchmark Environment
The choice to benchmark directly in Codex CLI, rather than through a custom harness, is deliberate. It means results reflect what actual users experience. If OpenAI changes how Codex handles tool calls, manages context, or retries failed steps, those changes show up in the tracker alongside any model-level shifts. This makes the tracker a measure of the end-to-end system, not just the underlying model.
The tracker currently uses high reasoning effort. Scheduling and methodology details are documented alongside the results, and the full dataset is updated daily at marginlab.ai/trackers/codex.
For teams evaluating Codex as a coding assistant or monitoring it in production, the tracker provides a concrete, auditable signal. The statistical framework is simple (Bernoulli trials, one-sided hypothesis tests), the baseline is frozen and reproducible, and the independence from model providers removes a obvious conflict of interest.