A new benchmark of leading AI models on expert-level medical questions reveals both how far these systems have come and where they still fall short. Natomy ran six frontier models through MedXpertQA, a dataset of 4,450 questions spanning diagnosis, treatment, and basic science, including a multimodal subset with real clinical images. The results show meaningful gaps between models, with some organ systems consistently harder than others.
Two Subsets, Different Challenges
The evaluation used two portions of MedXpertQA. The multimodal subset contains 2,000 questions paired with clinical images, each with five answer choices. The text-only subset contains 2,450 questions with ten answer choices, the standard multiple-choice format for medical board exams. Not every model was run on both subsets.
All models were evaluated zero-shot with chain-of-thought reasoning. No fine-tuning, no few-shot examples, and no reasoning-effort tuning. Each score represents a single pass at the provider's default settings, graded by exact-match on the model's final answer letter. GPT-4o's score comes from the original MedXpertQA paper's published leaderboard rather than a fresh run, but it used the same evaluation harness, so the numbers are directly comparable.
GPT-6-Astra Leads, But the Gap Varies by System
GPT-6-Astra posted the strongest scores across most organ systems in the multimodal subset. But the breakdown by organ system shows that some areas are hard for every model. The nervous system and cardiovascular system appear toward the lower end of accuracy across the board, while lymphatic and musculoskeletal questions tend to score higher. The ranking between models holds fairly steady across task types, though the absolute scores shift.
Gemini 3.8 Flash performed competitively, matching or exceeding GPT-6-Astra on several organ systems including lymphatic (94 vs 86) and nervous (92 vs 91). Claude Opus 5 and Claude Sonnet 5 showed a large gap between them, with Opus scoring substantially higher across every category. Claude Sonnet 5 struggled particularly with nervous system questions (57) and cardiovascular questions (57).
A Significant Chunk of "Wrong" Answers Have Defensible Arguments
The most important caveat in the evaluation is the dataset itself. MedXpertQA expands each question's original answer choices up to ten using an LLM-assisted process. That expansion occasionally produces two choices describing the same underlying fact, internally contradictory questions, and in the multimodal subset, low-resolution or mismatched images. In manual review, roughly 17 to 21 percent of the answers models got "wrong" had a real, defensible argument for the model's answer.
This means the raw numbers understate true accuracy for every model by a few points. It also highlights a broader problem with medical benchmarks: the difficulty of building gold-standard evaluation sets at expert level, where even human physicians might disagree on the best answer given ambiguous choices.
What This Means for Medical AI
Single-run evaluations at default settings tell one story. They do not capture what happens with repeated trials, best-of-N sampling, or carefully tuned reasoning effort. They also use smaller samples for organ-system and task-type breakdowns, some as few as 18 questions, which adds statistical noise to those finer-grained results.
Still, the gap between the top models and the bottom is large enough to be meaningful. For teams evaluating which models to deploy in clinical decision support workflows, these numbers provide a baseline. The organ-system breakdowns are particularly useful for identifying where a model might be reliable enough for triage assistance versus where it still needs human oversight.
The fact that multiple models now score above 80 percent on expert-level medical questions with zero-shot prompting is itself notable. Five years ago, this kind of performance would have seemed distant. The current frontier is not whether AI can pass medical exams, but how well it handles the ambiguity, edge cases, and high-stakes consequences of real clinical practice.