Causal discovery aims to recover directed causal relationships from data, and its evaluation depends on structural causal models: a causal graph plus the data-generating mechanisms that produce observations and interventions. But different studies use different graph families, mechanisms, noise distributions, sample sizes, and scoring protocols, making it hard to compare methods. The problem gets worse with causal discovery foundation models (CDFMs) like FoundCause, TabCausal, AVICI, Arrow, and CDFM itself. These models are pretrained on large collections of synthetic causal environments, which means their benchmark performance can reflect not just causal discovery ability but also how much the pretraining environments overlap with the test data. Once a fixed benchmark becomes public, it can enter future pretraining corpora. The test distribution becomes part of the training distribution.
Zi-Rong Li, Si-Yang Liu, Tian-Zuo Wang, and Han-Jia Ye at Nanjing University introduce CausalArena to address this. It is a unified and evolvable benchmark containing 1,200 executable SCM specifications across three complementary families: 1,000 synthetic SCMs for controlled breadth, 100 semantic operational SCMs with human-auditable real-world meaning, and 100 formula-grounded scientific SCMs anchored in explicit equations. All three families compile to a common executable specification and share the same observational and interventional evaluation protocol. The authors evaluate 18 methods spanning classical algorithms, neural approaches, and pretrained foundation models, and the central finding is that rankings shift substantially across families: no method dominates every slice, and strong performance on synthetic SCMs does not reliably transfer to semantic, scientific, or real-world data.
Why foundation models change the evaluation problem
Classical causal discovery methods are fitted independently on each dataset. A pretrained method like TabCausal or FoundCause learns reusable parameters from a collection of pretraining environments and then applies those parameters to new datasets. This means a reported score is not just a measure of causal structure-learning ability. It is a measure of that ability plus the overlap between the model's pretraining distribution and the evaluation graphs, mechanisms, and generators. If a pretrained model was trained on Erdos-Renyi graphs with linear mechanisms, its high score on an Erdos-Renyi benchmark tells you less about general causal discovery ability than it appears to.
The situation is further complicated by the fact that once a benchmark is public, its SCMs can enter future pretraining corpora. A permanently hidden test set would make construction and evaluation hard to inspect, but public benchmarks are vulnerable to contamination. CausalArena addresses this by treating pretraining overlap as part of how a result is interpreted, not as a problem to be eliminated. Training disclosure and labeled splits clarify what a reported score uses, and the shared executable interface allows newly constructed SCMs to be added as the arena evolves.
Four design requirements
The authors frame the benchmark around four requirements that pull in different directions. Breadth requires diverse graph families, mechanisms, noise distributions, dimensions, and intervention settings so that performance is assessed across many causal environments, not just one generator prior. Freshness requires that the evaluation pool can grow when pretrained models evolve, since no fixed public SCM suite can remain unseen indefinitely. Grounding requires that some environments attach operational or scientific meaning to variables, edges, and interventions, so a recovered edge is not just a matrix entry but an interpretable claim about a named process. Diagnosability requires going beyond a single pooled score to identify which structures, mechanisms, and environments cause methods to succeed or fail.
No single SCM family can carry all four equally. Synthetic SCMs are practical for breadth and factor-level diagnosis but lack meaning. Semantic and formula SCMs provide grounding but are harder to enumerate at the same scale. CausalArena assigns these roles to three complementary families, then compiles them to one executable representation so that sampling, interventions, wrappers, and scoring stay comparable.
Synthetic SCMs: the controlled breadth layer
The synthetic family contains 1,000 configurations over dimensions 10, 20, 30, 50, and 100, with 20 SCMs per graph-family-dimension pair. It spans 10 graph families (Erdos-Renyi, scale-free, chain, tree, layered, bipartite layered, small-world, block-sparse layered, dense local-module, collider-rich), 16 mechanism families (linear additive, polynomial, multiplicative, rational, exponential, trigonometric, piecewise, saturation, mixture, random neural, RFF/GP, tree-based, and others), 14 root-distribution families, 15 noise families, 4 root-dependency families, and 9 difficulty settings. The design deliberately varies factors that can otherwise become implicit generator priors: varsortability, noise heterogeneity, root correlations, and mechanism complexity.
Deterministic quality gates check DAG validity, mechanism executability, numerical ranges, and degenerate variables before release. The synthetic family's job is to provide the grid over which factor-level diagnosis works: you can read which graph families or mechanism types favor which methods, and where pretrained models succeed or fail relative to their documented pretraining coverage.
Semantic operational SCMs: meaning you can audit
The semantic family contains 100 scenarios from 10 operational domains: cybersecurity and IT operations, education, finance and credit, government services, healthcare delivery, housing and real estate, manufacturing, public health, urban transportation, and water and sanitation. Each domain contributes 10 scenarios with 17 to 25 variables and 25 to 67 directed edges per scenario. Variables are labeled by what they mean in the process, directed edges carry reasons for why changing a parent can affect a child, and interventions change settable fields with descendant recomputation.
The wildfire smoke scenario illustrates this concretely. Local outdoor PM2.5 is the true air near the house. The official monitor sees the same plume but not perfectly, so its reading can differ. If that reading is high enough, a shelter-in-place alert may be issued. The alert and previous indoor PM2.5 shape perceived smoke risk, which shortens hours outdoors and increases outdoor exposure reduction. Housing leakage sets effective infiltration, a portable filter raises device CADR, and room volume and adherence friction matter. Local outdoor PM2.5, infiltration, filtration, and behavior determine true indoor PM2.5. A consumer sensor gives a noisy reading, an indoor exceedance flag is set, daily inhaled dose combines hours outdoors with indoor and outdoor contributions, and respiratory vulnerability turns that dose into symptoms. Removing the edge from portable filter to true indoor PM2.5 drops a real control people can change, and treating the consumer sensor reading as the true indoor level mixes a measurement with the physical state.
Because semantic SCMs are authored through a reusable construction pipeline, new operational scenarios can be added without changing the evaluation interface. This provides the freshness pathway: as pretrained models evolve, new scenarios can be authored to test whether the model's capabilities generalize beyond its documented pretraining environments.
Formula-grounded scientific SCMs: equations with units
The formula family contains 100 scenarios spanning 10 scientific and engineering domains: biology and ecology, astronomy, chemistry, mechanics and fluids, earth systems, electromagnetism, energy systems, materials and structures, optics and waves, and thermodynamics. Each domain contributes 10 scenarios with 16 to 25 variables and 18 to 50 edges per scenario. Some mechanisms are named scientific equations with units and valid ranges written down. Formula nodes are computed exactly from their parents; randomness sits on inputs, instrument readings, or other non-formula quantities, not on arbitrarily jittering the equation itself.
The barometric pressure scenario demonstrates this. A raw barometer reading is corrected by a calibration drift bias that depends on the observation window (day versus night, changeable). Surveyed elevation and GPS altitude combine with an altitude fusion weight into fused station height, setting effective gravity. Ambient air temperature, biased by solar exposure, becomes the shielded thermometer reading. Relative humidity and shielded temperature determine layer-mean virtual temperature. Finally, calibrated station pressure, fused height, effective gravity, and layer-mean virtual temperature enter the hypsometric sea-level reduction equation, with dry-air gas constant as the physical constant. Changing the fusion weight or observation window recomputes later nodes in the same executable SCM, and residual checks keep non-intervened formula rows consistent with the reduction equation.
The formula family supports mechanism-level diagnosability. When a method fails on a formula-grounded scenario, the error can be traced to a specific equation or measurement step, rather than to an anonymous mechanism in a synthetic generator.
Results: ranking instability across families
The headline numbers tell a clear story of ranking instability. On the observation-only protocol, FoundCause achieves the highest average F1 at 0.59, followed by TabCausal at 0.48 and CDFM at 0.42. FoundCause leads on all three synthetic families (Syn 0.62, Sem 0.63, Form 0.52). But on real-world data, FoundCause drops to 0.21, while CDFM leads at 0.39 and TabCausal at 0.34. The method that dominates synthetic benchmarks is not the method that works best on real data.
Among classical methods, GIES achieves the best average F1 at 0.51 and is particularly strong on intervention-aware slices (best Semantic obs+int F1 at 0.57, Formula obs+int at 0.52). GIES uses interventional data effectively, while many continuous optimization methods like NOTEARS and DAGMA do not consume intervention indicators and are run only on observation-only splits. The graphical search methods (CDIS, GIES, IGSP, PC) show different strengths: GIES excels on semantic and formula data, CDIS is more stable across noise types, and IGSP is highly sensitive to intervention protocol design.
The pretrained methods show the most dramatic ranking shifts. FoundCause dominates on synthetic data (0.62 obs-only) but collapses to 0.21 on real data. TabCausal is more stable across families (0.50 Syn, 0.42 Sem, 0.47 Form) and leads on real-data intervention-aware evaluation (0.46). AVICI improves substantially from observation-only to intervention-aware evaluation (0.33 to 0.41 on Formula), suggesting its pretraining includes interventional settings that help it exploit interventional data. CauScale shows a pattern where its performance rises with dimension (the only method to do so), peaking at 0.48 for d=100, while most other methods decline.
What the factor analysis reveals
The factor-level analysis decomposes where methods succeed and fail. On graph structure, bipartite-layered graphs strongly favor CDIS (+0.192 double-centered F1) and GIES (+0.120), while sparse branching graphs (trees, scale-free) favor continuous-optimization methods and AVICI. On mechanisms, linear-additive settings favor LiNGAM (+0.069) and GIES (+0.053), while tree-based mechanisms favor AVICI (+0.100) and CDFM (+0.086). AVICI shows the largest mechanism interaction variance, meaning its performance varies most across mechanism types.
Dimension scaling reveals that FoundCause starts at 0.67 for d=10 and declines to 0.56 for d=100, remaining the highest throughout. CauScale is the only method that rises with dimension (0.35 at d=10 to 0.48 at d=100). Arrow and CDFM decline sharply (Arrow from 0.45 to 0.21, CDFM from 0.54 to 0.27). These patterns show that scalability is not uniform: some pretrained models generalize to higher dimensions while others do not.
Intervention protocol sensitivity shows that TabCausal and SDCD vary least across protocols (F1 standard deviation 0.009 and 0.010), while IGSP is most sensitive (standard deviation 0.100, dropping 0.27 under multi-target interventions). This means some methods are robust to how interventions are designed while others are fragile, a property that a single pooled score does not capture.
Pretraining-evaluation overlap analysis
The authors map each pretrained model's documented pretraining coverage against CausalArena's factor taxonomy. FoundCause has the highest mean coverage at 49% (covering 13 of 16 mechanism families), followed by TabCausal at 45% (covering 5 of 10 graph families, 8 of 16 mechanisms, 8 of 15 noise types). Arrow and AVICI have the lowest coverage (18% and 22%).
The critical question is whether high performance correlates with high pretraining overlap. FoundCause's OOD relative gain is about 60% above the graphical-search reference on held-out categories, meaning it performs well even on factors not documented in its pretraining. Its ID-OOD gap is essentially 0%, so its lead is not concentrated on documented support. TabCausal shows a similar pattern with a 4% ID-OOD gap. AVICI shows the largest gap at 6%, driven by graph (+9%) and root-family (+10%) differences between in-distribution and out-of-distribution performance. This means AVICI's reported score may be inflated by pretraining-evaluation affinity on those axes.
Real-data results confirm the instability
On real-world datasets (6 observational from CD-CSG, 7 interventional from Sachs, PetShop, and Causal Chambers), the rankings shift again. CDFM leads observation-only F1 at 0.39, while FoundCause drops to 0.21. TabCausal leads intervention-aware evaluation at 0.46. GIES achieves 0.29 on intervention-aware real data, the strongest classical method. These results confirm that synthetic benchmark rankings do not carry over to real data, and that the ranking instability is not an artifact of the synthetic generation process.
Runtime and resource costs
The pretrained methods are orders of magnitude faster per graph than classical optimization. Arrow processes a graph in 0.077 seconds (load amortized), AVICI in 0.096 seconds, CDFM in 0.199 seconds, and TabCausal in 0.210 seconds. FoundCause is slower at 4.754 seconds per graph but achieves the highest F1. Classical methods range from GIES at 2.78 seconds to SDCD at 192.64 seconds per graph. The accuracy-cost frontier is set by GIES (strong classical baseline at low cost) and TabCausal (strong pretrained at similar cost to faster but weaker pretrained methods). FoundCause pushes the frontier upward at higher cost.
Sample-size scaling shows that most F1 gain occurs between n=100 and n=1,000, with curves flattening beyond n=2,000. FoundCause improves from 0.43 at n=100 to 0.66 at n=10,000. TabCausal improves from 0.33 to 0.57. Arrow and CauScale peak near n=1,000 to 2,000 then decline, suggesting they are sensitive to sample-size shift from their pretraining distribution.
Limitations the authors acknowledge
Pretraining-evaluation overlap cannot be fully removed once an SCMs or generator is public. Continued evaluation benefits from periodically extending the SCM pool, but this is a moving target as pretrained models grow. Published real-world graphs are not independently verified unique true DAGs; they are expert-specified or experimentally motivated in source papers, which introduces ground-truth uncertainty that the benchmark inherits rather than controls.
The current release focuses on multivariate tabular causal structure recovery with known directed graph ground truth. It does not cover latent confounding, time-series causal discovery, or non-tabular data. The same executable interface could later support these extensions, but they are not yet implemented. The 1,200 SCMs provide meaningful breadth, but the semantic and formula families are harder to enumerate at the same factor resolution and scale as the synthetic family, limiting how densely the design space can be swept.
What this means for the causal discovery community
CausalArena demonstrates that a single leaderboard score is an incomplete summary of causal discovery ability. Rankings shift across SCM families, mechanisms, sample sizes, and intervention protocols. A method that dominates on one family can be mid-pack on another. This is not a failure of benchmark design; it is a property of the problem. Causal discovery is not one task but many, and different methods make different assumptions that favor different causal environments.
For practitioners, the practical takeaway is to evaluate methods on multiple SCM families before deploying them. If a method is to be used on real-world causal questions, real-data results and semantic/scenario-grounded evaluation matter more than synthetic benchmark scores. For researchers building foundation models, the pretraining-exposure analysis provides a diagnostic tool: if a model's high benchmark score correlates with high pretraining overlap on the evaluated factors, the score should be interpreted cautiously. The OOD relative gain metric quantifies how much of a model's performance comes from genuine generalization versus pretraining-evaluation affinity.
The evolvable design is the benchmark's most important feature for the long term. As new causal discovery foundation models appear and pretrained corpora grow, CausalArena can add new SCMs under the same executable interface without rewriting how methods are compared. This addresses the fundamental tension between public reproducibility and freshness: the benchmark can evolve while the protocol stays fixed.
Read the paper on arXiv