Quantitative research in finance follows a repetitive cycle: a hypothesis emerges, gets coded into a signal, gets validated against past price data, and either gets kept or discarded. Existing benchmarks evaluate individual pieces—formulas, code snippets, or single trading decisions—but they do not test whether a large language model can see a hypothesis through to a strategy that survives beyond the research window. Quant-Arena enters this gap as a benchmark and public leaderboard for end-to-end quantitative research on real A-share market data. The setup asks a model to build features and stock-scoring code, receive feedback only on the 2022–2023 period, then freeze one strategy for assessment on the sealed year 2024. A cash-account evaluator applies trading constraints and costs, including market impact estimated from a simulated order book. Scores lean heavily on realized excess return and are supplemented by signal quality and the diversity of features the strategy employs. Nine frontier models were put through the test, and GPT-6 Astra emerged on top, yet none could beat the equal-weight market in 2024. Quant-Arena remains open to any model or agent harness that wishes to submit.
The iterative research loop and its automation challenge
In quantitative finance, a researcher begins with a market intuition—perhaps that a specific valuation ratio predicts near-term returns, or that momentum carries through a particular liquidity window. That intuition gets translated into code: an executable signal, a factor model, or a set of features that score stocks. The coded signal then runs on historical data, producing a series of returns. The researcher examines those returns, adjusts the code or the feature set, and repeats. This loop can iterate many times before a strategy is deemed robust enough to trade.
Benchmarks that have emerged alongside large language models typically score one of three things: the correctness of a formula, the syntactic quality of generated code, or the accuracy of a single trading decision in a simulated environment. What they leave unanswered is whether an LLM can orchestrate the full loop: propose, implement, test, select, and freeze a strategy that retains its edge when confronted with truly unseen data. Quant-Arena was built precisely to address that question.
From formula scoring to end-to-end strategy synthesis: the gap Quant-Arena fills
Prior benchmarks tend to treat the researcher's loop as a sequence of independent tasks. A model might be asked to write a moving-average crossover strategy, and the evaluation checks whether the code compiles or whether the crossover points align with a known benchmark. Another benchmark might present a formula and ask the model to predict its sign on a test set. These are useful for measuring surface-level competence, but they do not measure whether the model can build a self-contained research pipeline that produces a strategy with real out-of-sample longevity.
Quant-Arena shifts the evaluation from component quality to pipeline durability. The model is given a window of historical data on which it may experiment freely—constructing features, writing scoring code, running backtests. After this exploratory phase, the model must commit to a single strategy. That strategy is then frozen and evaluated on a later year that was not available during the exploratory phase. No further tuning is permitted. The result is a single, immutable performance number for that strategy on the sealed year. This design forces the model to discover patterns that generalize, rather than overfitting to the data it has already seen.
Architecture of Quant-Arena: feature construction, strategy freezing, and cost-aware evaluation
The benchmark operates on real A-share market data, covering a broad universe of equities listed on Chinese exchanges. The time axis is divided into three non-overlapping periods: a training window, a feedback window, and a sealed test year. During the training and feedback windows—2022 and 2023—the model constructs features and stock-scoring code. These features can be any computable function of price, volume, order-book depth, or other market–microstructure data that is available at the time of signal generation. The model writes code that, given a universe of stocks, outputs a score for each instrument. Higher scores are expected to correspond to higher future returns.
Once the model submits a strategy, it is locked in. The cash-account evaluator then applies a set of realistic constraints: position limits, turnover caps, execution costs, and market impact. Market impact is not modeled as a simple fixed percentage; instead, it is estimated from a simulated order book that reacts to the size of each proposed trade. A large order against a thin limit-side of the book incurs higher impact than the same order against a deep book. This mechanism means that strategies which look profitable on paper can be eroded quickly when actual execution costs are accounted for.
The evaluator computes several metrics. The primary score is realized excess return, which subtracts a risk-free rate from the strategy's gross return. Secondary scores include signal quality, measured by the Sharpe ratio of the feature-based scoring function across the evaluation period, and feature diversity, which counts the number of distinct functional forms the strategy employs (e.g., moving averages, statistical arbitrage pairs, machine‑learning embeddings). These complementary metrics ensure that a strategy is not judged solely on a single lucky return number.
The nine frontier models and their performance profile
Nine recent large models were evaluated on Quant-Arena, each given the same data windows, the same feature-language constraints, and the same evaluation rules. The models vary in architecture—some are decoder-only transformers, others use mixture-of-experts routing, and a few integrate retrieval-augmented generation for market data. All were prompted to construct features and scoring code in a Python-like DSL that the evaluator can parse and run.
GPT-6 Astra placed highest among the nine, producing strategies that achieved the largest realized excess return on the 2024 sealed year. Its feature set included a mix of technical indicators and a learned embedding over order-book snapshots, and the accompanying code respected position limits and turnover caps without requiring post-hoc fixing. Despite its strong performance, the realized excess return still fell short of the equal-weight market return for 2024, which serves as a baseline that requires no model input other than equal capital allocation across all constituents.
The remaining eight models produced strategies with lower excess returns, and several showed deteriorating signal quality when market impact was factored in. A few models overfit to the 2022–2023 feedback window, producing strategies that collapsed in 2024. Others, while producing respectable returns, used such a narrow set of feature types that their diversity score was minimal, raising concerns about robustness if market regimes shift.
Why the equal-weight market benchmark matters
The equal-weight market benchmark is a simple strategy: at each rebalancing point, every stock in the universe receives the same dollar amount of capital. No forecasting, no feature engineering, no signal is required. The return of this benchmark in 2024 represents the return an investor would earn by simply holding the market and doing nothing else. It is not an easy baseline— A-share markets in 2024 included periods of high volatility, sector rotation, and liquidity squeezes—but it is a deterministic, rule‑free point of comparison.
That no model surpassed this baseline is the most striking result of the Quant-Arena evaluation. It suggests that, even with massive parameter counts and training on broad code and text corpora, current LLM agents still struggle to discover predictive patterns that generalize past the most recent historical window. The equal-weight benchmark also serves as a reminder that transaction costs, market impact, and the simple difficulty of beating a broad index are substantial barriers.
Feature diversity and signal quality as complementary metrics
Realized excess return alone can be misleading. A strategy might post a high return in one sealed year by accident, or by exploiting a short-lived market anomaly that will not repeat. Signal quality, captured by the Sharpe ratio of the scoring function, measures how consistently the model's features rank stocks by expected return across time. A high Sharpe indicates that the model has learned a stable mapping from market data to future performance, rather than a lucky coincidence.
Feature diversity adds another layer of scrutiny. A strategy that relies on a single technical indicator, such as a 20-day relative strength index, may achieve a high return if that indicator happens to be informative in the evaluation period. But if the market environment changes—say, a regime shift away from mean-reversion—the strategy may fail catastrophically. By counting the number of distinct functional forms a strategy employs, the diversity metric rewards models that construct multi-faceted signals. In the Quant-Arena results, models with higher feature diversity tended to have more stable signal quality, though the correlation was not perfect.
Operational constraints: trading costs, market impact, and the simulated order book
One of the most novel aspects of Quant-Arena is the inclusion of a cash-account evaluator that models real-world trading frictions. Execution costs are not a flat fee per trade; instead, they depend on the size of the order relative to the simulated order book's depth. If a model decides to buy 10% of the daily volume of a small-cap stock, the simulated order book reflects the price slippage that would occur. If the same model buys 10% of a highly liquid large-cap stock, the impact is proportionally smaller.
This design means that strategies heavy in small-cap or illiquid names can appear profitable on a gross-return basis but become loss-making once impact is deducted. Several of the evaluated models learned to concentrate capital in low-priced, low-volume stocks, a pattern that inflated backtested returns but collapsed when the simulated order book was applied. The cash-account evaluator penalizes such behavior, bringing the final scores closer to what a real trader would experience.
Market impact estimation also interacts with the feature‑selection process. Features that predict price movement in a frictionless backtest may lose predictive power when the cost of trading against the predicted direction is accounted for. Quant-Arena's dual reporting of gross and net (impact‑adjusted) returns makes this tension visible.
What the results mean for developers building trading agents
For engineers building LLM‑driven trading systems, Quant-Arena offers a concrete evaluation protocol. Rather than reporting "the model wrote correct code" or "the model predicted the next price direction," the protocol demands that the model produce a complete, frozen strategy that is then tested on unseen data with realistic costs. This protocol can be adopted internally: a team can define their research window, their cost model, and their sealed test period, then run their agent through the same loop.
The results also suggest practical design choices. Models that performed best were those that diversified their feature sets rather than latching onto a single indicator. Models that respected position limits and turnover caps from the outset fared better after the cash-account evaluator applied market impact. And perhaps most importantly, the equal-weight market benchmark reminds developers that simple, low‑friction strategies remain a strong baseline; any LLM‑generated approach must clear that hurdle after costs are accounted for.
Quant-Arena is not a fixed test suite; it is open to submissions from any model or agent harness. This openness means that as new architectures and training regimes appear, they can be evaluated on the same end‑to‑end loop, providing a continuously updating leaderboard of quantitative‑research competence.