Making Codebases Teach Themselves as RL Environments

Reinforcement learning for coding agents needs two things: diverse tasks and reliable rewards. The tasks need to cover real software work, not toy functions. The rewards need to be verifiable, not guessed by a judge model. The bottleneck is not either requirement in isolation. It is the pipeline that turns one of the largest corpora of open-source software in existence into executable RL environments at scale.

Existing methods for constructing coding RL environments rely on development artifacts. They mine issues, pull requests, commits, and existing tests to extract tasks. This works, but it ties task creation to the coverage of recorded changes. Codebases with poor issue hygiene, sparse tests, or undocumented features produce few or no tasks. The task space is shaped by the quality of the project's meta-data, not the richness of its functionality.

CodeMidas, from Xiaomi's LLM Core team with collaborators at Peking University, HKU, and Renmin University, takes a different approach. It uses source code as its only task-specific input. No issues. No commits. No pull requests. No existing tests. No written descriptions. The pipeline explores implemented functionality in open-source codebases, formulates behavioral specifications from what the code actually does, constructs tests grounded in execution of the original code, and validates everything through execution checks and solution rollouts. The result is 5,545 training tasks from 3,185 codebases spanning 23 programming languages and 15 technical domains.

What Makes Source Code a Better Starting Point

Implemented functionality provides both the basis for a task and a candidate solution. Its public interfaces and observable behavior define what an agent should implement. Executing the original code provides evidence for test expectations. The surrounding codebase can be adapted into a development starting point that preserves real project structure and dependencies. These elements support the construction of task statements, development environments, and executable verifiers without requiring any external meta-data.

The key insight is that the task statement should make required behavior explicit while leaving internal implementation choices open. Tests must enforce these requirements, rejecting incorrect solutions while still accepting alternative correct implementations. This is the test oracle problem applied to RL environment construction: tests that are too specific reject valid solutions, while tests that are too loose accept invalid ones.

CodeMidas solves this by grounding every test in execution of the original code. For outputs and properties fixed by the statement, assertions use reference execution to establish expected values. For aspects left unspecified, assertions check only the stated constraints. A reviewing agent then examines every assertion for restrictions unsupported by the statement, such as exact wording, incidental ordering, or internal structure, and replaces these with behavioral checks while preserving the checks required by the statement.

The Four-Stage Pipeline

Stage one, task design and codebase adaptation, has an agent inspect codebase structure and build metadata to identify functionality with public entry points and observable outcomes. Supported interfaces include command-line tools, pure library functions, and stateful library APIs. The agent traces public entry points and shared dependencies to define the task scope, removes the selected core implementation, adjusts the remaining code to form a coherent starting point, and retains the original implementation separately as a reference solution.

Stage two, execution-grounded test construction, maps behavioral requirements to test inputs and boundary cases. Tests use command executions for CLI tools, input-output cases for pure functions, and sequences of calls for stateful APIs. Stateful tests exercise dependencies across calls, including ordering and cleanup behavior. Each test records the specific requirement that it covers. A reviewing agent then checks assertions for over-specification and replaces behavioral checks where appropriate.

Stage three, environment preparation, starts from a uniform base container image and installs dependencies according to project declarations. Cleanup removes artifacts that could reveal the deleted implementation, including compiled outputs, cached copies, and files left by construction agents. Original tests related to the target functionality are also removed. Each task is then checked under training runtime settings in six fresh containers: two with the starting codebase (both must fail) and four with the reference solution (all must pass). These repetitions check the expected fail-to-pass transition and screen for unstable execution outcomes.

Stage four, post-rollout environment filtering, uses agent rollouts to further clean the dataset. Three checks run. Leakage filtering has an adversarial agent try to exploit residual leakage to recover a solution without doing the intended work, searching compiled artifacts, caches, and installed copies. Agreement on agent solutions has a coding agent try four times per task, then a reviewing agent checks whether the verifier correctly accepts correct implementations and rejects incorrect ones, flagging false positives and false negatives. Rollout outcome filtering keeps only tasks where a frontier model produces both successful and failed attempts, discarding tasks where everything passes or everything fails, since those outcomes may reflect weak tests or missing requirements.

Each task consists of a statement, a containerized development environment, and a hidden executable verifier. The solver receives the statement and adapted codebase with dependencies. The verifier is kept outside the solver's environment and injected only at grading to evaluate the completed implementation and return a binary execution reward for RL.

The Dataset

The pipeline retains 5,545 tasks from 3,185 codebases across 23 languages and 15 technical domains. Python accounts for 21.4%, TypeScript 18.3%, Go 16.2%, C++ 12.5%, and JavaScript 11.3%. The ten most frequent languages cover 98.2% of tasks. Technical domains include systems software (17.4%), web technologies (14.6%), and developer tools (13.6%), together accounting for 45.6%.

Reference solution size has a median of 142 lines, with an interquartile range of 66 to 305 lines. Reference patches touch at least two source files in 65.9% of tasks. The tasks are not trivial. They require reasoning across codebases, not single-function completions.

Training and Results

The authors train MiMo-V2.5 on the 5,545 tasks using GRPO with binary execution rewards, batch size 32, and 32 rollouts per task. They evaluate on five external benchmarks covering diverse forms of software work.

DeepSWE pass rate rises from 10.0% to 21.7%, a +11.7 percentage point gain. ProgramBench Almost Solved score rises from 4.5 to 21.5, a +17 point gain. Terminal-Bench v2.1 pass rate rises from 63.7% to 72.2%, a +8.5 point gain. These improvements span issue repair, whole-program construction, code translation, and terminal work.

On CodeMidas Val (200 randomly sampled tasks separate from the training set), pass rate rises from 35.0% to 44.7%, staying roughly 8 to 10 percentage points above the initial rate at evaluated checkpoints from step 40 onward. These gains accompany longer trajectories, indicating greater use of the available interaction budget.

Task Scale and Quality Matter More Than Raw Count

The ablation study is the paper's most practically important result. Training on 1k, 3k, and 5,545 high-quality tasks yields progressively higher scores on all three evaluations (DeepSWE: 17.57 to 19.05 to 21.70; CodeMidas Val: 41.30 to 43.22 to 44.73). This supports scaling high-quality training data.

The critical comparison is against a vanilla 8k sample drawn before filtering and cleaning, each with a task statement, development environment, and verifier, but without environment cleaning, execution consistency checks, or any of the three post-rollout filtering steps. The full CodeMidas dataset exceeds the vanilla 8k sample by 0.59, 4.59, and 4.49 percentage points on SWE-bench Pro, DeepSWE, and CodeMidas Val respectively. Even the high-quality 3k subset outperforms the vanilla 8k sample on all three evaluations.

This is the paper's clearest finding: a smaller, cleaned dataset beats a larger unfiltered one. The filtering pipeline is not overhead. It is the source of the training signal's quality. Environment reliability and training suitability, provided by cleaning, execution checks, and post-rollout filtering, matter more than raw task count.

What RL Trained Agents Do Differently

Trajectory analysis reveals three behavioral changes during RL training. Codebase exploration, measured by read and search calls before the first edit, increases from 27.2 to 40.1 calls. Code drafting, measured by the fraction of code fragments in Write or Edit payloads already present in preceding reasoning, increases from 0.358 to 0.629. Self-verification, measured by distinct verification commands after the final repository edit, increases from 2.03 to 2.53.

Self-verification is associated with higher pass rates. Within the same task and checkpoint on CodeMidas Val, rollouts with checks written and executed by the agent have a mean pass rate 4.2 percentage points higher than those without (95% CI: 1.8 to 6.6). Exploration and drafting show smaller, less confident associations.

These behavioral changes generalize beyond CodeMidas. On SWE-bench Pro, exploration increases from 23.1 to 35.5 calls and drafting ratios increase from 0.304 to 0.653. On ProgramBench, exploration increases from 55.7 to 83.6. On Terminal-Bench v2.1, exploration increases from 11.9 to 16.8. The interaction between exploration and interaction length varies by task type: on SWE-bench Pro, mean interaction length increases from 37.3 to 50.1 turns; on ProgramBench, it decreases from 155.1 to 122.8. Greater codebase exploration accompanies shorter overall interactions on whole-program construction.

How This Compares to Existing Pipelines

The paper includes a useful comparison table showing what each pipeline requires as task-specific input. SWE-rebench V2 requires commits and existing tests. R2E-Gym requires commits and existing tests. SWE-smith requires pull requests and commits. SWE-Flow requires existing tests. SWE-Hub requires existing tests. R2E requires written descriptions. MindForge requires written descriptions. CodeMidas requires none of these. Source code is the only task-specific input.

This matters because it determines the ceiling on task diversity. Pipelines that depend on issues produce only tasks that someone bothered to report. Pipelines that depend on tests produce only tasks that someone bothered to test. Pipelines that depend on documentation produce only tasks that someone bothered to document. CodeMidas can produce tasks from any functionality that exists in the code, regardless of whether it was reported, tested, or documented.

The language coverage difference is also significant. Most existing pipelines target a single language. CodeMidas covers 23, inheriting the breadth of the open-source codebase corpus it draws from.

What This Means for Training Coding Agents

The practical finding is that source code itself is a scalable foundation for coding RL. Implemented functionality can be transformed into verifiable learning environments that support generalization across diverse forms of software work. The pipeline does not require human annotation, existing tests, or development meta-data. It requires only the code and the ability to execute it.

The quality-over-quantity result is particularly actionable. Teams building coding RL environments should invest in filtering and validation rather than raw scale. A 3k dataset that has been cleaned, tested, and filtered through adversarial rollouts outperforms an 8k dataset that has not. The cost of filtering is real, but the training signal it produces is disproportionately valuable.

The behavioral analysis suggests that RL training teaches agents to explore more and verify their own work more thoroughly. These are behaviors that transfer across task types, which explains why training on CodeMidas tasks improves performance on benchmarks the agent has never seen. The agent is not memorizing solutions. It is learning to interact with codebases more effectively.

For developers building or fine-tuning coding agents, CodeMidas provides both a dataset and a recipe. The 5,545 tasks are released, and the pipeline is described in enough detail to replicate. The key steps, execution-grounded test construction, adversarial leakage probing, and rollout outcome filtering, are applicable to any codebase you want to turn into training data.

Read the paper on arXiv