Incredibuild's internal Software Factory recently turned its attention to Bazel, the build system behind much of Google's infrastructure, and came back with six upstream contributions and a security finding that should make any team using remote caching sit up. The result is a case study in what happens when you run an autonomous engineering loop against real production code: deterministic outputs, complete evidence, and clear authority boundaries stop being abstract ideals and start being survival requirements.

Why Bazel Was the Right Target

Bazel models software as a directed graph of declared work. It makes caching and remote execution first-class concepts and treats reproducibility as part of correctness. Those are precisely the problems Incredibuild has spent over two decades solving from a different layer of the stack. But Bazel is also a demanding target. Its Remote Build Execution model is powerful, and adopting it requires hermetic actions, rule coverage, toolchains, platforms, and REAPI infrastructure to all line up correctly. Real repositories rarely fit inside a clean action graph anyway; they contain nested Make and Ninja builds, custom compilers, test harnesses, scripts, packaging pipelines, and proprietary tools.

That is exactly why Bazel mattered to Incredibuild twice over: as a workload worth accelerating, and as a build system they needed to understand down to the cache boundaries.

How the Software Factory Operates

The Software Factory is not a demo where an agent writes code and walks away. It runs the entire engineering loop as a factory line: goal, plan, isolated task, implement, build, test, inspect, retry, and reviewable pull request. Orchestration manages the project as a DAG of tasks. Sandboxes isolate every individual task. Incredibuild handles the expensive build and test loops. Agents explore aggressively, but tests, evidence, and human review decide what gets promoted.

One rule does not bend: automated search and model output can propose patches, run experiments, or return cached results, but test verification and human review retain authority over what merges to main.

Six Upstream Contributions

Six changes from this effort were imported to Bazel master through Google's Copybara workflow. None of them is flashy. All of them serve the same purpose: making automation trustworthy.

  • PR #30365 made bazel-distfile.tar reproducible by removing locale-dependent ordering and made unknown archive types fail explicitly.
  • PR #30358 skipped the SSL tracking-issue mutation step on pull-request validation runs, so a PR can validate a monitor without ever creating or updating production tracking issues.
  • PR #30364 ignored COMMENTED reviews when computing community-review status, preventing a comment from erasing a prior APPROVED or CHANGES_REQUESTED decision.
  • PR #30363 parsed certificate expiration independently of host locale, so monitoring stopped depending on an unrelated LC_TIME setting.
  • PR #30360 corrected the documented BAZEL_DOC_TRIGGER_TOKEN scope to bazel-docs, matching the repository the workflow actually dispatches to.
  • PR #30359 paginated PR-review retrieval and made it fail closed if a later page failed.

The pattern is the point. Deterministic outputs, complete evidence, clear authority boundaries, and safe failure modes are not nice-to-haves for automated software engineering. They are the difference between an agent you can let run and one you have to babysit. Partial evidence must never be treated as complete evidence.

The Cache Flaw

While examining Bazel's caching architecture, the team turned its attention to the --experimental_repo_contents_cache flag. This feature restores the output of reproducible repository rules from a remote Action Cache and Content Addressable Storage. That improves performance, but it moves the trust boundary: cache-supplied metadata now participates directly in filesystem materialization.

Before the fix, Bazel trusted cache-supplied tree node names without sufficiently enforcing that the resulting path stayed inside the intended external repository directory. A node name containing parent-directory traversal or an absolute path could land outside it. The team did not stop at reading documentation. The Software Factory generated a reproducible, end-to-end test case against a real Bazel 9.0.2 binary. Under the required conditions, a poisoned remote-cache entry caused Bazel to overwrite a pre-existing file outside the repository root, while the build itself completed successfully.

Who Is Actually Exposed

Three conditions must all hold for exposure:

  • --experimental_repo_contents_cache is enabled with a remote cache, appearing in a .bazelrc, CI config, or wrapper script.
  • Someone untrusted can write to the relevant remote Action Cache or CAS.
  • The Bazel binary predates commit c37a6a1bf7f4, meaning bazel --version reports 9.2.0 or another build without the fix.

This is not a drive-by internet vulnerability. But that is exactly why it matters. Once cache metadata can steer filesystem writes, anyone who can write to your remote cache becomes part of your trusted computing base.

What Google Did and What You Should Do

Google's Bazel team fixed the issue in commit c37a6a1bf7f4, titled "Improve path validation in remote repository contents cache." The fix rejects remote tree node names containing separators, parent-directory traversal, or absolute paths; checks that injected file, directory, and symlink paths stay inside the repository directory; resolves symlink targets and verifies they remain contained; and adds a regression test asserting that invalid paths are rejected. The fix landed in Bazel 9.3.0rc2, while Bazel 9.2.0 still ships the pre-fix logic.

Teams using the flag should search every configuration location for its presence, check the exact binary they run (not the one they think they run), and either upgrade to a build containing the fix or disable the flag until a stable release is available. Cache write access should be treated as a security boundary, not a performance setting.

What This Taught About Autonomous SDLC

The experiment produced a clean scorecard: six upstream contributions landed, a security boundary was found and reproduced on a shipping binary, the issue was disclosed privately and responsibly, and Google fixed it upstream. Build acceleration runs on structured graphs, deterministic inputs, reusable outputs, process isolation, scheduling, and defined trust boundaries. Those same primitives form the foundation of automated software engineering. Caching matters because agents repeat work at enormous scale. Distribution matters because once coding gets cheap, build, test, analysis, and validation become the long pole. Isolation matters because many agent loops run concurrently. Observability matters because humans need a receipt, not a promise. And determinism matters because exploration and promotion cannot have the same authority.

Bazel turns source code into an explicit execution graph. The Software Factory extends that idea upward, turning the software development lifecycle itself into a managed graph. After more than two decades of making builds faster, the next step is making the entire engineering loop faster, from intent to tested, reviewable, evidence-backed software, without giving up the controls that make software trustworthy.