You Don't Adopt a Build System. You Inherit Its Operations.

The build broke on Boxing Day, and nobody on the team had touched the code. At 07:35 on the morning of 26 December 2025, an SSL certificate for a set of *.bazel.build domains expired, and continuous integration pipelines around the world started failing on a validation error most engineers had never seen. The certificate was not on anyone’s infrastructure. The domain was not one anyone’s team controlled. The renewal had been quietly failing for weeks, because a subdomain the certificate happened to cover had been removed and every automated attempt to renew it silently failed with no alert behind it. According to the Bazel team’s own postmortem, it took roughly eleven and a half hours from the first report to a restored certificate, on the day after a major holiday, with an on-call rota that was, understandably, mostly off.

Read the incident narrowly and it is a cautionary tale about certificate monitoring. Read it the way the affected teams experienced it and it is something more useful: a bill, arriving unannounced, for a decision made months or years earlier. The moment they committed their first BUILD file, they took on a remote registry, a content-addressed cache, and a set of domains as load-bearing parts of their own production supply chain. They just never saw the invoice until it came due at dawn on the twenty-sixth.

The cost you budget for is the wrong cost

Every proposal to adopt a heavyweight build system prices the same thing: the learning curve. The BUILD files nobody wants to write, the migration from the old system, the weeks of the team getting fluent in a new mental model, the inevitable bzlmod refactor. That cost is real, it is large, and it is almost entirely beside the point, because it is temporary. You pay it once, it declines as the team learns, and eventually it approaches zero.

The cost that actually decides whether the adoption survives is the one no proposal itemizes. A foundational tool has two separable properties. The first is its leverage: what it does for you, which for a modern build system means incremental builds, content-addressed caching, and the ability to prove a change cannot affect a target and therefore skip its tests entirely. The second is its operational surface: the set of systems you must now run, staff, and keep alive because the tool depends on them. Remote cache. Remote execution. A registry and the mirrors in front of it. Lockfiles that have to stay current. Download caches that have to be populated. The BUILD graph itself, which is now a piece of software your team maintains forever.

Leverage is what gets demoed. Operational surface is what gets inherited. The learning curve is paid down over a quarter; the surface is yours for as long as you run the tool, and it does not shrink as the team gets more comfortable. If anything it grows, because every capability you switch on, remote execution, a shared cache, a private registry, is another system with its own failure modes, its own certificates, and its own 07:35-on-a-holiday.

The leverage a build system gives you is bought once. The operational surface it hands you is rented for as long as you run it, and the rent is paid in on-call.

The surface is invisible until it pages you

The reason operational surface gets mispriced is that it is invisible in the state everyone evaluates the tool in: the happy path, on a working day, with the registry up. You run the benchmark, the incremental build is dramatically faster, the affected-target selection skips ninety percent of the test suite, and the decision looks obvious. What you cannot see in that moment is the dependency you just acquired on someone else’s certificate rotation, someone else’s DNS hygiene, someone else’s release cadence.

The Boxing Day outage is the clean illustration because the failure was so far from the code. The Bazel team’s postmortem is candid about the mechanism: the auto-renewal for the certificate had been failing repeatedly for about a month before it expired, and none of those failures raised an alarm. The single most telling part of the writeup is not the timeline, though. It is the remediation. The advice to users is to populate the download cache, keep the lockfile current in CI, and host internal mirrors for release binaries, registry metadata, and source archives. Translated out of postmortem language, that is: take the operational surface you thought the tool was carrying for you, and carry it yourself.

That is the anti-pattern to name and watch for. Call it the invisible dependency: a component your build now relies on that never appeared in the adoption proposal, has no owner on your side, and surfaces only when it breaks. The symptom is a red pipeline whose root cause is a system nobody on the team can point to on an architecture diagram. Every foundational tool ships with a few of these. The question at adoption time is not whether they exist. It is whether you have decided, in advance, to own them.

Teams do reverse these calls, and it takes them years

If the operational surface were cheap, you would expect adoptions to stick. They do not always. The most careful public measurement of this is a 2024 study by Mahmoud Alfadel and Shane McIntosh that looked specifically at downgrades, teams abandoning a modern build technology to go back to a more traditional one, a direction the literature had largely ignored.

The percentage matters less than the duration. A median of 638 days means these were not teams that tried the tool for a sprint and bounced. They committed, migrated, learned it, ran on it for the better part of two years, and then decided the trade was not worth it and paid to migrate off. The study’s thematic analysis of why points at exactly the surface: technical challenges, lack of platform integration, and team coordination overhead, the running costs, not the setup costs.

This is where the second anti-pattern lives. The sunk-cost stall is what keeps a team on a foundational tool long after the surface has outgrown the leverage, because reversing it means admitting that a large, visible, multi-quarter investment was a misread. The symptom is a build system that a shrinking number of people understand, defended with the years already poured into it rather than the value it returns this quarter. The 638-day median is the sound of that stall resolving, slowly and expensively.

Nobody spends 638 days on a build system they can walk away from lightly. The teams that reversed the call did it because the running cost, not the setup cost, finally exceeded the return.

There is a related failure that seeds the whole cycle, and it is worth saying plainly because it is common and slightly embarrassing. Resume-driven adoption is choosing the tool that the largest engineering organizations use, on the reasoning that it “scales better,” when your constraint is not scale. The symptom is a justification that references a company two orders of magnitude larger than yours. “It scales better” is the wrong axis for the decision entirely. A build system’s job is to remove a bottleneck you actually have. If affected-target selection saves a team of four hundred engineers an hour per change, it is transformative; for a team of twelve with a three-minute test suite, it is a large operational surface bolted on to solve a problem they will not have for years, if ever.

A test you can run before the first BUILD file

The decision is not “is this tool good” (it is, at the scale it was built for) but “does the leverage I need today clear the surface I would have to staff.” That resolves into three gates, and they are worth walking in order, because failing any one of them is a reason to wait.

flowchart TD
  A{"Affected-target selection<br/>is your real bottleneck<br/>today?"} -->|No| B["Stay on the lighter tool.<br/>Speed you don't need<br/>is negative leverage."]
  A -->|Yes| C{"A lighter tool can hit it<br/>at your current scale?<br/>(Turborepo, Nx, Go build,<br/>cached Make)"}
  C -->|Yes| D["Use it. Revisit at the<br/>next real ceiling."]
  C -->|No| E{"You can staff the operational<br/>surface with a named owner?<br/>(registry, cache, mirrors,<br/>lockfiles)"}
  E -->|No| F["Not yet. The surface<br/>outlasts the enthusiasm."]
  E -->|Yes| G["Adopt. Treat the surface<br/>as production<br/>from day one."]

The first gate is the honest one, because it is where resume-driven adoption dies. Affected-target selection, the ability to not run a test because the tool can prove the change did not touch it, is the headline capability of a scale build system, and it only pays when the thing it removes, a long build or a slow test suite spread across many independent parts, is genuinely hurting you now. Not next year, now.

The second gate is where most teams should stop. The market between “make with a shell script” and “full remote-execution build graph” is crowded with tools, Turborepo, Nx, and language-native builders among them, that deliver caching and affected-target selection with a fraction of the surface. They have real ceilings, and a genuinely large polyglot monorepo will hit them. But a tool that solves your bottleneck with less to run is not a compromise; it is the correct answer until you can prove you have outgrown it.

The third gate is the one the Boxing Day outage is really about. If you clear the first two, you do need the heavyweight tool, and then the only question left is whether you will own its surface deliberately or discover it during an incident. That means a named owner, not a rotating side task, for the registry, the cache, the mirrors, and the lockfiles, and treating each of them as a production system with monitoring on its certificates. The teams whose builds died on the twenty-sixth were not wrong to run Bazel. Some of them were exactly the scale it is built for. The ones who recovered in minutes rather than hours were the ones who had already internalized the surface: their own mirror, their own cache, their lockfile pinned, so an upstream certificate expiry was an inconvenience rather than an outage.

The constraints nobody puts on the slide

Three frictions decide whether this holds in practice, and all three tend to be missing from the adoption deck. The first is headcount. “A named owner for the operational surface” sounds like a formality until you try to name the person, and find that they already own three other things and the build system will be the one that gets attention only when it is on fire. A surface with no owner is not a surface you are running; it is an incident you have not had yet.

The second is that the lighter tools are not a free lunch either. They have ceilings, and a team that refuses to ever adopt the heavyweight option out of principle will eventually spend more engineering time working around a tool that has run out of room than it would have spent owning the surface of the right one. The gate is symmetric. Staying too long on the small tool is the same mistake as reaching too early for the big one, made in the other direction.

The third is political, and it is why the 638-day median is so long. Reversing a foundational tool decision is not a technical act; it is a public admission that a large, sponsored, cross-team investment was aimed at the wrong problem. The engineers usually see it first and the org resists it longest, and the gap between those two is measured in quarters. The cheapest place to avoid that cost is at the first gate, before the first BUILD file, when the decision is still reversible for the price of a conversation instead of a migration.

A build system benchmark measures leverage, and leverage is the part that demos well and gets cheaper as the team learns. Production measures operational surface, and the surface is the part that never appears in the proposal, does not shrink with familiarity, and eventually pages someone at dawn on a holiday over a certificate they did not know they owned. The teams who choose foundational tooling well are not the ones who picked what the largest engineering organizations run. They are the ones who priced the surface before they signed for it, and reached for the heavyweight tool only when the bottleneck was real and they had already decided who would carry what it left behind.

If you are weighing a build system, a service mesh, or any other foundational tool right now, the sharper question is not which option is most capable but which operational surface you are prepared to own. We help teams draw that line before the migration: sizing the real bottleneck, choosing the lightest tool that clears it, and staffing the surface where the heavyweight option genuinely fits. If that is your decision, we are glad to compare notes.