Adding Read Replicas Can Slow Your Writes: Shrink What Needs Agreement

On October 6, GitHub published an engineering post about rebuilding the storage layer behind Git while the service stays online. The headline number is a claimed gain of up to 35x in write throughput. The more useful sentence for most teams is a plain description of a trade-off the old design made: because the on-disk copies were the source of truth, every replica took part in every write, so a push was only as fast as the slowest replica, and adding read replicas made writes slower.

Many systems have that shape without anyone having decided it. Replicas get added to absorb read load, and the write path quietly inherits their tail latency and their failures. The pattern is general, the fix is an architectural decision about how much of an operation really needs agreement, and the answer is rarely “all of it.” This article uses GitHub’s account as the case, checks it against independent sources, and turns it into a short review you can run on your own storage.

What GitHub says the old design did

The GitHub post describes a system called Spokes that keeps a full copy of each repository on the local disks of several fileservers, five by default. Local disks give Git operations low-latency access to native repository data, and the extra copies add redundancy and spread read load. When a push updates a reference, a three-phase commit protocol with a quorum keeps the web UI, CI and API clients looking at a consistent repository.

The cost is spelled out. Since the disk copies are the source of truth, all replicas participate in each write. Adding replicas for reads therefore slows writes, losing a replica reduces read capacity, and losing quorum halts writes entirely. The post ties this to load: Git activity grew from 218.2 billion events a month in September 2025 to 473.3 billion in August 2026, and the author lists five pressure points, namely per-agent commit latency, write throughput, merge contention on a single ref, read fan-out from CI and scanners, and maintenance cost growing with write volume.

Two things in that account deserve a careful reading. First, the failure is not that a quorum exists. A majority quorum alone does not make a write wait for the slowest copy. The coupling comes from requiring every replica to participate, combined with a source of truth that lives on the same machines that serve reads. Second, the five pressure points are properties of any busy shared repository, not of one kind of client. Agents are the load that exposed the limit, and the argument holds if you replace them with a larger engineering organization, a monorepo or a heavy CI estate.

Why this is arithmetic, not bad luck

The coupling is easy to quantify, which is the reason to take it seriously. The following is an illustrative scenario with assumed numbers, not a measurement from GitHub or from any client. Suppose each replica independently takes longer than 100 milliseconds to acknowledge 1 percent of the time. A write that needs all five replicas is slow whenever any one of them is, which happens about 4.9 percent of the time. With nine replicas required, it is about 8.6 percent. Add capacity for reads and the slow-write rate rises on its own.

Now require only three of five acknowledgments. The write is slow only when at least three replicas are slow at once, which under the same independence assumption is roughly 0.001 percent. The two designs hold the same number of copies and differ by several orders of magnitude in how often a user feels a straggler. Real replicas are not independent, since they share racks, networks and noisy neighbors, so treat these figures as the shape of the effect rather than a forecast.

The Aurora design is the standard published example of the second approach. AWS’s description of its quorum design, written by one of the people who designed the service, says Aurora keeps six copies across three Availability Zones, writes to all six and treats a write as done when four respond, with a read quorum of three. It states the rules behind those numbers: the write quorum must exceed half the copies so two writes to the same item cannot both succeed independently, and read plus write quorums must exceed the copy count so any read overlaps the latest write. Overlap is a necessary property, not the whole protocol; ordering and concurrency control need further machinery on top. The same post lists slow quorum reads as a drawback to be handled elsewhere. The numbers are a trade, not a free win: a write acknowledged after four of six leaves two copies catching up, and a read that needs its quorum is the slow case.

Moving the agreement, not removing it

The new GitHub design does not abolish coordination. It narrows it. According to the post, only the reference update needs agreement. Storing objects, validating connectivity and scanning for secrets move off the critical path and run in parallel, which shortens the time to acknowledge a push. Compaction and garbage collection run on separate workers against durable storage rather than on the nodes serving traffic. Authoritative data lives in Azure Blob Storage, and lightweight compute workers cache data to serve reads, so losing a worker looks like a cache miss rather than a loss of capacity.

The Register’s coverage, published on October 9, describes the same split in its own words: each commit is written once to object storage that handles replication itself, reads go through a separate path, and reads and writes coordinate only through the branch pointers. It also adds the caveats the vendor post does not dwell on. The up-to-35x figure comes from internal tests and no benchmark methodology is given, no migration timeline was offered, and the trade-offs of the new design, such as consistency behavior and the new dependency on object storage, are not examined. The GitHub post itself does not describe the benchmark conditions, and it says a later post will cover the future architecture and the path to it.

That last point matters for how much weight to give the result. This is a design intent plus a promising internal number, not evidence of production behavior. Read it as a well-explained diagnosis and a plausible direction.

There is independent evidence that the one remaining point of agreement, the reference update, deserves its own engineering. The Git project’s reftable documentation explains that the older packed-refs approach rewrites the whole file for an atomic update, giving an example of 62 MB read and written for a change touching two references. Reftable aims for update cost proportional to the size of the change, and in the benchmark the document reports, a cold single-reference lookup among 866 thousand references took about 34 microseconds against about 410 milliseconds. The same document notes a single lock file that makes writers single-threaded. The lesson is not that reftable is GitHub’s answer; I found no statement in the post tying its design to that format. The lesson is that once agreement is narrowed to one small thing, the cost of that one thing becomes the whole story, and it is worth making proportional to the size of the change.

The Aurora paper makes a related point from the opposite side. Its authors argue that the network, not compute or storage, is the limit for high-throughput data processing, and they respond by pushing redo processing into the storage tier so less traffic crosses the network at all. Different system, same move: stop making the expensive resource do work that the cheap one can do, and keep the shared agreement as small as correctness allows.

Three designs for the same write

The comparison below is my synthesis of the three shapes discussed so far. It describes the structure, not the performance of any product.

Design	A write waits for	What adding read capacity does	Where it fits
Every replica participates	All copies, so the slowest one sets latency	Adds a straggler to every write and lowers write availability	Few replicas, modest write rates, strict locality requirements
Majority quorum	Enough copies to overlap any later read, typically more than half	Depends on whether new replicas join the voting set; if they do not, write latency is largely unchanged	Moderate scale where durability and simplicity matter most
Narrow agreement over durable shared storage	A small ordering decision, with bulk data written once to storage that replicates itself	Read capacity scales separately, since workers are caches	High write rates, bursty reads, expensive maintenance

The third row is not strictly better. It buys independence between reads and writes by placing a durable storage service in the critical path and by making cold reads depend on that service. It also demands a clear ordering point, and a single ref or log head is also a single contention point, which is exactly what the merge contention item in GitHub’s list describes. Choose it when the coupling is what hurts, not because it is the newest shape.

A review you can run this quarter

The practical value of this case is a set of questions that expose accidental coupling before it becomes an incident.

List what a write waits for. For your most important write path, name every component that must respond before the caller gets an acknowledgment. Include verification steps such as validation or scanning, not just storage.
Separate must-agree from must-happen. Most of that list has to happen eventually, not before acknowledgment. Identify the minimum that defines the commit point, and ask what a user would observe if the rest completed a few hundred milliseconds later.
Plot write latency against replica count. If you have changed the number of replicas in the past, compare write tail latency before and after. A rising tail with more replicas is the signature of full participation.
Check where maintenance runs. Compaction, vacuuming, reindexing and garbage collection that share nodes and I/O with serving traffic will compete with writes as volume grows.
Ask what a lost replica costs. If losing one copy reduces read capacity or threatens write availability, the replicas are being used both as redundancy and as capacity, and those are different jobs.
Price the alternative honestly. Moving the source of truth to a storage service changes your failure domains, your cold-read latency and your cost model. Write down what you give up before you commit.

When not to redesign

A narrower design is justified by evidence of coupling, not by the existence of a good example. If your write latency is flat as replicas change, your maintenance does not collide with traffic, and your growth is predictable, then the simple replicated setup is the right one and the redesign is cost without benefit. A rewrite also competes with everything else on the roadmap, and live migrations of the primary data path are among the riskiest projects a platform team can run, which is part of why the missing migration detail in GitHub’s first post is the interesting gap to watch.

There are cheaper steps to try first. Move non-essential verification off the acknowledgment path. Cap or reschedule maintenance. Relax full participation to a majority where your durability requirements allow it, and state what you are relaxing. Measure again after each step, because the aim is to learn which coupling you actually have.

If your write path has grown more dependencies than anyone intended, we help engineering teams map what a write actually waits for, measure how replica count and maintenance affect latency, and weigh a narrower design against cheaper fixes before any migration is committed. Talk with us about your storage and replication architecture.