Four Ways to Survive a Network Split
Consensus protocols solve the problem of multiple machines agreeing on a single view of history. By exploring Paxos, Viewstamped Replication, Zab, and Raft, developers can choose the right tool for their distributed systems and avoid hidden complexities. Understanding each protocol’s mental model c…
When a network split occurs, two nodes may disagree on who is the leader, yet both appear healthy. The core question is: which version of reality becomes the truth? Consensus algorithms answer this by forcing all replicas to agree on a single, ordered history. Though most engineers never write a consensus protocol, they rely on systems built on these ideas—databases, service‑discovery tools, and distributed locks. Understanding the mental models behind Paxos, Viewstamped Replication, Zab, and Raft turns opaque product limitations into predictable design choices.
What Consensus Means for Everyday Systems
A replicated log is the backbone of most fault‑tolerant services. Each log entry is an ordered command that must be applied in the same sequence on every replica. A quorum—typically a majority—must accept an entry before it is considered committed. This guarantees that any two quorums overlap, preventing two separate groups from committing conflicting histories after a network partition. The price of this safety is that if no majority can communicate, the cluster may stop accepting writes. That shutdown is intentional: it avoids creating divergent truths.
All four protocols discussed focus on crash‑tolerant failures, not Byzantine faults. They illustrate different ways to structure the same safety goal: stable leadership, efficient log replication, and safe recovery after failures.
Multi‑Paxos: Optimizing Repeated Decisions
Basic Paxos reaches agreement on a single value by having proposers ask acceptors to promise not to accept older proposals, then to accept a value under a numbered ballot. Repeating this for every log entry is costly. Multi‑Paxos introduces a stable leader that can drive many entries without re‑entering the prepare phase each time. The leader proposes the next command, a quorum accepts it, and the entry becomes committed. When the leader fails, the system falls back to the full Paxos protocol to establish a new ballot and reconcile any already‑chosen values.
Understanding Multi‑Paxos helps when evaluating a “Paxos‑based” product. Ask how leadership is managed, how log gaps are repaired, how membership changes are handled, and what operators can observe during recovery. Two products claiming Paxos may behave very differently because the safety core can be wrapped in many optimizations.
Viewstamped Replication: Primary‑Backup with State Transfer
VR starts from a primary‑backup model. Replicas operate in numbered views, and one replica is the primary for each view. The primary assigns operation numbers, sends prepares to backups, and commits once enough replicas acknowledge. If the primary appears to fail, replicas move to a higher view, exchange log information, and a new primary is elected deterministically from the view number and membership. The new primary must reconstruct a history that preserves all committed operations before it can serve writes.
VR’s key insight is that leader change is a state‑transfer protocol, not just an election. When evaluating a system that uses VR terminology—views, operation numbers, commit numbers—ask how state is transferred before a replica becomes primary, whether an out‑of‑date replica can be promoted, and how duplicate operations are detected.
Zab: Ordered Broadcast for Coordination Services
Zab was designed for ZooKeeper, a coordination service that requires a totally ordered stream of state changes. It separates the protocol into recovery and broadcast phases. In recovery, a leader is established and replicas synchronize on a valid history. In broadcast, the leader proposes transactions, followers acknowledge them, and committed transactions are delivered in order. Each transaction carries an zxid, a monotonically increasing identifier that includes an epoch and a counter.
Because ZooKeeper’s data model is a hierarchical namespace with coordination semantics, Zab emphasizes ordered broadcast over isolated agreement. When choosing a coordination service, consider whether the ordering contract matches your data’s semantics rather than the algorithm’s name.
Raft: Understandability as a Design Goal
Raft decomposes replicated‑log consensus into leader election, log replication, and safety. Terms act as logical eras of leadership; each log entry records the term it was created in. During an election, a voter rejects a candidate whose log is less up‑to‑date. A new leader repairs followers by replacing conflicting uncommitted suffixes. Raft’s commit rule ensures that committed entries cannot be lost, while uncommitted entries may disappear, explaining why a client timeout does not guarantee commitment.
Raft’s greatest contribution is its clarity. Engineers can explain, review, and debug the protocol more easily, reducing implementation errors. Products like etcd, Consul, and CockroachDB build on Raft, but their surrounding systems—data models, read paths, snapshots, and tooling—differ significantly.
Applying the Knowledge Without Writing Code
When selecting a distributed database or coordination service, ask:
- What is the unit of consensus? One log per cluster, per shard, or a metadata group?
- Which operations pass through consensus? Are reads linearizable, lease‑based, or locally served?
- What happens without a quorum? Does the system stop writes, serve stale reads, or expose a tunable consistency mode?
- How are leader changes handled? Is state transferred safely before a new leader starts accepting writes?
- What recovery mechanisms are built into the protocol and the product?
Answering these questions turns the abstract algorithm into concrete operational insights, helping you avoid hidden complexity and choose the right tool for your workload.
Why it matters
Consensus protocols are the foundation of any fault‑tolerant distributed system. By grasping their core ideas, engineers can make informed choices, anticipate failure modes, and avoid costly surprises when systems behave differently than expected.
Key points
- Consensus ensures all replicas agree on a single ordered history, preventing divergent truths.
- Multi‑Paxos uses a stable leader to reduce overhead while retaining Paxos safety.
- Viewstamped Replication treats leader change as a state‑transfer protocol, crucial for safe failover.
- Zab tailors ordered broadcast to ZooKeeper’s coordination semantics, not just isolated values.
- Raft prioritizes understandability, making it easier to implement correctly and debug.
- Choosing a system requires asking about consensus units, read/write paths, quorum behavior, and recovery mechanisms.
Frequently asked questions
What is a quorum in a replicated log system?
A quorum is a majority of replicas that must agree on a log entry before it is considered committed. It guarantees overlap between any two quorums, preventing conflicting histories after a partition.
Why does a system stop accepting writes after a network split?
If no majority can communicate, the system refuses to commit new entries to avoid creating divergent truths. This is a safety feature, not a bug.
Can a Paxos‑based product be faster than Raft?
Speed depends on implementation details, batching, and optimizations. Paxos offers a general safety core that can be wrapped in many efficient designs, but Raft’s clarity often leads to more reliable performance.
