ZooKeeper ensemble providing ZNodes watches and quorum for distributed coordination
Technologies

Apache ZooKeeper for system design interviews

A clear roadmap: why coordination is hard, ZNodes + watches + ensemble, the four patterns you build, ZAB in brief, and when to pick ZooKeeper vs etcd/Consul/KRaft.

Engineering manager 13 min read ·

Roadmap

ZooKeeper is a coordination service — a tiny, strongly consistent metadata store with push notifications. It is not a database for app data. Released in 2008, it still powers parts of the Apache stack; Kafka moved to KRaft. The patterns matter even if you use etcd or Consul instead.

01Store

Small ZNodes (<1MB).

02Notify

One-shot watches.

03Survive

3/5/7 quorum.

04Alt

etcd · Consul · cloud.

1 · The problem: multi-server chat

One chat server is easy — Alice and Bob share memory. Scale to two servers and Alice (Server 1) must find where Bob lives.

Alice and Bob connected to a single chat server.
Single server — no coordination needed.
Alice on Server 1, Bob on Server 2, no shared routing.
Two servers — Server 1 doesn't know where Bob is.

Common attempts and why they hurt:

  • Central DB — works until it's a SPOF, adds latency per message, and melts under lookup load.
  • Local caches — Bob moves servers; stale cache drops messages.
  • Server broadcast — O(n²) connections; noisy at dozens of servers.
  • Heartbeats only — network partitions create split-brain membership.
Chat servers querying a central database for user locations.
Database registry — SPOF + hot path.
Chat servers broadcasting presence to each other.
Peer broadcast — doesn't scale with n.

ZooKeeper's answer: one shared source of truth. Servers register; mappings update on connect; watches push changes; ephemeral nodes vanish on crash. No mesh, no DIY consensus.

Chat servers coordinated through ZooKeeper.
ZooKeeper — coordination hub, not message payload store.

2 · Three primitives

Everything in ZooKeeper is built from three ideas. Master these and the patterns in the next section click immediately.

A · ZNodes (the data model)

A hierarchical tree like a filesystem. Each ZNode has a path, a small payload, and metadata. Store coordination data — thousands of small nodes, not large blobs.

  • Persistent — until you delete it (config, feature flags).
  • Ephemeral — deleted when the client session ends (liveness, locks, presence).
  • Sequential — auto suffix …-0000000001 (ordering, election, lock queues). Combine with -e.
chat-app tree with servers users and config ZNodes.
Example tree — ephemeral servers/users, persistent config.
create /chat-app/config/max_message_size "1024"          # persistent
create -e /chat-app/servers/server2 "192.168.1.102:8080"  # ephemeral
create -s -e /chat-app/leader/node- "server1"             # sequential ephemeral

B · Ensemble (high availability)

Run 3, 5, or 7 servers (odd for majority). One leader handles writes; followers serve reads and vote. Survive as long as a quorum (majority) is up — 3 nodes tolerate 1 failure; 5 tolerate 2.

Chat servers connected to a three-node ZooKeeper ensemble.
Clients talk to the ensemble — not to a single ZK host.
ZooKeeper zk = new ZooKeeper(
    "zk1:2181,zk2:2181,zk3:2181", 3000, watcher);

C · Watches (push, don't poll)

Ask to be notified when a ZNode changes. Watches are one-shot — re-register after each event. Pattern: keep a local cache, invalidate on watch. ZooKeeper notifies; it isn't meant for high-QPS reads on every message.

Watch notification from ensemble to chat server local cache.
Watch fires → update local cache → no n² mesh.
zk.getChildren("/chat-app/users", true, null);  // set watch

public void process(WatchedEvent e) {
  if (e.getType() == EventType.NodeDataChanged) {
    byte[] data = zk.getData(e.getPath(), true, null); // re-watch
    routingTable.put(/* user */, new String(data));
  }
}

3 · Four patterns you build

Same three primitives → four interview-ready recipes.

Configuration

Persistent ZNodes + watches. Flip enable_reactions once; every watcher updates without restart. Use for dynamic flags — static deploy config stays in env/files or Parameter Store.

Service discovery

Ephemeral registration: create -e /services/api/instance1 "10.0.0.1:8080". Consumers list children + watch. Same idea as Consul, etcd, or Kubernetes Services.

Leader election

Each candidate creates a sequential ephemeral node. Lowest sequence = leader. Others watch their predecessor. Leader dies → node gone → next steps up. (Used historically by Kafka's controller and HBase.)

Sequential ephemeral nodes with lowest as leader.
App leader = lowest seq. ZK's internal leader election is a different algorithm.
create -s -e /leader/node- "server1"  # → node-0000000001  ← leader
create -s -e /leader/node- "server2"  # → node-0000000002  watches 0001

Distributed locks

Same sequential-ephemeral queue: lowest holds the lock; waiters watch the previous node. Good for long-lived / correctness-critical locks. Bad for hundreds of acquires/sec — use Redis for ticketing-style holds.

4 · Under the hood (skim OK)

ZAB

ZooKeeper Atomic Broadcast keeps the ensemble in sync. Writes go to the leader → broadcast to followers → commit after majority ACK. Same family as Raft/Paxos.

Leader broadcasting write to four followers with quorum commit.
Write path — respond only after quorum commit.
  • Reads — any follower, from memory (~10:1 read:write sweet spot). May be briefly stale; call sync() if you need latest.
  • Writes — leader + quorum; more expensive.
  • Durability — write-ahead transaction log + periodic snapshots. Dedicated log disk matters.
  • Sessions — heartbeat + timeout (often 10–30s). Expire → delete all that client's ephemeral nodes + watches.
  • Partitions — minority side stops writes (no split-brain).

5 · Today: alternatives & when to use

Still used where?

HBase, Hadoop, SolrCloud, Pulsar, ClickHouse replication. Kafka historically needed ZK; KRaft removes that dependency — mention both.

  • etcd — powers Kubernetes; modern APIs; same CP KV role.
  • Consul — discovery + health + KV.
  • Cloud — Parameter Store, Cloud Map, managed MSK — prefer when already on that cloud.

Limits

Watch hot-spots on popular nodes, expensive writes, entire dataset in RAM, non-trivial ops (JVM, txn-log disk, session tuning).

When ZK earns a box

  • Infrastructure design — "design a message queue": broker registration, partition leaders, consumer groups (pre-KRaft Kafka pattern).
  • Smart routing (Staff+) — collocate live-room users; gateway asks ZK which server owns the room.
  • Hierarchical / long-lived locks — when Redis TTL semantics aren't enough.
ZooKeeper above broker cluster with producers and consumers.
Message queue — ZK coordinates; brokers carry the data.
API gateway querying ZooKeeper for chat server placement.
Smart routing — gateway asks ZK where to pin the SSE connection.

When not to

  • Typical product API on K8s — use Services / cloud discovery.
  • High-frequency locks — Redis.
  • Large blobs or write-heavy workloads.

Cost and performance levers

Interview Q&A

Takeaway

Define the coordination need first. ZooKeeper (or etcd) is how you share tiny consistent state — not how you scale the app tier.

Wrapping up

Roadmap recall: problem (shared truth) → primitives (ZNode / ensemble / watch) → patterns (config, discovery, election, locks) → ZAB/sessions if asked → prefer modern alternatives unless infrastructure design needs the classic box.

Next: Kafka (KRaft), Kubernetes (etcd), Redis locks, Key technologies.

technologieszookeepercoordinationconsensusdistributed-systemsleader-election

← Lattice