ZooKeeper

CSCE 678

Weak vs Strong Consistency

• In distributed systems, consistency is often the target of weakening

• Examples of weak consistency:• NoSQL servers – Dynamo• Datanodes in HDFS

• But sometimes, strong consistency is needed

2

Use Cases of Strong Consistency

• Configuration management

• Message Queues

• Group membership

• Synchronization:• Mutexes and read/write locks• Barriers: process joins

3

(Will talk about them one by one)

Use Case: Configuration & Group Membership

4

Primary[Configuration]Primary: …Secondaries: …Port IDs: …Created by: …Push

Use Case: Message Queues

5

ProducerA

ProducerB

Consumer

enqueue

enqueue

dequeue

Use Case: Synchronization

• Mutexes

6

• Read/write locks

lock();

last_x = read(x);write(y, last_x);write(x, last_x + 1);

unlock();

rd_lock();

if (queue.size > 0) {wr_lock();x = queue.dequeue();wr_unlock();

}

rd_unlock();

Shared among readers

Exclusive for one writer

How to Define Strong Consistency?

• Linearizability• Also called atomic consistency or immediate consistency• As soon as a write operation finishes, the whole system

should see the latest data.

7

Serializability vs Linearizability

• For distributed systems, these two words are used in very specific contexts

• Serializability: for databases• Equivalent to Serializable Isolation (I) in ACID• No ordering for concurrent transactions

• Linearizability: for strongly-consistent reads/writes• In respect to operations, not transactions• Considering the global ordering of reads/writes

8

Serializability vs Linearizability

• Serializability

9

Client A

Client B

read(x), write(y), read(y)

read(y), write(x), write(y) read(y), read(x), write(y)

System read(x), write(y), read(y) read(y), write(x), write(y) read(y), read(x), write(y)

Transactions

No constraint of ordering between concurrent transactions.

(from A) (from B) (from B)

time

Serializability vs Linearizability• Actually, serializability can be implemented by linearizability

(i.e., a lock service)

10

Server 1

Server 2

read(x), write(y), read(y)

lock(x)

lock(y)

read (shared) lock

write (exclusive) lock

read(y), write(x), write(y)

blocked

ok ok

ok ok

Two-phase lock (2PL) Irrelevant to two-phase commit (2PC)

Serializability vs Linearizability

• LinearizabilityAs soon as a write operation finishes, the whole system should see the latest data.

11

Client A write(x, 1) => oktime

Server 1

Server 2insert

Client B read(x) => 1

More on Linearizability

• Reads concurrent with a write

12

Client Atime

Client B

read(x) => 0

Client C write(x, 1) => ok

read(x) => 1

read(x) => 1

Any read that begins before the writemust see the old value.

Any read that begins AFTER the write may see old or new value; but as soon as one client see the new value, all other clients should too.

More on Linearizability

• Write concurrent with another write

13

Client Atime

Client B

Client C

write(x, 1) => ok

read(x) => 1

Any write begins before another writemust be committed first.

write(x, 2) => ok

read(x) => 2

Linearizability vs Causality• Linearizability implies causal consistency

14

(1) Causality is based on happens-before relationshipin a client

If client sets x = 1 and sets y = 0, then the two values have causal relationship.

Client Atime

Client B

write(y, 0)write(x, 1)

read(y) => 0

Causality

read(x) => 0

Violate causal relationship!

read(x) => 0

15

Linearizability vs Causality• Linearizability implies causal consistency

(2) Linearizability implies total ordering of each object, soautomatically preserves happens-before.

A linearizable system doesn’t have to do anything to preserve causal consistency.

Client Atime

Client B

write(y, 0)write(x, 1)

read(y) => 0 read(x) => 1

read(x) => 0

Total order

How to Implement Linearizability?

• Read/write from single leader• Failover to a replica may lose linearizability

• Consensus algorithms• Using two-phase commits (2PC) or Paxos• Prevent stale replicas

• Most likely unlinearizable: multi-leader replication

16

Q: How does linearizability apply to CAP Theorem?

Linearizability in CAP Theorem

• Linearizability = Strong C• Read/write through single leader: lose A & P• Read through replicas, write through leader: lose P

• With consensus, linearizability can be:• Partition tolerant when ½ of replicas are connected• Fair availability with wait-free operations and fast,

lossless leader recovery

17

Apache ZooKeeper

• A coordination service for all use cases of strong consistency in a distributed system

• Wait-free: any operation will not block on other slow or failed clients

• ZooKeeper has no API for locking, but can be used to implement any locking mechanism

18

System Overview

19

ZooKeeper Service

ServerServerServer Server Server

Follower LeaderFollower Follower Follower

Client Client Client Client Client Client Client

Forward operations

Namespace

• ZooKeeper uses a filesystem-like namespace

20

/

/App1 /App2

/App1/p_1 /App1/p_2 /App1/p_3

…

znodes

Data Data Data Not large data files

(more like metadata)

Data

Namespace

• ZooKeeper has two types of znodes (paths)• Permanent (regular): clients explicitly create and delete

the znodes• Ephemeral: clients create the znodes, and either delete

them explicitly or let the system automatically deletes them when client sessions timeout.

21

API• create(path, data, flags)

• flags: regular/ephemeral, sequential (appending a seqnum)

• getData(path, watch) -> (data, version)

• setData(path, data, version)

• delete(path, version)

• exists(path, watch) -> true/false

• getChildren(path, watch) -> [paths]

• sync(path) path is ignored now

22

Asynchronous Operations• Operations can be synchronous or asynchronous

• Client can queue up multiple asynchronous operations

• Server responds by invoking callbacks

23

ZooKeeper Se

ServerServerServer

Follower LeaderFollower

Forward operati

Client

getData(x) create(y) …

callback for getData(x) callback for create(y) …

Event Notification

• All operations except sync are wait-free

• No locking API but clients can implement locks using watch events

24

1 l = “/my-lock”;2 if exists(l, watch=true) then wait for watch event;3 n = create(l, EPHEMERAL);4 if n is error then goto 2;

1 delete(l);

Lock (very naïve version)

Unlock

Server will push events to clientwhen the path is updated

A(asynchronous)-Linearizability

• Local order: all operations from the same client are processed FIFO (first-in-first-out)

• Global order: Linearizable writes

25

ServerClient A

ServerClient B

write Op 1, write Op 2, …

write Op 1, write Op 2, …

Some consensus to ensurethe write ops are appliedexactly as their global order.

A(asynchronous)-Linearizability• Reads are not linearized (may not see latest state)

• Read directly from server (identical replicas)• Other servers may have pending writes in queues• Solution: sync after writes

26

ServerClient A

ServerClient B

write Op 1, write Op 2, sync

read Op 1, read Op 2, …

After sync, all clients willsee the changes of priorwrites from client A.

Zab (Atomic Broadcast)

• All servers forward messages to a single leader• Important role as a sequencer• Leader can change if partitioned or failed

• The leader broadcasts (proposes) the messages to be delivered to all followers

27

ZooKeeper Service

ServerServerServer Server Server

Follower LeaderFollower Follower Follower

Forward operations

Zab (Atomic Broadcast)

• Two-phase commit (2PC)

28

Leader

Request, e.g., write(x, 1)Step 1. assign a monotonicallyincreasing id (zxid) to the request

Follower Follower

Zab (Atomic Broadcast)

• Two-phase commit (2PC)

29

Leader

Request, e.g., write(x, 1)Step 1. assign a monotonicallyincreasing id (zxid) to the request

Follower Follower

Step 2. Propose the changewith zxid to followers

Zab (Atomic Broadcast)

• Two-phase commit (2PC)

30

Leader

Request, e.g., write(x, 1)Step 1. assign a monotonicallyincreasing id (zxid) to the request

Follower Follower

Step 2. Propose the changewith zxid to followers

Step 3. Followersacknowledge theproposal.

Q: In what situation maythe follower reject the proposal?

Zab (Atomic Broadcast)

• Two-phase commit (2PC)

31

Leader

Request, e.g., write(x, 1)Step 1. assign a monotonicallyincreasing id (zxid) to the request

Follower Follower

Step 2. Propose the changewith zxid to followers

Step 3. Followersacknowledge theproposal.

Step 4. Leader commitsthe change if receives morethan ½ of acks (votes).

Leader Election

• If a server finds the leader disconnected or failed, it tries to become the leader (same 2PC protocol).

32

LeaderCandidate

Follower Follower

Becomes the leader whenreceives ½ of votes.

If followers receive multiple proposal,they vote for the candidate with the highest zxid.

References

• “ZooKeeper: Wait-free coordination for Internet-scale systems,” USENIX ATC ‘10 (by Hunt et al.)

• Zab protocol: “A simple totally ordered broadcast protocol”, LADIS ’08 (by Reed and Junqueira)

• “Linearizability: A Correctness Condition for Concurrent Objects”, TOPLAS 1990 (by Herlihy and Wing)

• “Designing Data-Intensive Applications”, O’Reilly 2017(by Martin Kleppmann)

33