Gryff: Unifying Consensus and Shared Registers

Matthew Burke Audrey Cheng Wyatt Lloyd

Cornell University Princeton University

1

Applications Rely on Geo-Replicated Storage• Fault tolerant: data is safe despite failures

2

Client

Applications Rely on Geo-Replicated Storage• Fault tolerant: data is safe despite failures

• Linearizable: intuitive for application developers

3

≡

Linearizable Replicated Storage Systems

4

Cloud Spanner

Status Quo: Consensus or Shared Registers

Consensus Shared Registers

Strong Synchronization ✔ ✘Low ReadTail Latency ✘ ✔

5

• Given the desire for fault tolerance and linearizability

Unify consensus and shared registers?

Consensus & State Machine Replication (SMR)• Generic interface: Command(c(.))

• Stable ordering: all preceding log positions are assigned commands

6

c1 c2 c3 c4

Consensus & State Machine Replication (SMR)• Generic interface: Command(c(.))

• Stable ordering: all preceding log positions are assigned commands

• Used in etcd, CockroachDB, Spanner, Azure Storage, Chubby

6

c1 c2 c3 c4

SMR Requires Stable Order• Allow for strong synchronization primitives like read-modify-writes

• High tail latency in practice (e.g., by serializing through a leader)

7

Consensus

Strong Synchronization ✔Low ReadTail Latency ✘

Shared Registers• Simple interface: Read()/Write(v)

• Unstable ordering: total order without pre-defined positions

8

w1 < w2 < w3 < w4

Shared Registers• Simple interface: Read()/Write(v)

• Unstable ordering: total order without pre-defined positions

8

w1 < w2 < w3 < w4

w5

Shared Registers• Simple interface: Read()/Write(v)

• Unstable ordering: total order without pre-defined positions

• Similar to Cassandra, Dynamo, Riak

8

w1 < w2 < w3 < w4w5 <

Shared Registers Use Unstable Order• Cannot implement strong synchronization primitives [Herlihy91]

• Flexibility of unstable order provides favorable tail latency

9

Consensus Shared Registers

Strong Synchronization ✔ ✘Low ReadTail Latency ✘ ✔

RMWs with low read tail latency?

Shared Objects: Interface for Unification• Interface: Read()/Write(v)/RMW(f(.))

• RMW(f(.))→ read base v, compute new value f(v), write f(v)

• Examples: etcd, Redis, BigTable

10

Consensus-after-Register Timestamps (Carstamps)

11

rmw1 rmw3 rmw4

rmw2

Unstable Order

StableOrder w1 w2 w3 w4< < <

Consensus-after-Register Timestamps (Carstamps)

11

rmw1 rmw3 rmw4

rmw2

Unstable Order

StableOrder w1 w2 w3 w4w5 << < < w6<

Carstamps• Tuple with three fields: (ts, id, rmwc)

• ts and id basis for unstable ordering of writes

• rmwc is set to 1 greater than rmwc of base to ensure stable ordering

12

w1 < w2

rmw1 rmw3

(3,1,0)

(3,1,1)

(4,1,0)

(4,1,1)

rmw2(3,1,2)

Consensus Shared Registers

Gryff

Strong Synchronization ✔ ✘ ✔Low ReadTail Latency ✘ ✔ ✔

Gryff Unifies Consensus and Shared Registers• Only uses consensus when necessary, for strong synchronization

13

Gryff Design• Combine multi-writer [LS97] ABD [ABD95] & EPaxos [MAK13]

• Modifications needed for safety:• Carstamps for proper ordering

• Synchronous Commit phase for rmws

• Modifications for better read tail latency:• Early termination for reads (fast path)

• Proxy optimization for reads (fast path more often)

14

See the paper for details!

Gryff in Action

15

Gryff in Action

15

(2,3,0) (1,0,0) (2,3,0)

Gryff in Action

15

c1

(2,3,0) (1,0,0) (2,3,0)

w1→ (3,1,0)

Writes always terminate in 2 phases

Executed (3,1,1)

(3,1,1)(3,1,1)

Gryff in Action

18

c1

(2,3,0)

c2

rmw1→ (3,1,1)

Writes always terminate in 2 phases

RMW carstamps directly after base

Read1Reply (3,1,1)

(3,1,1)(3,1,1)

Gryff in Action

19

c1

(2,3,0)

c2

r → (3,1,1)

Writes always terminate in 2 phases

RMW carstamps directly after base

Reads often terminate in 1 phase

Evaluation

Relative to state-of-the-art-consensus protocols:

1. How do Gryff’s read/write protocols affect read tail latency?

2. What is the latency distribution of Gryff’s reads, writes, and rmws?

3. What maximum throughput does Gryff achieve?

4. How does Gryff perform in tail-at-scale workloads?

20

Evaluation

Relative to state-of-the-art-consensus protocols:

1. How do Gryff’s read/write protocols affect read tail latency?

2. What is the latency distribution of Gryff’s reads, writes, and rmws?

3. What maximum throughput does Gryff achieve?

4. How does Gryff perform in tail-at-scale workloads?

21

Evaluation Setup

22

• Geo-replication with 3 regions

• Baselines: MultiPaxos (industry standard), EPaxos (leaderless)

not-to-scale ocean

72ms 88ms

Read Tail Latency (94.5% R, 4.5% W, 1% RMW, 25% Conflicts)

23

0

0.2

0.4

0.6

0.8

1

60 120 180 240

Fra

ctio

n o

f R

ead

s

Latency (ms)

MultiPaxos

0

0.2

0.4

0.6

0.8

1

60 120 180 240

Fra

ctio

n o

f R

ead

s

Latency (ms)

MultiPaxos

Read Tail Latency (94.5% R, 4.5% W, 1% RMW, 25% Conflicts)

24

0

0.2

0.4

0.6

0.8

1

60 120 180 240

Fra

ctio

n o

f R

ead

s

Latency (ms)

MultiPaxos

Read Tail Latency (94.5% R, 4.5% W, 1% RMW, 25% Conflicts)

24

serializing through far-away leader

0

0.2

0.4

0.6

0.8

1

60 120 180 240

Fra

ctio

n o

f R

ead

s

Latency (ms)

MultiPaxos

EPaxos

Read Tail Latency (94.5% R, 4.5% W, 1% RMW, 25% Conflicts)

25

0

0.2

0.4

0.6

0.8

1

60 120 180 240

Fra

ctio

n o

f R

ead

s

Latency (ms)

MultiPaxos

EPaxos

Read Tail Latency (94.5% R, 4.5% W, 1% RMW, 25% Conflicts)

25

delaying reads that conflict with concurrent writes

0

0.2

0.4

0.6

0.8

1

60 120 180 240

Fra

ctio

n o

f R

ead

s

Latency (ms)

MultiPaxos

EPaxos

Gryff

Read Tail Latency (94.5% R, 4.5% W, 1% RMW, 25% Conflicts)

26

0

0.2

0.4

0.6

0.8

1

60 120 180 240

Fra

ctio

n o

f R

ead

s

Latency (ms)

MultiPaxos

EPaxos

Gryff

Read Tail Latency (94.5% R, 4.5% W, 1% RMW, 25% Conflicts)

26

1 round to nearest majority in tail

Summary• Consensus: strong synchronization w/ high tail latency

Shared registers: low tail latency w/o strong synchronization

• Carstamps stably order read-modify-writes within a more efficient unstable order for reads and writes

• Gryff unifies an optimized shared register protocol with a state-of-the-art consensus protocol using carstamps

• Gryff provides strong synchronization w/ low read tail latency

27

Image Attribution• Griffin by Delapouite / CC BY 3.0 Unported (modified)

• etcd

• CockroachDB

• Spanner by Google / CC BY 4.0

28