1

Seneca: Remote Mirroring Done Write

Minwen Ji ,

Alistair Veitch and John Wilkes

HP Labs

April 21, 2023

2

Motivations: Reliability and Availability

• 2 out of 5 enterprises that experience a disaster go out of business within 5 years [Gartner Report]

• Outages cost >$250K/hour (25%) or >$5M/hour (4%) [Eagle Rock Online Survey]

3

Our Contributions

• A taxonomy of the design choices for remote mirroring

• An asynchronous protocol that is designed to handle many kinds and sequences of failures

• Checking the correctness of the protocol using I/O automata-based simulation

4

Remote Mirroring Overview

Competing goals:High performance, low cost, and low data loss

App App App

MirroringModule

MirroringModule

Local Remote

Wide Area Network

App App App

Primary

SecondarySecondary Primary

PrimaryLog

SecondaryLog

5

Design Choices• Synchronous vs. asynchronous

– Propagate update to mirror before vs. after write request returned to application

• Divergence: zero bounded, op/byte/time bounded, resource bounded, unbounded– Amount of data allowed to be out-of-sync between

mirrors

• As-is logging vs. write coalescing– Store all versions vs. a subset of versions of overwritten

data in log

6

Seneca’s Choices

• Synchronous vs. asynchronous– Low data loss vs. smooth traffic and high performance

• Divergence: zero bounded, op/byte/time bounded, resource bounded, unbounded– Low data loss vs. smooth traffic and high availability

• As-is logging vs. write coalescing– Little secondary log space vs. low primary log space

and low traffic

7

A Taxonomy

0

1

2

3

4

Divergence Bound

Pro

paga

tion

VeritasIBM-PRRCIBM-XRCEMCNetAppsHP-XPHP-SV3000Seneca

4 Async- Bitmap 3 Async- Coalesce2 Async1 Sync

Avail+Cost –Loss +

Perf +Cost –Loss +

8

Evaluation of Seneca’s Choices

Metrics:

Impact of asynchronous propagation and write coalescing on WAN traffic and log space

Traces Capacity Length Mean Write Rate

Cello2002 1.44TB 24 hours 0.78 MB/s

SAP 4TB 15 mins 1.95 MB/s

RDW 500GB 1.4 hours 0.34 MB/s

OpenMail 640GB 1 hour 1.70 MB/s

9

Simulation Results

• Mean traffic: 5-40% reduction with write coalescing allowed within 30 sec intervals

• 95th percentile usage: reduced from 93% of 4 T3 lines to 85% of 3 T3 lines

• Log space: 100 GB log will cover a network outage for 14-81 hours

Mean Traffic vs. Coalescing Interval Traffic CDF w/ Coalescing On/Off

Log Size vs. Coalescing Interval

10

How To Get Things Right

• Hard problems:– Rolling disasters

• Primary fails => secondary inconsistent => system inaccessible

– Failover dilemmas• Primary fails before propagation

• Secondary takes over and continues to update

• Old primary returns

• Our approach:– Finite state machines

11

Local Seneca State Remote Seneca State

Primary State

Secondary State

12

Checking Correctness• Simulation

– Started with Input/Output Automata (a model checking language) – Constrained random walks in the state space– Implemented in C

• Correctness criteria– Coverage, safety and liveness

• Latest results– Detected and fixed many non-trivial implementation bugs in a

relatively short time– Average failure injections before a bug is detected: 16435– Mean Time Between Failures for the protocol proper: 4100 years– The latest bug took 1.77M writes, 75.9K failures, 22.4K recoveries

and 6.6M internal events to detect

13

Summary

• A taxonomy of design space for remote mirroring

• Evaluation of Seneca’s design choices

• A finite state machine description of the Seneca remote mirroring protocol

• Checking the correctness of Seneca with simulations