cs378h

Programming at Scale: Dataflow and Consistency

Today

Questions?

Administrivia

• Rust lab due today!

• Project Proposal Due Thursday!

Agenda:

• Dataflow Wrapup

• Concurrency & Consistency at Scale

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

map

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

map

reduce

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

map

reduce

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

map

reduce

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

map

reduce

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

map

reduce

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

map

reduce

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

map

reduce

Review: K-Means

public void kmeans() {

while(…) {

for each point

find_nearest_center(point);

for each center

compute_new_center(center)

}

}

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

fncfnc

fncfnc

fncfnc

cnccnc

cnccnc

Input Output

map

reduce

Key idea: adapt workload to parallel patternsQuestions:• What kinds of computations can this express?• What other patterns could we use?

How Does Parallelization Work?

INPUTFILE(s)

Execution

Group by?

Execution

Group by?

Key idea → shuffle == sort!

Task Granularity And Pipelining

|map tasks| >> |machines| -- why?

Task Granularity And Pipelining

|map tasks| >> |machines| -- why?Minimize fault recovery timePipeline map with other tasksEasier to load balance dynamically

MapReduce: not without Controversy

MapReduce: not without Controversy

MapReduce: not without Controversy

MapReduce: not without Controversy

Backwards step in programming paradigm

MapReduce: not without Controversy

Backwards step in programming paradigmSub-optimal: brute force, no indexing

MapReduce: not without Controversy

Backwards step in programming paradigmSub-optimal: brute force, no indexingNot novel: 25 year-old ideas from DBMS lit

It’s just a group-by aggregate engine

MapReduce: not without Controversy

Backwards step in programming paradigmSub-optimal: brute force, no indexingNot novel: 25 year-old ideas from DBMS lit

It’s just a group-by aggregate engine

Missing most DBMS featuresSchema, foreign keys, …

MapReduce: not without Controversy

Backwards step in programming paradigmSub-optimal: brute force, no indexingNot novel: 25 year-old ideas from DBMS lit

It’s just a group-by aggregate engine

Missing most DBMS featuresSchema, foreign keys, …

Incompatible with most DBMS tools

MapReduce: not without Controversy

Backwards step in programming paradigmSub-optimal: brute force, no indexingNot novel: 25 year-old ideas from DBMS lit

It’s just a group-by aggregate engine

Missing most DBMS featuresSchema, foreign keys, …

Incompatible with most DBMS tools

So why is it such a big success?

Why is MapReduce backwards?

Why is MapReduce backwards?

Backwards step in programming paradigm

Why is MapReduce backwards?

Backwards step in programming paradigmSub-optimal: brute force, no indexing

Why is MapReduce backwards?

Backwards step in programming paradigmSub-optimal: brute force, no indexingNot novel: 25 year-old ideas from DBMS lit

It’s just a group-by aggregate engine

Why is MapReduce backwards?

Backwards step in programming paradigmSub-optimal: brute force, no indexingNot novel: 25 year-old ideas from DBMS lit

It’s just a group-by aggregate engine

Missing most DBMS featuresSchema, foreign keys, …

Why is MapReduce backwards?

Backwards step in programming paradigmSub-optimal: brute force, no indexingNot novel: 25 year-old ideas from DBMS lit

It’s just a group-by aggregate engine

Missing most DBMS featuresSchema, foreign keys, …

Incompatible with most DBMS tools

Why is MapReduce backwards?

Backwards step in programming paradigmSub-optimal: brute force, no indexingNot novel: 25 year-old ideas from DBMS lit

It’s just a group-by aggregate engine

Missing most DBMS featuresSchema, foreign keys, …

Incompatible with most DBMS tools

So why is it such a big success?

MapReduce and Dataflow

MapReduce and Dataflow

• MR is a dataflow engine

MapReduce and Dataflow

• MR is a dataflow engine

MapReduce and Dataflow

• MR is a dataflow engine

• Lots of others• Dryad

• DryadLINQ

• Dandelion

• CIEL

• GraphChi/Pregel

• Spark

MapReduce and Dataflow

• MR is a dataflow engine

• Lots of others• Dryad

• DryadLINQ

• Dandelion

• CIEL

• GraphChi/Pregel

• Spark

MapReduce and Dataflow

• MR is a dataflow engine

• Lots of others• Dryad

• DryadLINQ

• Dandelion

• CIEL

• GraphChi/Pregel

• Spark

Taxonomies:

• DAG instead of BSP

• Interface variety• Memory FIFO• Disk• Network

• Flexible Modular Composition

Dryad (2007): 2-D Piping• Unix Pipes: 1-D

grep | sed | sort | awk | perl

Dryad (2007): 2-D Piping• Unix Pipes: 1-D

grep | sed | sort | awk | perl

• Dryad: 2-D

grep1000 | sed500 | sort1000 | awk500 | perl50

Dataflow Engines

Dataflow Job Structure

grep

sed

sortawk

perlgrep

grepsed

sort

sort

awk

Inputfiles

Vertices (processes)

Outputfiles

Channels

Stage

Dataflow Job Structure

grep

sed

sortawk

perlgrep

grepsed

sort

sort

awk

Inputfiles

Vertices (processes)

Outputfiles

Channels

Stage

How to implement?

Channels

X

M

Items

Finite streams of items

• distributed filesystem files(persistent)

• SMB/NTFS files (temporary)

• TCP pipes(inter-machine)

• memory FIFOs (intra-machine)

Channels

X

M

Items

Finite streams of items

• distributed filesystem files(persistent)

• SMB/NTFS files (temporary)

• TCP pipes(inter-machine)

• memory FIFOs (intra-machine)

Key idea:Encapsulate data movement behind

channel abstraction → gets programmer out of the picture

Spark (2012) Background

Commodity clusters: important platformIn industry: search, machine translation, ad targeting, …

In research: bioinformatics, NLP, climate simulation, …

Cluster-scale models (e.g. MR) de facto standardFault tolerance through replicated durable storage

Dataflow is the common theme

Spark (2012) Background

Commodity clusters: important platformIn industry: search, machine translation, ad targeting, …

In research: bioinformatics, NLP, climate simulation, …

Cluster-scale models (e.g. MR) de facto standardFault tolerance through replicated durable storage

Dataflow is the common theme

Multi-coreIteration

Motivation

Programming models for clusters transform data flowing from stable storage to stable storage

E.g., MapReduce:

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Input Output

Motivation

Programming models for clusters transform data flowing from stable storage to stable storage

E.g., MapReduce:

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Input Output

Benefits of data flow: runtime can decide where to run tasks and can automatically

recover from failures

Iterative Computations: PageRank

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Input Output

Iterative Computations: PageRank

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Input Output

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Output

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Output

Iterative Computations: PageRank

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Input Output

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Output

MapMap

MapMap

MapMap

ReduceReduce

ReduceReduce

Output

Solution: augment data flow model with “resilient distributed datasets” (RDDs)

Programming Model

• Resilient distributed datasets (RDDs)• Immutable collections partitioned across cluster that can

be rebuilt if a partition is lost

• Created by transforming data in stable storage using data flow operators (map, filter, group-by, …)

• Can be cached across parallel operations

Programming Model

• Resilient distributed datasets (RDDs)• Immutable collections partitioned across cluster that can

be rebuilt if a partition is lost

• Created by transforming data in stable storage using data flow operators (map, filter, group-by, …)

• Can be cached across parallel operations

• Parallel operations on RDDs• Reduce, collect, count, save, …

Programming Model

• Resilient distributed datasets (RDDs)• Immutable collections partitioned across cluster that can

be rebuilt if a partition is lost

• Created by transforming data in stable storage using data flow operators (map, filter, group-by, …)

• Can be cached across parallel operations

• Parallel operations on RDDs• Reduce, collect, count, save, …

• Restricted shared variables• Accumulators, broadcast variables

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”) WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”) WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Base RDDBase RDD

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”) WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Transformed RDDTransformed RDD

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Cached RDDCached RDD

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).countParallel operationParallel operation

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

tasks

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

tasks

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

tasks

results

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

tasks

results

Cache 1Cache 1

Cache 2Cache 2

Cache 3Cache 3

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

Cache 1Cache 1

Cache 2Cache 2

Cache 3Cache 3

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

cachedMsgs.filter(_.contains(“bar”)).count

Cache 1Cache 1

Cache 2Cache 2

Cache 3Cache 3

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

cachedMsgs.filter(_.contains(“bar”)).count

tasks

Cache 1Cache 1

Cache 2Cache 2

Cache 3Cache 3

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

cachedMsgs.filter(_.contains(“bar”)).count

tasks

Cache 1Cache 1

Cache 2Cache 2

Cache 3Cache 3

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

cachedMsgs.filter(_.contains(“bar”)).count

tasks

results

Cache 1Cache 1

Cache 2Cache 2

Cache 3Cache 3

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

cachedMsgs.filter(_.contains(“bar”)).count

. . .

tasks

results

Cache 1Cache 1

Cache 2Cache 2

Cache 3Cache 3

Example: Log Mining• Load error messages from a log into memory, then

interactively search for various patterns

lines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))

messages = errors.map(_.split(‘\t’)(2))

cachedMsgs = messages.cache()

Block 1Block 1

Block 2Block 2

Block 3Block 3

WorkerWorker

WorkerWorker

WorkerWorker

DriverDriver

cachedMsgs.filter(_.contains(“foo”)).count

cachedMsgs.filter(_.contains(“bar”)).count

. . .

tasks

results

Cache 1Cache 1

Cache 2Cache 2

Cache 3Cache 3

Result: full-text search of Wikipedia in <1 sec (vs20 sec for on-disk data)

RDD Fault Tolerance

• RDDs maintain lineage information that can be used to reconstruct lost partitions

• Ex:cachedMsgs = textFile(...).filter(_.contains(“error”))

.map(_.split(‘\t’)(2))

.persist()

HdfsRDDpath: hdfs://…

HdfsRDDpath: hdfs://…

FilteredRDDfunc: contains(...)

FilteredRDDfunc: contains(...)

MappedRDDfunc: split(…)

MappedRDDfunc: split(…)

CachedRDDCachedRDD

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Data-Parallel Computation Systems

Execution

Application

Storage

Language

ParallelDatabases

SQL

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

ParallelDatabases

SQL

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

Sawzall

ParallelDatabases

SQL Sawzall

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

Sawzall

ParallelDatabases

SQL Sawzall

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

Sawzall

ParallelDatabases

SQL Sawzall

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

Sawzall

Hadoop

HDFSS3

ParallelDatabases

SQL Sawzall

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

Sawzall

Hadoop

HDFSS3

Pig, Hive

ParallelDatabases

SQL ≈SQLSawzall

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

Sawzall

Hadoop

HDFSS3

Pig, Hive

ParallelDatabases

SQL ≈SQLSawzall

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

Sawzall

Hadoop

HDFSS3

Pig, Hive

ParallelDatabases

SQL ≈SQLSawzall

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

CosmosAzure

SQL Server

Dryad

DryadLINQScope

Sawzall

Hadoop

HDFSS3

Pig, Hive

ParallelDatabases

SQL ≈SQLSawzall

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

CosmosAzure

SQL Server

Dryad

DryadLINQScope

Sawzall

Hadoop

HDFSS3

Pig, Hive

ParallelDatabases

SQL ≈SQL LINQ, SQLSawzall

Cosmos, HPC, Azure

Data-Parallel Computation Systems

Execution

Application

Storage

Language

Map-Reduce

GFSBigTable

CosmosAzure

SQL Server

Dryad

DryadLINQScope

Sawzall

Hadoop

HDFSS3

Pig, Hive

ParallelDatabases

SQL ≈SQL LINQ, SQLSawzall

Cosmos, HPC, Azure

Spark

(Yet) Another Framework

22

(Yet) Another Framework

22

Consistency

(Yet) Another Framework

22

Da

taM

od

el

Consistency

(Yet) Another Framework

22

Da

taM

od

el

Consistency

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

• Atomicity• Consistency• Isolation• Durability

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

• Basically Available• Soft State• Eventually Consistent

• Atomicity• Consistency• Isolation• Durability

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Key Value Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Key Value Stores

Document Stores

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Key Value Stores

Document Stores

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

• Shared-Disk• Range-Sharding• Hash-Sharding• Consistent Hashing

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

• Primary-Backup• Commit-Consensus

Protocol• Sync/Async

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

• Logging• Update In Place• Caching• In-Memory Storage

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support• Secondary Indexing• Query Planning• Materialized Views• Analytics

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

Wide-Column Stores

(Yet) Another Framework

22

Strong: ACID Eventual: BASE

Da

taM

od

el

Consistency

Sharding/Partitioning

Key Value Stores

Document Stores

Replication

Storage

Query Support

Still not a perfect framework

Cons:

● Many dimensions contain sub-dimensions

● Many concerns fundamentally coupled

● Dimensions are often un- or partially-ordered

Pros:

• Makes important concerns explicit

• Cleanly taxonomizes most modern systems

Consistency

Consistency

How to keep data in sync?

Consistency

How to keep data in sync?

• Partitioning → single row spread over multiple machines

Consistency

How to keep data in sync?

• Partitioning → single row spread over multiple machines

Consistency

Partitions

How to keep data in sync?

• Partitioning → single row spread over multiple machines

Consistency

Partitions

How to keep data in sync?

• Partitioning → single row spread over multiple machines

• Redundancy → single datum spread over multiple machines

Consistency

Partitions

How to keep data in sync?

• Partitioning → single row spread over multiple machines

• Redundancy → single datum spread over multiple machines

Consistency

Partitions

How to keep data in sync?

• Partitioning → single row spread over multiple machines

• Redundancy → single datum spread over multiple machines

Consistency: the core problem

R1 R2writer reader

Write(k,v) Read(k,v)

Consistency: the core problem

• Clients perform reads and writes

R1 R2writer reader

Write(k,v) Read(k,v)

Consistency: the core problem

• Clients perform reads and writes• Data is replicated among a set of servers

R1 R2writer reader

Write(k,v) Read(k,v)

Consistency: the core problem

• Clients perform reads and writes• Data is replicated among a set of servers

• Writes must be performed at all servers

R1 R2writer reader

Write(k,v) Read(k,v)

Consistency: the core problem

• Clients perform reads and writes• Data is replicated among a set of servers

• Writes must be performed at all servers

• Reads return the result of one or more past writes

R1 R2writer reader

Write(k,v) Read(k,v)

Consistency: the core problem

• Clients perform reads and writes• Data is replicated among a set of servers

• Writes must be performed at all servers

• Reads return the result of one or more past writes

R1 R2writer reader

Write(k,v) Read(k,v)

• How should we implement write?

Consistency: the core problem

• Clients perform reads and writes• Data is replicated among a set of servers

• Writes must be performed at all servers

• Reads return the result of one or more past writes

R1 R2writer reader

Write(k,v) Read(k,v)

• How should we implement write?• How to implement read?

Consistency: CAP Theorem

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability:

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance:

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Why care about CAP Properties?Availability

•Reads/writes complete reliably and quickly.•E.g. Amazon, each ms latency → $6M yearly loss.

Partitions• Internet router outages• Under-sea cables cut• rack switch outage• system should continue functioning normally!

Consistency • all nodes see same data at any time, or reads return latest

written value by any client.• This basically means correctness!

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Why care about CAP Properties?Availability

•Reads/writes complete reliably and quickly.•E.g. Amazon, each ms latency → $6M yearly loss.

Partitions• Internet router outages• Under-sea cables cut• rack switch outage• system should continue functioning normally!

Consistency • all nodes see same data at any time, or reads return latest

written value by any client.• This basically means correctness!

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Why care about CAP Properties?Availability

•Reads/writes complete reliably and quickly.•E.g. Amazon, each ms latency → $6M yearly loss.

Partitions• Internet router outages• Under-sea cables cut• rack switch outage• system should continue functioning normally!

Consistency • all nodes see same data at any time, or reads return latest

written value by any client.• This basically means correctness!

Why is this “theorem” true?

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Why care about CAP Properties?Availability

•Reads/writes complete reliably and quickly.•E.g. Amazon, each ms latency → $6M yearly loss.

Partitions• Internet router outages• Under-sea cables cut• rack switch outage• system should continue functioning normally!

Consistency • all nodes see same data at any time, or reads return latest

written value by any client.• This basically means correctness!

Why is this “theorem” true?

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Why care about CAP Properties?Availability

•Reads/writes complete reliably and quickly.•E.g. Amazon, each ms latency → $6M yearly loss.

Partitions• Internet router outages• Under-sea cables cut• rack switch outage• system should continue functioning normally!

Consistency • all nodes see same data at any time, or reads return latest

written value by any client.• This basically means correctness!

Why is this “theorem” true?

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Why care about CAP Properties?Availability

•Reads/writes complete reliably and quickly.•E.g. Amazon, each ms latency → $6M yearly loss.

Partitions• Internet router outages• Under-sea cables cut• rack switch outage• system should continue functioning normally!

Consistency • all nodes see same data at any time, or reads return latest

written value by any client.• This basically means correctness!

Why is this “theorem” true?

if(partition) { keep going } → !consistent && available

Consistency: CAP Theorem

• A distributed system can satisfy at most 2/3 guarantees of:1. Consistency:

• all nodes see same data at any time

• or reads return latest written value by any client

2. Availability: • system allows operations all the time,

• and operations return quickly

3. Partition-tolerance: • system continues to work in spite of network partitions

Why care about CAP Properties?Availability

•Reads/writes complete reliably and quickly.•E.g. Amazon, each ms latency → $6M yearly loss.

Partitions• Internet router outages• Under-sea cables cut• rack switch outage• system should continue functioning normally!

Consistency • all nodes see same data at any time, or reads return latest

written value by any client.• This basically means correctness!

Why is this “theorem” true?

if(partition) { keep going } → !consistent && availableif(partition) { stop } → consistent && !available

CAP Implications

• A distributed storage system can achieve at most two of C, A, and P.

• When partition-tolerance is important, you have to choose between consistency and availability

Consistency

Partition-tolerance Availability

RDBMSs

(non-replicated)

Cassandra, RIAK,

Dynamo, Voldemort

HBase, HyperTable,

BigTable, Spanner

CAP Implications

• A distributed storage system can achieve at most two of C, A, and P.

• When partition-tolerance is important, you have to choose between consistency and availability

Consistency

Partition-tolerance Availability

RDBMSs

(non-replicated)

Cassandra, RIAK,

Dynamo, Voldemort

HBase, HyperTable,

BigTable, Spanner

CAP is flawed

CAP Implications

• A distributed storage system can achieve at most two of C, A, and P.

• When partition-tolerance is important, you have to choose between consistency and availability

Consistency

Partition-tolerance Availability

RDBMSs

(non-replicated)

Cassandra, RIAK,

Dynamo, Voldemort

HBase, HyperTable,

BigTable, Spanner

PACELC:

if(partition) {choose A or C

} else {choose latency or consistency

}

CAP is flawed

Consistency Spectrum

Strong

(e.g., Sequential)Eventual

More consistency

Faster reads and writes

Spectrum Ends: Eventual Consistency

• Eventual Consistency• If writes to a key stop, all replicas of key will converge

• Originally from Amazon’s Dynamo and LinkedIn’s Voldemort systems

Strong

(e.g., Sequential)Eventual

More consistency

Faster reads and writes

Spectrum Ends: Strong Consistency

• Strict:• Absolute time ordering of all shared accesses, reads always return last write

• Linearizability: • Each operation is visible (or available) to all other clients in real-time order

• Sequential Consistency [Lamport]:• "... the result of any execution is the same as if the operations of all the processors

were executed in some sequential order, and the operations of each individual processor appear in this sequence in the order specified by its program.

• After the fact, find a “reasonable” ordering of the operations (can re-order operations) that obeys sanity (consistency) at all clients, and across clients.

• ACID properties

Many Many Consistency Models

EventualStrong

(e.g., Sequential, Strict)

CausalRed-Blue

CRDTsPer-key sequential

Probabilistic

Many Many Consistency Models

EventualStrong

(e.g., Sequential, Strict)

CausalRed-Blue

CRDTsPer-key sequential

Probabilistic

• Amazon S3 – eventual consistency

• Amazon Simple DB – eventual or strong

• Google App Engine – strong or eventual

• Yahoo! PNUTS – eventual or strong

• Windows Azure Storage – strong (or eventual)

• Cassandra – eventual or strong (if R+W > N)

• ...

Many Many Consistency Models

EventualStrong

(e.g., Sequential, Strict)

CausalRed-Blue

CRDTsPer-key sequential

Probabilistic

Question: How to choose what to use or support?

• Amazon S3 – eventual consistency

• Amazon Simple DB – eventual or strong

• Google App Engine – strong or eventual

• Yahoo! PNUTS – eventual or strong

• Windows Azure Storage – strong (or eventual)

• Cassandra – eventual or strong (if R+W > N)

• ...

Some Consistency Guarantees

Strong Consistency See all previous writes.

Eventual Consistency See subset of previous writes.

Consistent Prefix See initial sequence of writes.

Bounded Staleness See all “old” writes.

Monotonic Reads See increasing subset of writes.

Read My Writes See all writes performed by reader.

A D F

D A A

C B A

B C D

C B B

C C C

Some Consistency Guarantees

Strong Consistency See all previous writes.

Eventual Consistency See subset of previous writes.

Consistent Prefix See initial sequence of writes.

Bounded Staleness See all “old” writes.

Monotonic Reads See increasing subset of writes.

Read My Writes See all writes performed by reader.

A D F

D A A

C B A

B C D

C B B

C C C

Some Consistency Guarantees

Strong Consistency See all previous writes.

Eventual Consistency See subset of previous writes.

Consistent Prefix See initial sequence of writes.

Bounded Staleness See all “old” writes.

Monotonic Reads See increasing subset of writes.

Read My Writes See all writes performed by reader.

A D F

D A A

C B A

B C D

C B B

C C C

Strong

RMWMonotonicBoundedPrefix

Eventual

metric =set of allowable read results

strength

The Game of Soccer

The Game of Soccer

The Game of Soccer

The Game of Soccer

The Game of Soccerfor half = 1 .. 2 {

The Game of Soccerfor half = 1 .. 2 {

while half not over {

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

score = Read (“home”);

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

score = Read (“home”);

Write (“home”, score + 1);

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

score = Read (“home”);

Write (“home”, score + 1);

} } }

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

score = Read (“home”);

Write (“home”, score + 1);

} } }

hScore = Read(“home”);

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

score = Read (“home”);

Write (“home”, score + 1);

} } }

hScore = Read(“home”);

vScore = Read(“visit”);

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

score = Read (“home”);

Write (“home”, score + 1);

} } }

hScore = Read(“home”);

vScore = Read(“visit”);

if (hScore == vScore)

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

score = Read (“home”);

Write (“home”, score + 1);

} } }

hScore = Read(“home”);

vScore = Read(“visit”);

if (hScore == vScore)

play-overtime

The Game of Soccerfor half = 1 .. 2 {

while half not over {

kick-the-ball-at-the-goal

for each goal {

if visiting-team-scored {

score = Read (“visitors”);

Write (“visitors”, score + 1);

} else {

score = Read (“home”);

Write (“home”, score + 1);

} } }

hScore = Read(“home”);

vScore = Read(“visit”);

if (hScore == vScore)

play-overtime

Official Scorekeeper

33

score = Read (“visitors”);

Write (“visitors”, score + 1);

Official Scorekeeper

Desired consistency?

33

score = Read (“visitors”);

Write (“visitors”, score + 1);

Official Scorekeeper

Desired consistency?

Strong

33

score = Read (“visitors”);

Write (“visitors”, score + 1);

Official Scorekeeper

Desired consistency?

Strong

= Read My Writes!

33

score = Read (“visitors”);

Write (“visitors”, score + 1);

Official Scorekeeper

Desired consistency?

Strong

= Read My Writes!

33

score = Read (“visitors”);

Write (“visitors”, score + 1);

Referee

34

vScore = Read (“visitors”);

hScore = Read (“home”);

if vScore == hScore

play-overtime

Referee

Desired consistency?

34

vScore = Read (“visitors”);

hScore = Read (“home”);

if vScore == hScore

play-overtime

Referee

Desired consistency?

Strong consistency

34

vScore = Read (“visitors”);

hScore = Read (“home”);

if vScore == hScore

play-overtime

Radio Reporter

35

do {BeginTx();

vScore = Read (“visitors”);hScore = Read (“home”);

EndTx();report vScore and hScore;sleep (30 minutes);

}

Radio Reporter

Desired consistency?

35

do {BeginTx();

vScore = Read (“visitors”);hScore = Read (“home”);

EndTx();report vScore and hScore;sleep (30 minutes);

}

Radio Reporter

Desired consistency?

Consistent Prefix

35

do {BeginTx();

vScore = Read (“visitors”);hScore = Read (“home”);

EndTx();report vScore and hScore;sleep (30 minutes);

}

Radio Reporter

Desired consistency?

Consistent Prefix

Monotonic Reads

35

do {BeginTx();

vScore = Read (“visitors”);hScore = Read (“home”);

EndTx();report vScore and hScore;sleep (30 minutes);

}

Radio Reporter

Desired consistency?

Consistent Prefix

Monotonic Readsor Bounded Staleness

35

do {BeginTx();

vScore = Read (“visitors”);hScore = Read (“home”);

EndTx();report vScore and hScore;sleep (30 minutes);

}

Radio Reporter

Desired consistency?

Consistent Prefix

Monotonic Readsor Bounded Staleness

35

do {BeginTx();

vScore = Read (“visitors”);hScore = Read (“home”);

EndTx();report vScore and hScore;sleep (30 minutes);

}

Sportswriter

36

While not end of game {

drink beer;

smoke cigar;

}

go out to dinner;

vScore = Read (“visitors”);

hScore = Read (“home”);

write article;

Sportswriter

Desired consistency?

36

While not end of game {

drink beer;

smoke cigar;

}

go out to dinner;

vScore = Read (“visitors”);

hScore = Read (“home”);

write article;

Sportswriter

Desired consistency?

Eventual

36

While not end of game {

drink beer;

smoke cigar;

}

go out to dinner;

vScore = Read (“visitors”);

hScore = Read (“home”);

write article;

Sportswriter

Desired consistency?

Eventual

Bounded Staleness36

While not end of game {

drink beer;

smoke cigar;

}

go out to dinner;

vScore = Read (“visitors”);

hScore = Read (“home”);

write article;

Statistician

37

Wait for end of game;

score = Read (“home”);

stat = Read (“season-goals”);

Write (“season-goals”, stat + score);

Statistician

Desired consistency?

37

Wait for end of game;

score = Read (“home”);

stat = Read (“season-goals”);

Write (“season-goals”, stat + score);

Statistician

Desired consistency?

Strong Consistency (1st read)

37

Wait for end of game;

score = Read (“home”);

stat = Read (“season-goals”);

Write (“season-goals”, stat + score);

Statistician

Desired consistency?

Strong Consistency (1st read)

Read My Writes (2nd read)

37

Wait for end of game;

score = Read (“home”);

stat = Read (“season-goals”);

Write (“season-goals”, stat + score);

Stat Watcher

38

do {

stat = Read (“season-goals”);

discuss stats with friends;

sleep (1 day);

}

Stat Watcher

Desired consistency?

38

do {

stat = Read (“season-goals”);

discuss stats with friends;

sleep (1 day);

}

Stat Watcher

Desired consistency?

Eventual Consistency

38

do {

stat = Read (“season-goals”);

discuss stats with friends;

sleep (1 day);

}

Official scorekeeper:score = Read (“visitors”);

Write (“visitors”, score + 1);

Statistician:

Wait for end of game;

score = Read (“home”);

stat = Read (“season-goals”);

Write (“season-goals”, stat + score);

Referee:

Radio reporter:do {

vScore = Read (“visitors”);

hScore = Read (“home”);

report vScore and hScore;

sleep (30 minutes);

}

Sportswriter:

While not end of game {

drink beer;

smoke cigar;

}

go out to dinner;

vScore = Read (“visitors”);

hScore = Read (“home”);

write article;

Stat watcher:

stat = Read (“season-runs”);

discuss stats with friends;

Sequential Consistency

• weaker than strict/strong consistency• All operations are executed in some sequential order

• each process issues operations in program order

• Any valid interleaving is allowed

• All agree on the same interleaving

• Each process preserves its program order

Sequential Consistency

• weaker than strict/strong consistency• All operations are executed in some sequential order

• each process issues operations in program order

• Any valid interleaving is allowed

• All agree on the same interleaving

• Each process preserves its program order

• Why is this weaker than strict/strong?

Sequential Consistency

• weaker than strict/strong consistency• All operations are executed in some sequential order

• each process issues operations in program order

• Any valid interleaving is allowed

• All agree on the same interleaving

• Each process preserves its program order

• Why is this weaker than strict/strong?

• Nothing is said about “most recent write”

Linearizability

Linearizability

• Assumes sequential consistency and• If TS(x) < TS(y) then OP(x) should precede OP(y) in the sequence

• Stronger than sequential consistency

• Difference between linearizability and serializability?

• Granularity: reads/writes versus transactions

Linearizability

• Assumes sequential consistency and• If TS(x) < TS(y) then OP(x) should precede OP(y) in the sequence

• Stronger than sequential consistency

• Difference between linearizability and serializability?

• Granularity: reads/writes versus transactions

•Example:•Stay tuned…relevant for lock free data structures

•Importantly: a property of concurrent objects

Causal consistency

Causal consistency

• Causally related writes seen by all processes in same order.

Causal consistency

• Causally related writes seen by all processes in same order. • Causally?

Causal consistency

• Causally related writes seen by all processes in same order. • Causally?

Causal:If a write produces a value thatcauses another write, they are causally related

X = 1if(X > 0) {

Y = 1}Causal consistency → all see X=1, Y=1 in same order

Causal consistency

• Causally related writes seen by all processes in same order. • Causally?

Causal consistency

• Causally related writes seen by all processes in same order. • Causally?

• Concurrent writes may be seen in different orders on different machines

Causal consistency

• Causally related writes seen by all processes in same order. • Causally?

• Concurrent writes may be seen in different orders on different machines

Causal consistency

• Causally related writes seen by all processes in same order. • Causally?

• Concurrent writes may be seen in different orders on different machines

Not permitted

Causal consistency

• Causally related writes seen by all processes in same order. • Causally?

• Concurrent writes may be seen in different orders on different machines

Not permitted

Causal consistency

• Causally related writes seen by all processes in same order. • Causally?

• Concurrent writes may be seen in different orders on different machines

Not permittedPermitted

Consistency models summary

Consistency models summary

Consistency Description

Strict Absolute time ordering of all shared accesses matters.

LinearizabilityAll processes must see all shared accesses in the same order. Accesses are furthermore ordered

according to a (nonunique) global timestamp

Sequential All processes see all shared accesses in the same order. Accesses are not ordered in time

Causal All processes see causally-related shared accesses in the same order.

FIFOAll processes see writes from each other in the order they were used. Writes from different processes

may not always be seen in that order

(a)

Consistency Description

Weak Shared data can be counted on to be consistent only after a synchronization is done

Release Shared data are made consistent when a critical region is exited

Entry Shared data pertaining to a critical region are made consistent when a critical region is entered.

(b)