CS 3471 CS 347: Parallel and Distributed Data Management Notes13: Time and Clocks.

CS 347 1

CS 347: Parallel and Distributed

Data ManagementNotes13: Time and Clocks

CS 347 2

Reference:

• Leslie Lamport, Time, clocks, and the ordering of events in a distributed system, Communications of the ACM, Volume 21 , Issue 7 (July 1978), pages: 558 - 565.

• Available at:http://portal.acm.org/citation.cfm?id=359563

CS 347 3

Time and clocks• Time and clocks are fundamental

concepts in distributed systemse.g.:timeoutsidentifying calls, transactions,…setting prioritiesversions of data...

CS 347 4

• What is time?• How can clocks be implemented?

Questions

CS 347 5

Ordering of events• More basic than time: event ordering

event a happened event b 9 am before 10 am

• Do not need physical time to order events

event a happened before event b can affect

CS 347 6

Ordering of events

• “” is a partial ordering[ total ordering: for any two

events a,b (ab) either a b or b

apartial ordering: a,b can be concurrent ]

CS 347 7

The modelProcess Process

Process P Q RP4

P3

P2

P1

R4

R3

R2

R1

Q7

Q6

Q3

Q2

Q5

Q1

Q4

T

I

M

E

CS 347 8

• Events within a process are totally ordered• Sending or receiving a message is an

event• No assumptions about message

transmission timesE.g.: P1 P2

P1 Q2 P3, Q3 concurrent

CS 347 9

DefinitionThe relation on the set of events of a

system is the smallest relation satisfying the following 3 conditions:1) if a,b events in same process, and a comes before b, then a b2) if a is sending a message and b is receipt of same message by a different process, then a b3) if a b and b c, than a c

• assume a a• if a b and b a, than a,b are concurrent

CS 347 10

Logical ClocksProcess Process Process P Q R

P4

P3

P2

P1

R4

R3

R2

R1

Q7

Q6

Q3

Q2

Q5

Q1

Q4

T

I

M

E

CR=9

CR=8

CR=7CR=6

CR=5

Overview• Logical Clocks• Physical Clocks

– basic properties– synchronization scheme– synchronization with clock server– probabilistic synchronization

CS 347 11

CS 347 12

Logical Clock

• Ci =logical clock (counter) at process i

• C[b] = reading of Cj when event b occurs at process j

• Clock condition for any events a,b IF a b THEN C[a] < C[b]

• No relationship to physical time

CS 347 13

Clock Condition

If a and b are events in process i and a comes before b, then Ci[a] < Ci[b]

If a is the sending of a message by process i and b is the receipt of that message by process j, then

Ci[a] < Cj[b]

C1

C2

If a b THEN C[a] < C[b]can be satisfied if the following conditions hold:

CS 347 14

Process Process Process P Q R

P4

P3

P2

P1

R4

R3

R2

R1

Q7

Q6

Q3

Q2

Q5

Q1

Q4

T

I

M

E

CR=7

CR=5

CQ=10

7>5by C1

10>7by C2

CS 347 15

To implement C1, C2:IR1 Each process i increments Ci

between any two successive eventsIR2 - Let a be event “process i sends

message to j”- Message contains timestamp Tm = Ci[a]- When message arrives at j(1) IF Tm > Cj THEN Cj Tm (2) event “arrival of message”takes place

CS 347 16

Note: Ci[a] < Cj[b] a b

Note: a,b concurrent Ci[a] = Cj[b]

Note: Ci[a] = Cj[b] a,b concurrent

CS 347 17

Ordering Events Totally (breaking ties)Example:• A server wants to execute requests in order

a is event that originated one request (at client)b is event that originates second request (at other client)

• If C[a] < C[b], service a first (it could be that a b)

• If C[a] = C[b], pick one (a,b concurrent)e.g., pick one with lower [ node#, process id ]

CS 347 18

Total Ordering• Let “< ” be a total ordering of the

processes• Define total ordering of events “”

a b (a event in process i; b in j) if and only if

(1) Ci[a] < Cj[b] or(2) Ci[a] = Cj[b] and i < j

• “” not unique

CS 347 19

Anomalous behavior

C=100 C=50

Reserve a Telephone Reserve a seat call seat

server

Client A Client B1

2

3

CS 347 20

Anomalous behavior

C=100 C=50

Reserve a Telephone Reserve a seat call seat

server

Client A Client B1

2

3

C= 9am C= 9am

CS 347 21

Solutions(1) Include “telephone call” in “system”(2) Use perfect physical clocks(3) Use real physical clocks

- may be “slightly off”- does not eliminate anomaly, but reduces its likelihood- useful for fault detection

CS 347 22

Physical ClocksCi(t) = clock reading at process i

at physical time t

time=t

Ci(t)

CS 347 23

time=t

Ci(t)Ci

+(x)

Ci-(x)

Ci-(x)=lim Ci(x - ||)

Ci+(x)=lim Ci(x + ||)

0

x

• Assume Ci(t) is a continuous, differentiable function except when clock is reset (message arrived)

0

CS 347 24

• How do we enforce clock condition?If a b then C(a) < C(b)

CS 347 25

Each process i increments Cibetween any two

successive eventsFor each process i, if i does not

receive a message at time t, then Ci is differentiable at t and

dCi(t) dt

IR1

IR1’

Logicalclocks

> 0

process i

a

b

TIME

CS 347 26

• Let a be event “process i sends message m to j”• m contains timestamp Tm = Ci[a]• when m arrives at j

(1) IF Tm > Cj THEN Cj Tm(2) event “arrival of m” takes place

IR2

Logicalclocks

CS 347 27

• Let a be event “process i sends message m to j”at physical time t

• m contains timestamp Tm = Ci(t)• Let m be minimum transmission delay for m• m arrives at process j at physical time t’

t’ > t + m • IF Tm + m > Cj-(t’) THEN Cj+ Tm + m • After clock adjust, event “arrival of m” takes place

IR2’

optional

CS 347 28

ij

t

t’

m

m

• IF Tm + m > Cj-(t’) THEN Cj+ Tm + m

•

•

TIME

CS 347 29

Additional properties

Clock driftThere exists a constant k <<1such that for all processes i

dCi(t) dt - 1 < k

PC1

CS 347 30

For typical crystal controlled clocks, k < 10-6

time

Ci

x x+1

y

y+1k

k

y+1+k

y+1-k

dCi(t)

dt - 1 < k

CS 347 31

Clock synchronizationfor all i,j: | Ci(t) - Cj(t) | <

PC2

CS 347 32

can be satisfied if a message is sent on every link at least every seconds

PC2

diameter = d

d=4 (not 3)

••

•

•

••

CS 347 33

let m be the transmission time of message mm = minimum delay + unpredictable delaym = m + Zm

Assume constant such that for all m: Zm Assume that for all m: m =

•

•

t1

t2

maximumtransit

time: +shortest

transit time:

CS 347 34

Theorem PC2 holds for all t > to + d

with d(2k + )

(assuming + << ).

CS 347 35

Argument for d=1 slow fasts f

Worst case scenario:t1: m leaves f for s with Tm = Cf(t1)t2: m arrives at s: Cs(t2) = Cf(t1) +

Cf(t2) - Cs(t2) is as large as possible, t2+ : no communication for sec; f is as fast as

possible, s as slow as possible. Just before communication at t2+ clocks are as far apart as possible,

CS 347 36

t2 +t2

f

s

k

k

time

clockreading

So, = 2 + Now just need largest possible

CS 347 37

•

•

Cf(t1) t1

Cs(t2) t2

f s

+

Cf(t2) = Cf(t1) + (1+k)(+)Cs(t2) = Cf(t1) + = (1+k)( + ) - max = k(+) +

CS 347 38

=2k + max = k(+) + Thus:

= 2 + + k(+) Assuming that k<<1 and (+) << we get: 2 +

CS 347 39

SummaryPhysical clocks:• clock condition a b C(a) <

C(b)• drift < k• | Ci(t) - Cj(t) | < • can be implemented as discrete

CS 347 40

Uses for Physical Clocks:(1) To order events(2) Timeouts

Example: “Reply to my request by time t1”At time t2= (++)(1+) + t1

we can timeoutmy clock may drift

CS 347 41

my node other

t1on my clock

max trans+

t1

at remote clock

t2 = (++)(1+) + t1

CS 347 42

Do physical clocks rule out anomalous behavior?

No anomalous behavior if Cs(t2)>Cf (t1)

min

ffast clock

sslow clock

message

out of system

Cf (t1)

Cs(t2)

Cs(t1)=Cf (t1) -

CS 347 43

Worst possible scenario:• Smallest possible transmission time min

• Cs, Cf as far apart as possible Cs(t1) = Cf (t1) -

• Smallest possible Cs(t2) =Cf (t1) - + S.P.increment =Cf(t1) - + min(1-)

No anomalous behavior if Cs(t2)>Cf (t1)

CS 347 44

• No A.B. if Cs(t2) > Cf(t1)I.e., if Cf(t1) - + min(1-) > Cf(t1)

1 - k

min >

CS 347 45

Using a clock service• With IR2’, a fast clock speeds up all clocks• Use instead a single, reliable clock service

(eg. WWV), assume it is perfect “real time” (k=0)• IR2’ is now: Cj+(t’)Tm+min whenever timing message arrives• If max. transmission time from central clock is

< + then k(1 + k(Why? )

CS 347 46

Using a clock service - analysis

0 error at synchronization time, node j node j server

Cj=+

Cj=+ ++

+

time

slowestclock message

fastestclock message

CS 347 47

Using a clock service - analysis - cont.

elapsed time

servertime

0

Node that got fastestmessage

Node that got slowest

message time

possibledifference

slow clock

fast clock

CS 347 48

Why can period be ?

+

t

t +

serverj

fast

slow

CS 347 49

elapsed time

servertime

0



time

k(

k

worst casefor clock

reset

CS 347 50

servertime

0



time

k(

k

worst casefor clock

reset

|Cs(t) - Cj(t)| |Ck(t) - Cj(t)|

2

CS 347 51

Setting clocks back

Cj(t)

timetimingmsg

arrives

cannotuse

othersolutions?

CS 347 52

How can we reset a clock after a failure?

CS 347 53

Probabilistic clock synchronization

|Cj(t) - Ck(t)| 2kk

Cs

k = drift=update period =max. transmission variability

CS 347 54

Probabilistic clock synchronization

|Cj(t) - Ck(t)| 2kk

Cs

k = drift=update period =max. transmission variability

% ofmessages

transmission time

CS 347 55

Idea #1 When site j wants to synchronize, it requests time from server

T2v

CjT++v/2

server

time

jassume k =0

CS 347 56


T2v

CjT++v/2

server

time

j

T++v

T

assume k =0

CS 347 57


T2v

CjT++v/2

server

time

j

T++v

T

T+

T

assume k =0

CS 347 58


T2v

CjT++v/2

server

time

j

T++v

T

T+

T

|Cs(t) - Cj(t)| v/2assume k =0

CS 347 59

• If we are lucky, v will be small and we get tight synchronization

• If we are unlucky, v is large and we lose!!

|Cs(t) - Cj(t)| v/2

CS 347 60

Idea #2(a) pick desired desired bound, D(b) repeat request until v/2 D(c) if more than q requests, we die!

CS 347 61

area=p

2 2D

p.d.f ofround trip delay

Prob. 1 request fails = Pr[roundtrip time > 2D]= p

Prob. q requests fail = pq

Prob. We can synchronize = 1 - pq

Choose q so that it is “very likely” that algorithm works...

CS 347 62

• If k = 0, we are done– step 1: all nodes synchronize– step 2: we stay synchronized forever

• If k > 0, synchronize every seconds

CS 3471 CS 347: Parallel and Distributed Data Management Notes13: Time and Clocks.

Documents

Transcript of CS 3471 CS 347: Parallel and Distributed Data Management Notes13: Time and Clocks.