Failure Recovery of Overlay Tree-based Structures Ing. Vladimír Dynda Doc. RNDr. Ing. Petr...

Failure Recoveryof Overlay Tree-based

Structures

Ing. Vladimír Dynda

Doc. RNDr. Ing. Petr Zemánek, CSc.(supervisor)

Czech Technical University in PragueFaculty of Electrical Engineering

Department of Computer Science and Engineering

Doctoral Thesis

Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures

Agenda

Introduction Solution

BR Platform Bypass Routing Leader Link Election Tree Reconnection

Summary of Results Conclusion


T = (TM, CE)

Introduction

Problem statement

S = (N, L)

TM CE

FC

T1

T0

T2

T3

T4 T5T6

TR = (TM\FC, CE’ )

1


Introduction

Problem statement Failure recovery

Reconnection of T0, T1, ..., TN-1 into a restored network TR= (TM \ FC, CE’)

Correctness – TR is acyclic

Completeness – TR contains all the fragments

2


Introduction

Problem statement Environment

Asynchronous distributed system No central authority / no global knowledge Unlimited sizes of S and T Arbitrary traffic direction in T

Failures Node failures only Fail stop failure model Failures must not split S

3


Introduction

Goals of the thesis Proposal of a generic recovery platform Illustration of the tree restoration

methods Simulation & verification of the

theoretical properties Survey of possible applications

4


Introduction

State of the art On-demand / preplanned recovery Preplanned methods

Employ pre-computed backup structures

Existing preplanned methods Complete graph (Narada) Ancestor list (Yang-Fei, EFTMRP, HMTP) Administrative hierarchy (Nice, Nemo) Secondary trees (Dual-tree, Coop-net) Link to random nodes (HMTP, Yoid)

5


Introduction

State of the art Weaknesses of the existing methods

Poor scalability Restricted set of applicable trees Single points of failure Fixed level of fault tolerance Unrecoverable multiple failures Non-local restoration

6


Agenda





BR Platform

Bypass ring platform Ensures correctness and completeness Forms a basis for a tree reconnection Fabric of redundant links in T:

Bypass rings of optional diameter Alternative paths in the event of failure Location & routing among the fragments

7


BR Platform

Failure recovery

T = (TM, CE)

FC


BC(FC)Leader link electionTree reconnection

Leadern1

n2

n1

BRT(n1,2)BRT(n1,3)

BRT(n1,4)

n2

BRT(n2,2)

Bypass routingBypass rings

8


BR Platform

Elemental steps of the recovery1. Initialization of the platform2. Failure detection3. Designated nodes discovery4. Leader link election5. Tree reconnection6. Bypass rings reconfiguration

Bypass

ro

uti

ng

Correctness Completeness&

9


Agenda





Bypass Routing

Partially ordered tree (POT)

T = (TM, CE)

A0

09

67

93

B9

CE E8

1D

5E42

F711

3C

72

17

B2

0F79

9F

24

4A

SeqT(A0)

SeqT(3C)

R +(A0,3C)

R -(A0,3C)

BT(A0,3C)

R +(A0,3C)

Ordered neighborsequenceOrdered rays

10


BRT(n,dmax)

Bypass Routing

Bypass ring BRT(n, d)

n

R-(n,n0)

n0n1

n2n3

R +(n,n1)

BRT(n,2)

BRT(n,3)

BRT(n,4) BT(n,n1)

BT(n,n2)

BT(n,n3)

BT(n,n0)

dmax = 4

SeqT(n)

R -(n,n1)

R +(n,n2)

R -(n,n2)

R +(n,n3)

R -(n,n3)

R +(n,n0)

11


Bypass Routing

Bypass rings

BT(n,n

1)

n n1

n2

n3

n4

n5

ndmax

R + (n,n 1

)

BRT(n1,2)BRT(n1,3)

BRT(n2,4)BRT(n2,5)

BRT(nm,dmax)

T = (TM, CE)

FC

12


Bypass Routing

Routing algorithm <FC>T = BT(ni, nj), nj AT(ni) FC

( )i Tn A FC

FC

R +(ni1,nj1)

BC(FC)

B T(n

i2,n

j2)

BT (n

i3 ,nj3 )

BT(ni1,nj1)

T = (TM, CE)ni3

nj3

ni1

nj1

ni2

nj2

13


Bypass routing

Example

T = (TM, CE)

FC

BC(FC)

A0

09

67

93

B9

CEE8

1D

5E42

F711

3C

72

17

B2

0F79

9F

24

4A

R +(72,3C)BRT(3C,2)

BRT(3C,3)BRT(A0,4)

14


Bypass Routing

Properties Memory overhead at node n T:

O(degT(n) * dmax) Routing is successful if

lenX(ni, ni+1) dmax, X = R+(ni, nj)

for all neighbors ni and ni+1 BC(FC) Lower bound of maximum size of FC:

dmax/2 nodes for arbitrary clusters

15


Agenda





Leader Link Election

Leader link election (LLE) Guarantees correctness Communication structure – BC(FC) Node states

Passive – initial state of the election Active – leader candidates Relay – election is lost

16



LLE on ordered rings

FC

n0

n1

n2

n3

n4

n5

n6

nN-1

ID(n0) < ID(n1) < ... < ID(nN-1)

ELECTION(n0)

ID(n0) < ID(n1)

ID(n1) < ID(n2)

ELECTION(n1)

ID(nN-1) < ID(n0)

BC(FC) = BRT(n,2)

n

SeqT(n)

Leader

<FC AT(FC)>

17


HIDT(4F,16)


LLE in partially ordered trees

nr

FC

SeqT(nr)

BC(FC)

<FC AT(FC)>

Leader

Hierarchical identifierHIDT(nr,ni)

4F

97

D8

16

4F.A1.BA.D8

4F.A1.BA.97

4F.A1.16

HIDT(4F,D8)

HIDT(4F,97)A1

BA

R+

Sweep process

ELECTION(A1.BA.97)

SeqT(A1) A1.BA < A1.16

ELECTION(4F.*)

SWEEP(4F.A1)

18



Example

T = (TM, CE)

FC

A009

B9

CE

E8

1D

3C

72

B2

0F 67

93

5E42

F711

17

79

9F

24

4A<FC

AT(FC)>

ELECTION(A0.B9.CE)

nr

A0.B9 < A0.1D

ELECTION(3C.A0.1D)

3C.A0 < 3C.A0

nr

SWEEP(3C.A0)

Leader

19



Properties Average message complexity:

O(N logb N); b is the average branching factor of FC nodes in T

Time complexity: O(N)

20


Agenda





Tree Reconnection

Reconnection methods Reconnect the fragments located by the

routing algorithm Abide by the results of LLE Designed to meet the specific

application requirements Influence properties of the restored tree

21


Tree Reconnection

LR method

n1

n3

n2

BC(FC)

22


q0 =

= q0

Tree Reconnection

HR-x method

n1

n3

n2

BC(FC)

= q0

q1

q2

= q3

q1

q2

q5 =

q3

q4

q1

q2

q3

(q0, qi) if i 1 (mod x)(qi-1, qi) otherwise

HR-1

23


Tree Reconnection

HR-x method

n1

n3

n2

BC(FC)

(q0, qi) if i 1 (mod x)(qi-1, qi) otherwise

HR-2

24


Tree Reconnection

Example

FC

A009

B9

CE

E8

1D

3C

72

B2

0F 67

93

5E42

F711

17

79

9F

24

4A<FC

AT(FC)>

ELECTION(A0.B9.CE)

ELECTION(3C.A0.1D)

SWEEP(3C.A0)


HR-2

25


Tree Reconnection

Properties

26


Tree Reconnection

Properties

27


Agenda





Summary of Results

Properties of the BR platform Node memory overhead:

O(degT(n) * dmax)

Average message complexity: O(N logb N) for arbitrary failures N for single failures

Lower bound of max. recoverable failure: dmax/2 nodes for arbitrary failed clusters

dmax-1 nodes for internal failed clusters

28


Summary of Results

Simulation results Successfully recovered cluster

Average diameter: dmax-2

Average size: 1.5 dmax

Linear recovery time dmax parameter

Controls fault-tolerance vs. costs dmax=4 provides ample tolerance for GFS

29


Summary of Results

Properties of the platform Locality Multiple failure recovery Scalability Application requirements consideration

Optional level of fault tolerance Protection selectivity Designated nodes discovery Tree reconnection method

Independence of the protected tree type30


Summary of Results

Applications Overlay multicast

Applicable in all types

Network-layer multicast Extension with BR(n,1) needed

Sample application – GFS multicast Designed for large-scale P2P systems Based on a layered administrative hierarchy Employs BR platform to achieve fault-

tolerance

31


Agenda





Conclusion

Thesis summary Analysis of overlay trees environment

and identification of recovery properties Proposal of BR platform Design of the specialized leader election Illustration of the tree reconnection Simulation of the platform Outline of the overlay multicast scheme

32


Conclusion

Ideas for further research Autonomous management of fault-

tolerance level and protection selectivity More sophisticated tree reconnection

methods Extension of the platform for

network-layer multicast

33

Thank You

Failure Recovery of Overlay Tree-based Structures Ing. Vladimír Dynda Doc. RNDr. Ing. Petr...

Documents

Transcript of Failure Recovery of Overlay Tree-based Structures Ing. Vladimír Dynda Doc. RNDr. Ing. Petr...