Multicast Survivability in Hierarchical Broadcast Networks

Multicast Survivability in Hierarchical Broadcast Networks

Azad AzadmaneshUniversity of Nebraska

Computer Science DepartmentOmaha, Nebraska 68182

[email protected]

Axel KringsUniversity of Idaho

Computer Science DepartmentMoscow, Idaho 83844

[email protected]

Daryush LaqabUniversity of Nebraska

Computer Science DepartmentOmaha, Nebraska 68182

[email protected]

Abstract: Despite the increasing number of applications that benefit from multicasting, most reliable multicastprotocols consider omission faults only. This research is concerned with survivability of multicast communicationwhere the communication medium is shared among the hosts. The approach presented specifies that a networkwith N nodes is resistant against network omissions and malicious nodes if N ≥ 2t + b + 2, where t and b are thenumber of malicious and benign faulty hosts, respectively. The simulation results show that the network traffic isdynamically adapted to the rate of faults and that the network performs well under omission and malicious faults,unless the rate of faults is high. Furthermore, it is shown that each additional network segment requires (2tg + 1)gateways, where tg indicates the number of faulty gateways.

Key–Words: Byzantine agreement, Hybrid fault models, Multicast, Peer-to-Peer networks, Ad-hoc networks

1 Introduction

In multicast communication, a message is sent to agroup of receivers. Some applications of multicast-ing are in multiplayer computer games, webcasting,audio/video-conferencing, process coordination, androuting in ad-hoc networks [3, 10, 14, 17, 21, 22, 24].Multicast communication has been used over the In-ternet where the interconnection is inherently point-to-point. It has also been used in multi-point com-munication media such as Ethernet. Although, thereexists a fair amount of research in the area of multi-casting, most of the research is focused on omissionfaults. An omission occurs when a node expectinga message does not receive it, e.g., due to link fail-ure, crash of the transmitting node, or if the messageis corrupted during transmission and not correctableby the ECC component. As the research with respectto malicious behavior or a mixture of failure types islimited, the main goal is to ensure the multicast pro-tocol designed not only tolerates omission faults, butit also tolerates malicious behavior. Different failuremodes and their impact on network fault tolerance andnetwork performance will be investigated. As this re-search is concerned with general broadcasts, the prin-ciples discussed here are applicable to all broadcastenvironments, including wireless networking such asad-hoc and sensor networks.

A multicast is considered reliable when all cor-rectly operating nodes either deliver a message multi-casted or none of them deliver it if at least one node

fails to deliver the message. Here a distinction is madebetween receiving and delivering a message. A nodemight receive a message but not deliver the messageor there might be a delay between receiving and deliv-ering a message. This delay is caused by the receivingoverhead such as queuing time, ECC process, and themulticasting overhead to ensure multicast propertiesare met.

This research uses the concept of acknowledg-ment(ACK) and negative acknowledgment (NACK)handshaking by nodes, such as in the Trans protocol[15]. In the Trans protocol, each node communicateswith others through broadcasting messages. Once anode receives a message, it piggybacks an ACK forthe received message to its next outgoing message.Should a node determine it has not received a mes-sage, the node adds a NACK for the missing mes-sage to the very next outgoing message. This enablesthe network to be robust against omission faults. Thenodes can also construct empty messages just for thepurpose of sending a list of ACKs or NACKs.

The Trans protocol is essentially an error detec-tion and error recovery protocol. It depends on thephysical layer to put the outgoing messages on the busand to pick up the incoming messages from it. Theprotocol does not deal with delivering of messages tothe application layer. Therefore, it is primarily a datalink layer protocol intended for detecting and recover-ing from omission faults over a bus medium.

The authors of Trans have also designed anotherprotocol, called Total protocol [16], which places a

WSEAS TRANSACTIONS on COMMUNICATIONS ISSN: 1109-2742 663

Azad Azadmanesh, Axel Krings, Daryush Laqab Issue 6, Volume 7, June 2008

total order on the delivery of messages. The Totalprotocol was later enhanced to handle Byzantine fail-ures [18]. A Byzantine fault is not constrained by anyassumption regarding its behavior. For example, aByzantine faulty node is capable of transmitting con-flicting messages to the receiving nodes. To removethe impact of Byzantine behavior, the protocol as-sumes the underlying multicast protocol uses digitalsignatures to identify the source of a message and todistinguish among different messages.

The rest of this study is organized into the fol-lowing sections. Section 2 touches upon some back-ground and introductory information. Sections 3 and4 are concerned with a single shared medium, suchas an Ethernet segment. Section 3 focuses on theapproach and design of a reliable multicast protocol.The section also discusses the fault types that can beendured by the protocol. Section 4 shows simulationresults based on the protocol design discussed in theprevious section. Section 5 sheds some light on therequirements to extend a single segment network intomulti-level multicast segments. A summary of thework is given in Section 6.

2 Background and DefinitionsAn efficient multicast technique conserves bandwidthby eliminating unnecessary duplication of messages.Messages branch-out at routers only when it is neededto reach receivers on different paths. Multicast proto-cols can be categorized into inter-domain and intra-domain protocols. Inter-domain multicast protocolsare used when the receivers are not on the samenetwork. Therefore, inter-domain multicast requiresthe assistance of the network layer of intermediaterouters to forward the messages. Multicast Back-bone (MBone) [8] is an inter-domain multicast pro-tocol that provides delivery of messages to a groupof receivers across an inter-network. MBone is com-monly used for broadcasting video across the Inter-net. Border Gate Protocol (BGP), which is in usewidely, is a robust inter-domain routing protocol re-sponsible for exchanging routing information. Multi-cast BGP (MBGP) [4] is an extension of BGP to en-able inter-domain multicasting and multicast routingpolicy. Intra-domain protocols are designed for LANprotocols, such as Ethernet, that operate at the datalink layer. The Internet authority has reserved the ad-dresses in the range 224.0.0.0 to 224.0.0.255 for mul-ticasting on a LAN segment. No router would forwardthese messages with such addresses, as the Time-to-Live (TTL) is set to 1. The addresses 224.0.1.0through 238.255.255.255 are reserved as global mul-ticast addresses used for message multicasting across

the Internet. One major challenge with implementinga protocol is managing the multicast group member-ship. During the multicast process, it is possible fora given set of nodes to join the multicast group, or toleave the group. Internet Group Management Proto-col (IGMP) is a protocol that allows routers to keeptrack of group membership [9]

2.1 Fault ToleranceFault Tolerance is referred to the ability of a givensystem to function properly and accurately in the pres-ence of faults. This is possible when some redundancyis introduced to the system. The degree of redun-dancy, which can be a mixture of hardware, software,and time, is a design time decision based on factorssuch as the number and types of faults. The mask-ing or detection of failure is usually done by increas-ing the amount of communication among the nodesand imposing an agreement algorithm within the sys-tem. The objective of the agreement algorithm is toensure that every non-faulty node agrees on the valueof the message that is going to be delivered to the ap-plication (user). The best known agreement algorithmis the Byzantine General Problem (BGP) [11, 20],wherein all the receiving nodes need to decide on avalue broadcast by a single node. The authors demon-strate that agreement is guaranteed if the number offaulty nodes t that behave in Byzantine manner is lessthan one-third of the total number of nodes N , i.e.,N > 3t.

Although there are many different types of faults,the behavior of a fault can be generalized into one ofthe following categories [1, 23, 25]:

• Benign - These faults share the attribute of be-ing obvious and self incrementing to every node.For example, when a value that is broadcasted isoutside of an expected range. Often, this class offaults is the least complex to accommodate for.

• Symmetric - In contrast to benign faults, the er-rors caused by symmetric faults may not be dis-tinguished from legitimate ones. These faultscan be in transmissive or omissive form. Morespecifically, in a transmissive symmetric (TS) be-havior, the same erroneous message is receivedby all receivers, e.g., a faulty node broadcastsa wrong time-of-day. In the omissive symmet-ric (OS) behavior, no message is received. Forexample, a message becomes corrupted in trans-mission in such a way that it is detectable andthus discarded by every correct node. Thesefaults are more common in multi-point networks.

• Asymmetric - These faults are not constrainedby any assumption regarding their behavior.



These faults can be partitioned into multi-valuedasymmetric (MVA), strictly omissive asymmetric(SOA), and single error asymmetric (SEA). In aMVA fault, also called a Byzantine fault, the re-ceiving nodes perceive conflicting messages withregard to the same message. These faults arecommon in point-to-point networks and difficultto achieve in multi-point networks [2]. When aSOA fault occurs, each correct node receives ei-ther a single correct message or no message atall. This behavior often occurs in multi-pointnetworks, where a single correct message broad-casted is not received by some nodes. Finally, inSEA behavior, the same dubious message trans-mitted is received by at least one node but not byall of them. This mode of behavior, for exam-ple, can occur in a broadcast environment wherea malicious message is lost in transit and so it isnot received by some receivers.

2.2 Multicast CommunicationApproaches to multicast protocols can be viewed as:1) distributed versus centralized, 2) tree-based, or 3)ring-based. These protocols are not necessarily exclu-sive of each other. In a centralized control, a node hasthe responsibility of making sure the messages are re-ceived by all the group members. The group memberssend their messages to the central node before beingbroadcast to all members. In a distributed approach,all members share the responsibility of delivery andmaking sure everyone receives the broadcast message.The advantage of the distributed approach is the faulttolerance aspect of the protocol in that no single pointof failure exists. In the tree-based protocols, the hostsare divided into groups, and each group is headed by asingle leader. This way, the group members are likelyto be leaders of other groups. The main advantage ofsuch protocols is the reduction in message acknowl-edgments sent by the receivers, as each group leaderis responsible only for one group and not for the entirereceivers. Finally, the premise of ring-based protocolsis to support atomic and total ordering for messagestransmitted. For instance a site that has the token is al-lowed to send its messages to the group members, andis responsible for making sure all group members re-ceive its messages. By having the token pass throughthe group members in a predetermined order providestotal message order [6, 10, 13].

In general, there are four categories of multicastprotocols with regards to message ordering [5]:

• Unordered delivery, in which the multicast pro-tocol does not make any guarantees about the or-der in which the messages are accepted by differ-

ent hosts.

• FIFO-ordered delivery requires the multicastprotocol to deliver incoming messages from agiven host in the same order in which the hostoriginally sent the messages.

• Causally-ordered delivery means that the mes-sages are delivered to the upper layers while pre-serving the causality between the messages. Inother words, if in a given multicast group, mes-sage m1 causes (precedes) m2, and m2 causesm3, all receiving hosts will deliver m1 beforem2, and m2 before m3.

• Totally-ordered delivery means that the messagesare delivered in the same order to the upper layersby all nodes.

3 Network Design and ProtocolAt times, broadcast and multicast of messages can beused interchangeably. However when needed, distinc-tion between the two forms of message transmissionwill be made to remove any confusion. We assume thefollowing:

A1. Unordered-delivery of messages is assumed.Each message consists of a header and a pay-load. The header contains information such asan identifier consisting of the node identifierand the sequence number of the message. Thesequence numbers distinguish among messagescoming from the same node. To further distin-guish among messages with the same sequencenumbers being transmitted from different nodes,each sequence number is pre-pended with thenode identifier. The payload contains informa-tion like data, ACK list, NACK list, and rebroad-cast information to distinguish the message froma message broadcast for the first time.

A2. A message broadcast by a non-faulty node is re-ceived by at least t other non-faulty nodes. Asno trust can be placed on the faulty nodes, a non-faulty node must leave a trace of the messageamong other non-faulty nodes. This will be fur-ther explained in Section 4.

A3. A node cannot impersonate another node. Thatis, a faulty node cannot create a message and pre-tend that it originated from a different node. Fur-thermore, the sequence numbers for new mes-sages are generated in hardware, so that a nodewould not be able to generate two new messageswith the same sequence number.



A4. A receiving node can not determine whether amessage broadcast for the first time is from afaulty or a non-faulty node.

The following conditions are referred to as Relia-bility Conditions (RCs):

• Agreement: A message delivered by a correctnode is eventually delivered by every correctnode.

• Integrity: A message is delivered only once byevery correct node.

• Validity: A message broadcast by a correct nodewill eventually be delivered by the node.

The inclusion of the Validity condition may seemunnecessary as a node always delivers its own mes-sage. However, it is necessary to ensure that this con-dition in combination with the Agreement conditionguarantees the delivery of the correct messages by allcorrect nodes.

3.1 The Fault ModelMost multicast protocols assume message omissionsare due to accidental failures. However, a faulty nodecan create confusion among hosts, or even behaveasymmetrically. The following shows a series of sce-narios that a malicious host can take advantage of:

A. A malicious host sends a NACK for a messagethat it has already received. In this scenario, amalicious node adds to its negative acknowledg-ment list a NACK for a message that it has al-ready received. The faulty node will then appendthe list to the next outgoing message. This sce-nario impersonates a legitimate message omis-sion, as it will cause the non-faulty nodes to re-broadcast the message that they think is not re-ceived by a node. If it happens frequently, it candegrade network performance, causing a situa-tion similar to the ”denial of service” attack. Thenon-faulty nodes would then spend more time onbroadcasting unnecessary messages.

B. A malicious host sends a NACK for a messagethat has not yet been sent. This scenario willcause the non-faulty nodes to also NACK the un-sent message. As the message was never sent,it will never be received by any of the non-faulty nodes. These negative acknowledgmentswill cause the non-faulty nodes to continuouslyNACK the message. For this scenario to occur,the message sequence number must be higherthan that of the last message sent by the same

node. Consequently the unsent message wouldbe NACKed until a message with that sequencenumber is transmitted. Since a NACK identifiesthe original source of the message, it is unwise toask the originator to intervene, as the originatormight be one of the faulty nodes in the system.

C. A malicious host sends an ACK for a messagethat it has already acknowledged. This scenariowill not cause harm to the correct operation of thenetwork, since these ACKs will be ignored by allnon-faulty nodes that have received the message.

D. A malicious host sends an ACK for a messagethat has not yet been sent. This scenario occurswhen a malicious node adds to its positive ac-knowledgment list an ACK for a message thatwas never sent previously. The fallacious posi-tive ACK will then be piggybacked to the nextoutgoing message sent by the malicious node.This scenario causes the non-faulty nodes tocontinuously NACK the unsent message, whichmight degrade the network performance if thereis a large gap between the sequence number ofthe last message sent and the sequence numberof the unsent message for which an ACK is trans-mitted by the faulty node.

E. A malicious host alters a message before it re-broadcasts the message. This scenario can oc-cur when a malicious node receives a NACK.Whether the faulty node has received the mes-sage, it can broadcast a falsified message in re-sponse to the NACK. Consequently, the non-faulty node that sent the NACK cannot determinewhich copy of the messages rebroadcast by othernodes is the original message.

Scenario C would not violate the RC. The im-pact of scenarios B and D are the same, in that thenon-faulty nodes will continuously NACK the unsentmessage. Therefore, items B and D require the samesolution, and a solution to either item is a solution tothe other one. Capitalizing on different malicious be-havior discussed so far, the scenarios A - E can becondensed into the following cases:

M1. A malicious node sends a NACK continuouslyfor a message that has already been sent, with theintent to flood the network with messages, caus-ing performance breakdown. This will addressscenario A discussed above.

M2. A malicious node NACKs a message that wasnever sent, with the intent to cause other nodes toalso negatively acknowledge the unsent message



continuously. This will address scenarios B andD discussed above.

M3. A malicious node rebroadcasts a modified mes-sage or a manufactured message, pretending thatthe message was received from another node,with the intent of causing the hosts to accept thewrong message or creating discrepancies in mes-sage content received by different hosts. This isscenario E discussed above.

In addition to the three categories of malicious be-havior discussed above (M1-M3), a Byzantine nodecan behave omissively, i.e., it can behave OS by notsending messages at times. A faulty node can be per-ceived to have behaved in an SEA manner as well, inthat some of its messages are lost in transmission. Ad-ditionally, the transmission medium might be able tobehave in SOA. For instance, if a message transmittedby a correct node is not received by some nodes.

For a MVA error to occur, a Byzantine node mustbe able to transmit a message in more than one form tomultiple receivers, or a message must be transformedinto multiple, undetectable forms during transmission,due to noise or hardware failure. Both faulty be-haviors are difficult to achieve, if not impossible, ina shared medium such as Ethernet [2], and thus arenot considered in this research. However, these faultsneed serious consideration in ultra-dependable sys-tems, such as flight-control systems, which have afailure rate requirement in the order of at least 10−9

[1, 7, 19].We assume, however, that Byzantine failure at the

protocol level is still a possibility. A faulty node canbehave as in M3 at different times, to create inconsis-tency among different nodes with respect to a specificmessage. For example, a faulty node can broadcasta different form of a message any time it receives aNACK for the same message identifier.

Based on the previous discussion, the behaviorof the t faulty nodes can be perceived as: TS, OS,SEA, or MVA at the protocol level. In addition, thebenign faulty nodes are recognized globally and thusare removed beforehand. Therefore, n = N − b isthe number of nodes that participate in the multicastprocess, where b is the total number of Benign faultynodes. It will be shown that the RCs are guaranteed ifN ≥ 2t+b+2, where N is the number of nodes in thesystem and t is the maximum number of faulty nodes.Furthermore, it will be shown that the RCs are pre-served under scenarios M1-M3, which in turn implythat scenarios A - E are covered.

3.2 The Protocol ApproachThe following rules guarantee the RCs:

1. A non-faulty node accepts a message transmit-ted for the first time. Hence, a node receiving amessage with a new identifier can not determinewhether the message originated from a maliciousnode (See A4). A corrupted, not correctable mes-sage is simply discarded.

2. In response to NACKing a message, a non-faultynode accepts a copy of the message if (t+1) iden-tical rebroadcasts of the message are receivedfrom different nodes. The (t + 1) identical mes-sages ensure that at least one of the rebroadcastshas emanated from a non-faulty node. This fur-ther ensures that NACKing by a legitimate nodein response to an ACK or NACK by a faulty nodedoes not result into accepting a bogus message.This addresses cases M2 and M3.

3. A message is ignored if (n− t− 1) NACKs fromdifferent nodes are received for the same mes-sage. Therefore, if faulty nodes refrain from pro-viding a NACK, a non-faulty node needs to re-ceive a NACK from every other non-faulty node.To further limit the overhead caused by a ma-licious behavior, a bound can be placed on thenumber of rebroadcasts for each node or a mes-sage. This covers case M1.

If a message is received by some correct nodes,every non-faulty node will eventually receive (t + 1)identical rebroadcasts of the message. Therefore, theAgreement and Validity conditions will be guaran-teed. As each new message contains a different se-quence number and each correct node can keep trackof the identifier of the last message delivered, the In-tegrity condition will also be preserved. Furthermore,the requirement of (n − t − 1) NACKs attempts tominimize network traffic and in the worst case ensuresnetwork liveness, by preventing the non-faulty nodesto continuously engage in transmitting a NACK mes-sage. A node will stop sending a NACK if it eitherreceives (t+1) identical messages, thus accepting themessage, or it will ignore the message if it receives(n− t− 1) NACKs from different nodes.

Implicit in the above discussion is the fact thatno non-faulty node will relay a message to anothernode, unless it accepts the message first, as the resultof receiving (t + 1) identical messages. Otherwise, anon-faulty node might relay a dubious message, andthus contribute to the total (t + 1) needed by anothernon-faulty node to accept the message.

In response to a NACK, since it is possible that all(t+1) rebroadcasts are from the non-faulty nodes andthe t faulty nodes refrain from broadcasting, the totalnumber of nodes in the system must be n ≥ (t+1)+t.Counting the number of benign faults and the node



itself receiving (t + 1) identical messages from othernodes, the lower bound on the total number of nodesis N ≥ 2t+b+2. This also implies that there must beat least two non-faulty nodes in the system to ensurethe RCs are meaningful.

4 Protocol SimulationA simulation program has been developed to test theprotocol and examine the network operation perfor-mance [12]. The program is able to handle M1 - M3,through the behaviors in the form of SO, SOA, TS,and the protocol level TA as described in Subsection3.1. The simulation runs show that the RCs can not beguaranteed if n ≥ 2t + 2 is not satisfied.

If a node does not receive a message due to anomission, it is highly likely that the message will bereceived during one of the rebroadcasts of the mes-sage. Obviously, the lower the omission probability,the sooner the message will be received. If the omis-sion probability for each node is p, the probability thata message is not received by a specific node after rbroadcasts is pr. Fig. 1 displays this probability for rranging from 1 to 20, and p ranging from 0.1 to 0.9.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 3 5 7 9 11 13 15 17 19Broadcasts

Mes

sage

Om

issi

on P

rob. 0.1

0.20.30.40.50.60.70.80.9

xFig. 1: Probability of a message not received after bbroadcasts.

As the figure shows, the number of rebroadcastsdecreases sharply as the omission probably is reduced.At around 0.50 omission probability, a message is re-ceived within the first 10 rebroadcasts. To better un-derstand the effect of omissions, the simulation pro-gram is run with 10 nodes, 10 messages per node,ranging omission probabilities from 10% to 90%, andm broadcasts of each message, ranging m from 1 to30. For each value of m, one broadcast and (m − 1)rebroadcasts are done to all nodes. An original mes-sage is marked as “omitted” if there exists a node thatdoes not receive the message in m attempts. There-fore, the minimum and maximum number of originalmessages omitted can be 0 and 100 respectively. Foreach combination of omission probability and broad-casts, the program is run 5 times. The total number oforiginal messages omitted are then averaged over the

5 runs. Although the results of one simulation mightbe somewhat different from another, Fig. 2 shows con-sistency with Fig. 1 in the fast tendency of messagesbeing received when the omission probability is lowor moderate. Hence, there would not be a real need tocreate extra overhead caused by an agreement processfor each new message, which can cumulatively be toohigh. Each node can independently deliver a messagereceived, knowing that other nodes would eventuallydo the same. Note that Fig. 1 is with respect to a singleparticular node, whereas the messages not received inFig. 2 is with respect to 10 nodes.

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 25 30 35m

Aver

age

Orig

inal

Mes

sage

s O

mitt

ed

0.10.20.30.40.50.60.70.80.9Linear (0.1)Linear (0.2)Linear (0.3)Linear (0.4)Linear (0.5)Linear (0.6)Linear (0.7)Linear (0.8)Linear (0.9)

Fig.2: Downward trend of messages not received.

The next simulation run shows the average num-ber of non-faulty nodes that receive the original broad-cast of a message. The program is run with t setto 2 for a network of 10 nodes. Each node is set tobroadcast 10 messages, and the omission probabilityis ranged from 0.10 to 0.90. For each omission prob-ability, the program is executed 10 times. As a result,a total of 1000 messages is collected for each value ofomission probability. The total of messages are thenaveraged to show the average number of nodes receiv-ing a message on the first broadcast. Fig. 3 shows, for

0

1

2

3

4

5

6

7

10 20 30 40 50 60 70 80 90Omission Probability

Avg.

Num

ber o

f Non

-faul

ty N

odes

Rec

eivi

ng F

irst

Bro

adca

st o

f a M

essa

ge

Fig.3: Average Number of non-faulty nodes receivingfirst broadcast of a message.

omission probability as high as 0.80, that at least onenon-faulty node receives a message. For 0.90 omis-sion probability, the average number of non-faulty



nodes receiving a message is 0.76. Although, the con-dition that a message broadcast for the first time mustbe received by at least one non-faulty is necessary, thefollowing discussion shows that this condition is notsufficient to guarantee the RCs.

An interesting question is to obtain the total num-ber of broadcasts to deliver the messages originatedfrom the network nodes in the presence of omissionand malicious faults. A related question is to find theoverhead in broadcast per message. A simulation isrun for a network of 20 nodes, each broadcasting 10messages. The number of malicious nodes, which arecapable of modifying messages, ranges from 0 to 9and the omission probability of the network is set to0.10. Thus, there is a total of 200 original messagesfor each configuration. The program is then run 10times, for a total of 2000 messages. Fig. 4 shows thetotal number of messages sent between the nodes un-til all non-faulty nodes accept and deliver all the mes-sages.

0

2000

4000

6000

8000

10000

12000

14000

0 1 2 3 4 5 6 7 8 9t

Tota

l Num

ber o

f Mes

sage

s Se

nt

Fig.4: Total number of messages sent to deliver 2000messages.

Fig. 5 shows the average message overhead peroriginal message for the previous experiment. As-sume T is the total number of messages sent, and Mis the number of original messages in the network.Then:

Overhead per message =T −M

M=

T

M− 1

This figure shows that the maximum average over-head occurs when t = 4, with T ≈ 12000 andM = 2000. The average message overhead is about(0 + 5)/2 = 2.5. Furthermore, the figure shows thatmessage overhead is not much when the total numberof faulty nodes is moderate, that is when the number

of faulty nodes is less than half of what the networkcan endure to operate properly. However, one shouldobserve that this is a simulation, so every run mightyield a different result. To be close to reality, the av-erage overhead was found over 10 different configu-rations.

0 0.1455

1.15

0.2565

5.058

0.8545

2.9643.429 3.539 3.4545

0

1

2

3

4

5

6

0 1 2 3 4 5 6 7 8 9t

Aver

age

Ove

rhea

d pe

r Mes

sage

Fig. 5: Average number of extra messages broadcastto deliver an original message.

Consider a situation where n = 10, t = 2, and amessage broadcast by a non-faulty node A is receivedby only one other non-faulty node B. Node B willACK the message causing other non-faulty nodes toNACK the message. At this point, other non-faultynodes would not be able to receive (t + 1) = 3 iden-tical messages, as there are only two non-faulty nodeshaving a copy of the message. Additionally, a non-faulty node C that did not receive the message mightnot be able to receive (n− t− 1) = 7 NACKs to dis-card the message. As there are 6 non-faulty nodes thatdid not receive the message, if the faulty nodes decidenot to send in a NACK, then C would receive only 6NACKs. As a result, node C would not be able to ac-cept the message, and would not be able to discard themessage either. A similar problem might occur if themessage is originally broadcast by a faulty node andreceived by less than (t + 1) non-faulty nodes. Con-sequently, to avoid this situation, it is assumed thata new broadcast must be received by at least (t + 1)non-faulty nodes. Fig. 3 depicts the result of an exper-iment with the same value of n = 10 and t = 2. Thefigure shows that with omission probability as high as0.5, on average at least (t + 1) = 3 non-faulty nodesreceive the original messages.

Consider a non-faulty node that broadcasts a mes-sage that needs to be received by at least t nodes of theremaining (n − t − 1) non-faulty nodes. Assume theomission probability is p. The probability that exactly



j non-faulty nodes do not receive the message is:(n− t− 1

j

)p j (1− p) (n−t−1)−j

Therefore, the probability that at least t non-faultynodes will receive the message is:

(n−t−1)−t∑i=0

(n− t− 1

i

)p i (1− p) (n−t−1)−i

Fig. 6 shows this probability graphically for p rangingfrom 0.1 to 0.5, with t set to 2 and increasing n startingat the minimum value 6. The figure signifies that theprobability of at least t non-faulty nodes receiving abroadcast approaches 1 quickly as number of nodes isincreased, especially when the omission probability pis not high. Also the graph shows consistency with thesimulation results in Fig. 3 at n = 10.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20n

Prob

. >=

t Nod

es R

ecei

ve th

e M

essa

ge

0.10.20.30.40.5

Fig.6: The probability that at least t non-faulty nodesreceive the first broadcast of a message.

5 Hierarchical MulticastingUp to this point the multicasting paradigm was limitedto only include nodes within a single network. Nowwe consider multiple networks, connected by gate-ways.

5.1 Simple connected networks

Fig. 7 shows a network consisting of two subnetworksA and B that are connected by gateways. There area total of ng gateways directly connected to the twoneighboring subnetworks. As in the previous discus-sion, each subnetwork uses a single shared medium.Two multicast scenarios are now possible. Either themulticast spans over a single subnet or it spans overboth subnets, i.e., the multicast includes nodes in net-works A and B. Since single networks have been al-ready discussed in the previous sections, the focus willbe on the latter scenario.

Let’s consider the connectivity of the two subnets.If ng = 1, then all communication between A and Bpasses though the same gateway. In configurationswith ng > 1, the gateways operate in a redundantmode where each gateway forwards messages to allsubnetworks that are directly connected to and havenodes participating in the multicast. Note that this isdifferent from standard network configurations whereonly one gateway usually forwards packets, i.e., theconfiguration builds a spanning tree and multiple gate-ways play only a role if an active gateway in the span-ning tree fails.

GatewaysNet A Net B

Fig.7: Configuration with two networks

Consider the case where the subnetworks A andB communicate via a single gateway, i.e., ng = 1.Furthermore assume that a message broadcast orig-inates on A and the multicast membership includesnodes in both subnets. Since the gateway is the onlypoint of connection between the two subnetworks, thegateway will rebroadcast the message onto subnet B.However, the nodes in subnet B will not be able to re-ceive the required (t + 1) identical copies of the samemessage from different nodes to accept the message.Additionally, the gateway can not forward the mes-sage onto B as a new one, since the message did notoriginate from the gateway. Furthermore, if the gate-ways are treated as standard network nodes, the re-quirement to receive (t + 1) identical copies wouldrequire the number of gateways to be at least (2t + 1).However, this large number of gateways most likely isnot practical for t > 2.

Therefore, to incorporate multiple subnetworksconnected by gateways, the fault model needs to beadapted. Up to this point we considered t to be thenumber of faulty nodes. However, it seems reasonableto treat standard network nodes and gateways differ-ently with respect to failure probability. It can be ar-gued that a gateway is more reliable, or less prone tofailure, due to the fact that it is not used as a generalpurpose computer and it is a special piece of hardwarewith dedicated software.

Let na and nb denote the number of nodes in net-work segment A and B that participate in the multi-cast respectively, i.e., n = na + nb. The n nodes arein addition to the ng gateways. Furthermore, let tndenote the number of faulty nodes in all subnetworks



and let tg be the number of faulty gateways betweenthe two subnets, with the failure probability of a gate-way to be much less than that of a network node, i.e.,tg � tn.

It can be shown that to tolerate the failure of tnnodes and tg gateways, one needs

n ≥ 2tn + 2

nodes andng ≥ 2tg + 1

gateways, where no assumption is made about the dis-tribution of the faulty nodes in the different subnet-works. Hence, the bound for N is the same as that inthe single network case, i.e., N = na + nb + b.

The criterion for accepting a message via thebroadcast paradigm across the gateways is different.Since authentication is assumed, i.e., no gateway canimpersonate to be a different node, each receivingnode can recognize whether a message is broadcastby a gateway. Therefore, a simple majority mes-sage forwards by the non-faulty gateways is sufficient.Consequently, a receiving node can vote on a cor-rect message using a (2tg + 1) majority of receivedmessages. This voting effectively constitutes gatewayfault masking and can be viewed as a restoring organ.

The overhead resulting from sending a singlemessage to another network is of factor ng plus theoverhead associated with voting. Note that this onlyapplies to messages sent across subnetworks and notto messages sent to the same subnetwork that containsthe originating node. Furthermore, since the failurerate of a gateways is assumed significantly lower thanthat of a network node, tg and thus ng will be small.For example, tg is likely to be equal to 1 or 2 resultingin 3 or 5 gateways, respectively.

A multicast network can be represented by a spe-cial kind of network graph in which the vertices arenodes or gateways and the edges are logical networkconnections. Given two graphs Gi and Gj with re-spective vertex sets Vi and Vj and edge sets Ei andEj , the union G = Gi ∪ Gj has V = Vi ∪ Vj andE = Ei ∪ Ej . The join G = Gi + Gj consists ofGi∪Gj together with all edges joining Vi and Vj , i.e.,∀vp ∈ Vi and ∀vq ∈ Vj , ep,q ∈ E, the edge set of thejoin graph. Fig. 8 shows an example of a join graph.The edges defining the join operation between G1 andG2 are shown as dashed lines. Note that the edge con-nectivity is logical, i.e., in the context of broadcasting,if a vertex emits a message, all connected vertices re-ceive the message (in the error-free case). This shouldnot be confused with point-to-point messages, wherevertices emit messages to neighboring vertices on oneedge at a time.

34

25

1

34

25

1

+ =G1

G2

G1 + G2

Fig.8: Join operation (+) of two graphs

S

Net A Net BGateways

Fig.9: General join graph of two subnetworks

Let’s revisit the requirement that a new broad-cast must be received by at least tn other non-faultynodes in the context of multiple subnetworks. Thefact that these nodes may now span over two neighbor-ing networks has no consequence other than the over-head associated with sending (2tg + 1) messages foreach message that needs to cross gateways. As longas no more than tg gateways are affected by failureor malicious act, each node which receives forward-ing messages from gateways will be able to determinewhether it should accept the message, i.e., if (tn + 1)identical messages are received, or discard the mes-sage, i.e., if (n− tn − 1) NACKs are received.

The join graph associated with Fig. 7 is shown inFig. 9. Node S broadcasts to the nodes in the twosubnetworks, shown in dark gray, and the gatewaysare depicted in the light-gray shaded oval. Formally,node S broadcasts on subnetwork A and to the gate-ways. The gateways, by the nature of the join opera-tion, forward the messages to all participating nodesin B, which vote on the messages received from thegateways. The additional overhead associated withthe gateways is captured by the join operation of thetwo subnetworks, when a new message is broadcastedfor the first time. Specifically, a new message broad-casted onto subnetwork A needs to be rebroadcastedby at most ng gateways. But, the message overheaddue to rebroadcasts crossing the gateways is less. Thisis because the number of gateways is small, and thus(tg + 1) identical copies are needed to accept a mes-sage in comparison to (tn+1) needed by the nodes onthe originating subnetwork. For example, three gate-ways can mask one failure, which is a relatively smallprice to pay.



5.2 General hierarchical multicasting

Whereas the previous subsection considered the spe-cial case of two subnetworks connected by gateways,this subsection is concerned with general networkconfigurations. Consider a scenario with ns net-work segments that are interconnected with gateways.Fig. 10 shows a configuration with ns = 4 and de-

S

Net A Net BGateways

Net C

SharedGateways

Net D

Fig.10: General join graph with multiple networks

picts the possible scenarios in which subnetworks canbe connected with gateways. The segments A and Bare connected via a group of gateways, which are notconnected to any other networks. On the other hand,networks A, C, and D share gateways. In fact thegateways connecting A and D are a subset of the gate-ways connecting A and C. The network model of theprevious subsection can now be extended to multiplesubnetworks. The following cases can be enumerated,where the associated network interconnections are in-dicated in parenthesis:

1. Two subnetworks are interconnected by non-shared gateways. This is the scenario for subnet-work (A, B), where the number of non-sharedgateways is ng(A,B). In order to mask tg(A,B)

gateway faults, at least (2tg(A,B) + 1) gatewaysare needed.

2. Multiple subnetworks share gateways. Thisis the case involving subnetworks A, C, andD. To mask tg(A,C) gateway faults, a total ofng(A,C) ≥ 2tg(A,C) + 1 gateways are needed.Similarly, a total of ng(A,D) ≥ 2tg(A,D) +1 gate-ways are needed to mask tg(A,D) gateway faults.However, with respect to ng(A,C) no more thantg(A,D) of the shared gateways (depicted by thedarker shaded oval) may fail, and the remain-ing (tg(A,C) − tg(A,D)) faults must occur on non-shared gateways (drawn in a lighter shade of grayin the oval).

The discussion above assumed subnetworks to beonly one hop away from the initial sending node. Mul-

tiple hops are considered in Fig. 11. There is no as-sumption about the distribution of the tn faulty nodes,e.g., they could be concentrated in specific subnets ordistributed randomly over all subnets. In the first case,multi-hop messages are forwarded from one group ofgateways to the next and due to the majority argumentof (2tg + 1) messages, there will always be a majorityof correct messages relayed to the next hop.

S

Net A Net BGatewaysA,B

Net CGatewaysB,C

Fig.11: General join graph with multiple hops

Considering ng gateways between any two neigh-boring subnetworks and ns segments, so that n =n1 + n2 + . . . + nns , the bounds in the previoussubsection regarding the two subnetworks still holdfor the ns segments, i.e., N ≥ 2tn + b + 2 andng ≥ 2tg + 1. Similarly, the overhead associatedwith a multicast spanning over multiple ns subnets isinduced by the message traffic of the redundant gate-ways, i.e., (2tg +1) messages per subnet hop, and it isbound by a total of nsng = ns(2tg + 1) gateways. Ifgateways are shared, as shown in Fig. 10 for networksA, C, and D, the number of gateways is reduced bythe number of shared gateways.

6 ConclusionUsing a broadcast medium, it is difficult or unlikelythat messages can be received asymmetrically, but ithas been shown that an asymmetric behavior is stillpossible by allowing a faulty node to broadcast differ-ent messages for a given original message at differenttimes. It has been shown that (tn + 1) identical mes-sages are needed to deliver a message if the messageis not received in the first broadcast, and (n− tn − 1)NACKs are needed to ignore a message. The researchdid not discuss transmission of messages by the useof digital signatures. If messages are digitally signed,the receipt of only one digitally signed message is suf-ficient to accept the message, instead of (tn +1) iden-tical messages. But there would not be any changes in(n− tn − 1) NACKs that are needed to address M1.

The protocol assumes that a faulty node can be-have in OS, TS, and TA manners at the protocol level.In addition, it is assumed that the network mediumcan temporarily experience omissions in the form ofOS or SOA, due to different scenarios such as loss of



a message or not being able to correct errors.It has been shown that to tolerate the failure of tn

nodes and tg gateways, one needs N ≥ 2tn + b + 2nodes, possibly spread over ns subnets, and ng ≥2tg + 1 gateways, connecting any two adjacent sub-nets.

One of the requirements in achieving reliablemulticasting is reaching agreement in delivering amessage. Normally, a Byzantine agreement algorithmis needed to ensure that every non-faulty node agreeson delivering a message. The process of reaching thisagreement for every message will become very pro-hibitive if new messages are transmitted continuously,or if a point-to-point topology were used. The pro-posed protocol reduces message complexity, as eachnode can independently determine the safety of de-livering a message to the user. Furthermore, simu-lation has shown message complexity is dynamicallyadjusted according to the level of network fault con-dition, whilst the correct operation of the network ismaintained.

References:

[1] M.H. Azadmanesh, R.M. Kieckhafer, ExploitingOmissive Faults in Synchronous ApproximateAgreement, IEEE Transactions on Computers,Vol.49, No.1010, 2000, pp. 1031-1042.

[2] O. Babaoglu, R. Drummond, Streets of Byzan-tium: Network Architectures for Fast ReliableBroadcasts, IEEE Transactions on Software En-gineering, Vol.SE-11, No.6, 1985, pp. 546-554.

[3] F. Barsotti, A. Caruso, S. Chessa, The LocalizedVehicular Multiast Middleware: A Frameworkfor Ad Hoc Inter-Vehicles Multiast Communica-tions, WSEAS Transactions on Communications,Vol.5, No.9, 2006, pp. 1763-1768.

[4] T. Bates, Multiprotocol Extensions for BGP-4, Network Working Group, RFC 2858,http://faqs.org/rfcs/rfc2858.html, 2000.

[5] K. Birman, T. Joasph, “Communication Sup-port for Reliable Distributed Computing”, Lec-ture Notes in Computer Science, Vol.448, 1987,pp. 124-137.

[6] X. Defago, A. Schiper, P. Urban, Total OrderBroadcast and Multiast Algorithms, ACM Com-puting Surveys, Vol.36, No.4, 2004, pp. 372-421.

[7] K. Driscoll, B. Hall, H. Sivencrona, P. Zum-steg, Byzantine Fault Tolerance, From Theoryto Reality, Lecture Notes in Computer Science(LNCS), Computer Safety, Reliability, and Se-curity, Vol.2788, 2003, pp. 235-248.

[8] H. Eriksson, MBone: The Multicast Backbone,CACM, Vol.37, No.8, 1994, pp. 54-60.

[9] W. Fenner, Internet Group Management Pro-tocol, Network Working Group, RFC 2236,http://faqs.org/rfcs/rfc2236.html, 1997.

[10] K.P. Kihlstrom, The SecureRing Group Commu-nication, ACM Transactions on Information andSystem Security, Vol.4, No.4, 2001, pp. 371-406.

[11] L. Lamport, et al, The Byzantine Generals Prob-lem, ACM TOPLAS, Vol.4, No.3, 1982, pp. 382-401.

[12] D. Laqab, Survivable Multicast Communicationin Bus-based Networks, MS Thesis, ComputerScience Department, University of Nebraska-Omaha, 2006.

[13] B.N. Levine, J.J. Garcia-Luna-Aceves, A Com-parison of Reliable Multicast Protocols, Multi-media Systems, Vol.6, No.5, 1998, pp. 334-348.

[14] X. Li, M.H. Ammar, S. Paul, Video Multicastover the Internet, IEEE Network, Vol.13, No.2,1999, pp. 46-60.

[15] P.M. Melliar-Smith, L.E. Moser, Trans: AReliable Broadcast Protocol, IEE Proceedings,Vol.140, No.6, 1993, pp. 481-493.

[16] P.M. Melliar-Smith, L.E. Moser, V. Agrawala,Broadcast Protocols for Distributed Systems,IEEE Transactions on Distributed and ParallelSystems, Vol.1, No.1, 1990, pp. 17-25.

[17] C.K. Miller, Multicast Networking and Applica-tions, Addison-Wesley, 1999.

[18] L.E. Moser, P.M. Melliar-Smith, Byzantine-Resistant Total Ordering Algorithms, Informa-tion and Computation, Vol.150, No.1, 1999, pp.75-111.

[19] M. Paulitsch, J. Morris, B. Hall, K. Driscoll, E.Latronico, P. Koopman, Coverage and the Use ofCyclic Redundancy Codes in Ultra-DependableSystems, Proceedings of the International Con-ference on Dependable Systems and Networks(DSN), 2005, pp. 346-355.

[20] M. Pease, et al, Reaching Agreement in the Pres-ence of Faults, JACM, Vol.27, No.2, 1980, pp.228-234.

[21] C. Shih, T. Shih, Cluster-based Mulitcast Rout-ing Protocol for MANET, WSEAS Transactionson Computers, Vol.6, No.3, 2007, pp. 566-572.

[22] H. Shin, K. Cho, A IP Multicast Technique forthe IPTV Service, WSEAS Transactions on Com-munications, Vol.6, No.1, 2007, pp. 274-277.

[23] P.M. Thambidurai, Y.K. Park, Interactive Con-sistency with Multiple Failure Modes, Proceed-ings of the 7th Reliable Distributed SystemsSymposium, 1988, pp. 93-100.



[24] B. Wang, C. Hou, A Survey on Multicast Rout-ing and its QoS Extension: Problems, Algo-rithms, and Protocols, IEEE Network, Vol.14,No.1, 2000, pp. 22-36.

[25] P.J. Weber, Dynamic Reduction Algorithms for

Fault Tolerant Convergent Voting with HybridFaults, PhD Dissertation, Electrical & ComputerEngineering, Michigan Technological Univer-sity, 2006.



Multicast Survivability in Hierarchical Broadcast Networks

Documents

Transcript of Multicast Survivability in Hierarchical Broadcast Networks