Simulating mobile ad hoc networks: a quantitative...
27
Simulating mobile ad hoc networks: a quantitative evaluation of common MANET simulation models Calvin Newport Dartmouth College Computer Science Technical Report TR2004-504 June 16, 2004 Abstract Because it is difficult and costly to conduct real-world mo- bile ad hoc network experiments, researchers commonly rely on computer simulation to evaluate their routing pro- tocols. However, simulation is far from perfect. A grow- ing number of studies indicate that simulated results can be dramatically affected by several sensitive simulation parameters. It is also commonly noted that most simu- lation models make simplifying assumptions about radio behavior. This situation casts doubt on the reliability and applicability of many ad hoc network simulation results. In this study, we begin with a large outdoor routing ex- periment testing the performance of four popular ad hoc algorithms (AODV, APRL, ODMRP, and STARA). We present a detailed comparative analysis of these four im- plementations. Then, using the outdoor results as a base- line of reality, we disprove a set of common assump- tions used in simulation design, and quantify the impact of these assumptions on simulated results. We also more specifically validate a group of popular radio models with our real-world data, and explore the sensitivity of various simulation parameters in predicting accurate results. We close with a series of specific recommendations for simu- lation and ad hoc routing protocol designers. 1 Introduction It is difficult to perform an accurate evaluation of a mo- bile ad hoc network (MANET). In a perfect world, ad hoc network protocols would always be validated with exten- sive real world experimentation. The best way to predict the behavior of a network is to deploy it in a real environ- ment, and then observe what happens. For obvious rea- sons, however, such experimentation is rarely done. As Zhang et al. point out, “running MANET systems in a non-trivial size is costly due to high complexity, required hardware resources, and inability [to test] a wide range of mobility scenarios” [ZL02]. This difficulty is supported by our own anecdotal experience. Conducting the outdoor experiment reported in this study required over two years of preparation by a team of more than ten researchers and student interns. In fact, Zhang et al. report that their lit- erature search uncovered only a “few” real world systems that have ever been implemented, and none that have been tried on a scale beyond a dozen nodes [ZL02]. Our review of the MANET literature, two years later, confirms this observation. Because of this difficulty of running real world experi- ments, it is clear that, at least for now, computer simula- tion will remain the standard for ad hoc network evalua- tion. However, this reliance on simulation demands in re- turn a careful scrutiny of common simulation approaches. For simulation to be used as a meaningful evaluation tech- nique, there must be a concerted effort to understand the models being used—including their specific characteris- tics, and their relative validity. This conclusion is sup- ported by an increasing body of research that demon- strates that the outcome of wireless network simulation is quite sensitive to the underlying models. For example, in an experiment conducted by Takai et al. [TMB01], it is shown that altering parameters in commonly used ra- dio models has a non-uniform effect on ad hoc protocol behavior, sometimes even reversing the relative ranking among protocols tested in the same scenario. The simu- lation results reported in this study similarly demonstrate dramatic changes in outcomes when different radio mod- els are used (see Section 4.5). And a recent article in IEEE Communications warns that “An opinion is spreading that one cannot rely on the majority of the published results on performance evaluation studies of telecommunication networks based on stochastic simulation, since they lack credibility” [PJL02]. It then proceeds to survey 2200 pub- lished network simulation results to point out systemic flaws. We of course do not suggest that there is one right an- swer to the question of simulation validity. Accordingly, 1
Transcript of Simulating mobile ad hoc networks: a quantitative...
Simulating mobile ad hoc networks: a quantitative evaluation ofcommon MANET simulation models
Calvin Newport
Dartmouth College Computer Science Technical Report TR2004-504
June 16, 2004
Abstract
Because it is difficult and costly to conduct real-world mo-bile ad hoc network experiments, researchers commonlyrely on computer simulation to evaluate their routing pro-tocols. However, simulation is far from perfect. A grow-ing number of studies indicate that simulated results canbe dramatically affected by several sensitive simulationparameters. It is also commonly noted that most simu-lation models make simplifying assumptions about radiobehavior. This situation casts doubt on the reliability andapplicability of many ad hoc network simulation results.
In this study, we begin with a large outdoor routing ex-periment testing the performance of four popular ad hocalgorithms (AODV, APRL, ODMRP, and STARA). Wepresent a detailed comparative analysis of these four im-plementations. Then, using the outdoor results as a base-line of reality, we disprove a set of common assump-tions used in simulation design, and quantify the impactof these assumptions on simulated results. We also morespecifically validate a group of popular radio models withour real-world data, and explore the sensitivity of varioussimulation parameters in predicting accurate results. Weclose with a series of specific recommendations for simu-lation and ad hoc routing protocol designers.
1 Introduction
It is difficult to perform an accurate evaluation of a mo-bile ad hoc network (MANET). In a perfect world, ad hocnetwork protocols would always be validated with exten-sive real world experimentation. The best way to predictthe behavior of a network is to deploy it in a real environ-ment, and then observe what happens. For obvious rea-sons, however, such experimentation is rarely done. AsZhang et al. point out, “running MANET systems in anon-trivial size is costly due to high complexity, requiredhardware resources, and inability [to test] a wide range of
mobility scenarios” [ZL02]. This difficulty is supportedby our own anecdotal experience. Conducting the outdoorexperiment reported in this study required over two yearsof preparation by a team of more than ten researchers andstudent interns. In fact, Zhang et al. report that their lit-erature search uncovered only a “few” real world systemsthat haveeverbeen implemented, and none that have beentried on a scale beyond a dozen nodes [ZL02]. Our reviewof the MANET literature, two years later, confirms thisobservation.
Because of this difficulty of running real world experi-ments, it is clear that, at least for now, computer simula-tion will remain the standard for ad hoc network evalua-tion. However, this reliance on simulation demands in re-turn a careful scrutiny of common simulation approaches.For simulation to be used as a meaningful evaluation tech-nique, there must be a concerted effort to understand themodels being used—including their specific characteris-tics, and their relative validity. This conclusion is sup-ported by an increasing body of research that demon-strates that the outcome of wireless network simulationis quite sensitive to the underlying models. For example,in an experiment conducted by Takai et al. [TMB01], itis shown that altering parameters in commonly used ra-dio models has a non-uniform effect on ad hoc protocolbehavior, sometimes even reversing the relative rankingamong protocols tested in the same scenario. The simu-lation results reported in this study similarly demonstratedramatic changes in outcomes when different radio mod-els are used (see Section 4.5). And a recent article inIEEECommunicationswarns that “An opinion is spreading thatone cannot rely on the majority of the published resultson performance evaluation studies of telecommunicationnetworks based on stochastic simulation, since they lackcredibility” [PJL02]. It then proceeds to survey 2200 pub-lished network simulation results to point out systemicflaws.
We of course do not suggest that there is oneright an-swer to the question of simulation validity. Accordingly,
1
we do not attempt to identify any oneright model that al-ways performs best. But we do, however, argue that simu-lation designers should explicitly address the assumptionsmade in their models, and the influence that these assump-tions may have on simulation results. We believe thatthis approach will allow the MANET community to confi-dently find relevance for simulation outcomes beyond thespecific simulator configuration in which a particular ex-periment was run.
With this in mind, we identify several specific ques-tions that simulation designers should consider and striveto answer to ensure that their results are as meaningful aspossible:
• What assumptions are made by the radio propagationmodel?
• How realistic are these assumptions?
• What is the effect of these assumptions on the re-sults?
• Has the radio model been validated with experimen-tal data, and if so, how does it perform relative toreality? For example, does the model tend to pre-dict a higher rate of network connectivity than whatis observed under experimental conditions? Does itexaggerate the maximum range of the network’s ra-dios? What is the significance of these variations forunderstanding the simulation results?
• What simulation parameters are used? How sensitiveare the results to changes in these parameters? Andhow do the values used constrain the applicability ofthe results?
In this study, we detail the necessity of these questions,and use real experimental data combined with a wirelessnetwork simulator to explore answers as they apply to agroup of commonly used radio propagation models. Morespecifically, we a) describe in detail the implementationand results of a large outdoor MANET routing experi-ment; b) identify the extent to which most ad hoc networkresearchers make common simplifying assumptions aboutthe radio model used in simulation; c) use experimentaldata to quantitatively demonstrate that these assumptionsare far from realistic; d) explore how these assumptionsmay lead to misleading results in ad hoc network simula-tion; e) validate the predictive power of our selected mod-els against an experimental baseline; f) explore the roleof important parameters in simulation results; and g) listrecommendations for the designers of protocols, models,and simulators.
The results described in this study span over two yearsof work, including one published paper [LYN+04], one
technical report [KNE03] (with a revision currently un-der conference submission), and another conference pa-per in preparation that describes the large-scale outdoorMANET routing experiment, and analyzes the perfor-mance data. This thesis is the first complete synthesisof these various experiments into one comprehensive ex-amination of accuracy in ad hoc network simulation. Wehope that our use of real experimental results to groundour simulation analysis will make this work particularlyuseful for simulation designers, and provide an importantcontribution to the growing field of ad hoc networking re-search.
2 Outdoor routing experiment
As mentioned above, few MANET researchers conductreal-world experiments. The cost and complexity are pro-hibitive for most projects. In this section, however, wethrow caution to the wind, and join a team that is up to thechallenge of testing four popular ad hoc routing protocolsin a dynamic outdoor environment. Specifically, researchengineer Robert Gray organized a real-world routing ex-periment as part of a larger multi-disciplinary universityresearch initiative led by Dartmouth Professor George Cy-benko.1 Gray worked on the scenario design with SusanMcGrath, Eileen Entin and Lisa Shay, and the algorithmswere implemented by Aaron Fiske, Chris Masone, NikitaDubrovsky, and Michael DeRosa. Our role in this projectis to gather and organize the data produced by the experi-ment and then provide a detailed comparative analysis ofthe results.
This description of real network behavior, in addition tobeing a stand-alone contribution to the research commu-nity, also forms an empirical baseline that aids our subse-quent validation of ad hoc network simulation.
2.1 The algorithms
The outdoor experiment tests four algorithms. APRL,which stands for Any-Path Routing without Loops, isa proactive distance-vector routing protocol [KK98].Rather than using sequence numbers, APRL uses pingmessages before establishing new routes to guaranteeloop-free operation. AODV, or Ad-hoc On-Demand Vec-tor, is an on-demand routing algorithm—routes are cre-ated as needed at connection establishment and main-tained thereafter to deal with link breakage [PR99].ODMRP stands for the On-Demand Multicast RoutingProtocol [LGC02]. For each multicast group, ODMRPmaintains a mesh, instead of a tree, for alternate and re-dundant routes. ODMRP does not depend on another uni-cast routing protocol and, in fact, can be used for uni-
1http://actcomm.dartmouth.edu
2
cast routing. STARA, the System and Traffic DependentAdaptive Routing Algorithm, is based on shortest-pathrouting [GK97]. It uses mean transmission delay insteadof hop count as the distance measure.
2.2 The experiment
The outdoor routing experiment took place on a rectan-gular athletic field measuring approximately 225 (north-south) by 365 (east-west) meters. This field can beroughly divided into four flat, equal-sized sections, threeof which are at the same altitude, and one of which is ap-proximately four to six meters lower. There was a short,steep slope between the upper and lower sections.
Each Linux laptop2 had a wireless card3 operating inpeer-to-peer mode at 2 Mb/s. This fixed rate made itmuch easier to conduct the experiment, since it obviatedthe need to track (and later model) automatic changes toeach card’s transmission rate.
To reduce interference from the campus wireless net-work, the experiment was conducted on a field physicallydistant from campus, and the cards were configured touse wireless channel 9 for maximum separation from thestandard channels (1, 6 and 11). In addition, each lap-top collected signal-strength statistics for each receivedpacket.4 Finally, each laptop had a Garmin eTrex GPSunit attached via the serial port. These GPS units did nothave differential GPS capabilities, but were accurate towithin thirty feet during the experiment.
Each laptop recorded its current position (latitude, lon-gitude and altitude) once per second, synchronizing itsclock with the GPS clock to provide sub-second, albeitnot millisecond, time synchronization. Every three sec-onds, thebeacon service programon each laptopbroad-casta beacon packet containing the current laptop posi-tion (as well as the last known positions of the other lap-tops). Each laptop that received such a beacon updatedits internal position table, and sent aunicast acknowledg-mentto the beacon sender via UDP. Each laptop recordedall incoming and outgoing beacons and acknowledgmentsin another log file. The beacons provide a continuous pic-ture of network connectivity, and, fortunately, also repre-sent network traffic that would be exchanged in many real
2A Gateway Solo 9300 running Linux kernel version 2.2.19 withPCMCIA Card Manager version 3.2.4.
3We used a Lucent (Orinoco) Wavelan Turbo Gold 802.11b. Al-though these cards can transmit at different bit rates and can auto-adjustthis bit rate depending on the observed signal-to-noise ratio, we used anad hoc mode in which the transmission rate was fixed at 2 Mb/s. Specif-ically we used firmware version 4.32 and the proprietary ad hoc “demo”mode originally developed by Lucent. Although the demo mode hasbeen deprecated in favor of the IEEE 802.11b defined IBSS, we used theLucent proprietary mode to ensure consistency with a series of ad hocrouting experiments of which this outdoor experiment was the culminat-ing event.
4We used thewvlan cs , rather than theorinoco cs , driver.
MANET applications. Finally, every second each laptopqueried the wireless driver to obtain the signal strength ofthe most recent packetreceivedfrom every other laptop,and recorded this signal strength information in a thirdlog.5 Querying every second for all signal strengths wasmuch more efficient than querying for individual signalstrengths after each received packet.
These three logs provide all the data that we need tocompare the performance of the four routing algorithms.The laptops automatically ran each routing algorithm for15 minutes, generating random UDP data traffic for thir-teen out of the fifteen minutes, and pausing for two min-utes between each algorithm to handle cleanup and setupchores. The traffic-generation parameters were set to pro-duce the traffic volumes observed in previously exploredprototype situational-awareness applications [Gra00], ap-proximately 423 outgoing bytes (including UDP, IP andEthernet headers) per laptop per second, a relatively mod-est traffic volume. The routing algorithms produce addi-tional traffic to discover or maintain routing information.Note that each transmitted data packet was destined foronly a single recipient, reducing ODMRP to the unicastcase.
Finally, the laptops moved continuously. At the start ofthe experiment, the participants were divided into equal-sized groups of ten each, each participant given a laptop,and each group instructed to randomly disburse in one ofthe four sections of the field (three upper and one lower).The participants then walked continuously, always pick-ing a section different than the one in which they werecurrently located, picking a random position within thatsection, walking to that position in a straight line, and thenrepeating. This approach was chosen since it was sim-ple, but still provided continuous movement to which therouting algorithms could react, as well as similar spatialdistributions across each algorithm.
During the experiment, seven laptops generated no net-work traffic due to hardware and configuration issues, andan eighth laptop generated the position beacons only forthe first half of the experiment. We use the data from theremaining thirty-two laptops, although when we simulatelater, we use thirty-three laptops since only seven laptopsgenerated no network traffic at all. In addition, STARAgenerated an overwhelming amount of control traffic, andthough we exclude the STARA portion of the experimentfrom later analysis of radio behavior, we still present itsdata in the outdoor results sections that follow. The reasonwe exclude it later is because its unusual behavior makesit a poor reference point for the specific task of validatingsimulation results.
5For readers familiar with Linux wireless services, note that we in-creased the IWSPY limit from 8 to 64 nodes, so that we could capturesignal-strength information for the full set of laptops.
3
2.3 The results
To evaluate the relative performance of these algorithmswe use the following four metrics:message delivery ra-tio, communication efficiency, hop count, and end-to-endlatency. Combined, these measures provide a good un-derstanding of the various factors involved in the differentobserved behaviors. In the sections that follow, we give adetailed definition of each metric, and compare the resultsfor each algorithm.
Before proceeding, there are several important termsused in our analysis that must first be defined:
• A messageis a group of dummy bytes produced bya node’s traffic generator for intended transportationto a randomly-selected destination. All of our gen-erated messages are small enough to fit into a singledata packet.6
• A data packetis any transmitted packet containingmessage data. Therefore, everymessagerequires thetransmission of at least onedata packetto reach itsdestination. A message also can generate more thanone data packet, depending on the route length, andthe delivery strategy of the algorithm. It also is pos-sible for a message to generatenodata packets, if thesending node fails to identify any active route towardthe message’s destination.
• A control packetis any transmitted packet that doesnot contain message data. Control packets are themeans by which most algorithms communicate rout-ing information with nearby nodes.
2.3.1 Message delivery ratio
We calculate the message delivery ratio for each algorithmby dividing the total number of messages received at theirintended destination by the total number of messages gen-erated. This metric measures each algorithm’s overall suc-cess in providing reliable communication.
We note that this metric is typically referred to aspacketdelivery ratioin similar examinations of routing protocolbehavior. In this study, however, we substitute the wordmessagefor packet to keep our terminology consistentwith the precise definitions provided above.
Figure1 shows the message delivery ratio for each ofthe four algorithms. A striking result from this compari-son is the dominance of ODMRP. This high delivery rateis best explained by ODMRP’s aggressive flooding ap-proach to route discovery. Instead of using control packetsto discover message routes, this algorithm floods the net-work with data packets. This greatly increases the chancethat a message will reach its intended destination.
6Each message was approximately 1200 bytes in size, including allrelevant headers.
0
0.2
0.4
0.6
0.8
1
AODV APRL ODMRP STARA
% o
f Mes
sage
s R
each
ing
Des
tinat
ion
Algorithm
Figure 1: Message delivery ratio comparison of all fouralgorithms.
We also note that AODV performs better than APRL.This result indicates the advantage of a reactive approachto route discovery in our mobile scenario. If the nodesin our experiment had been more static, or if the physi-cal environment had otherwise provided less opportunityfor link breakages, it is possible that we would see lessof a gap between AODV’s and APRL’s delivery ratios.In addition, APRL does not minimize a hop count met-ric when choosing a route. Subsequently, its average hopcount (shown in Section 2.3.3 below) for successful mes-sage transmissions is larger than what we see for AODV.This use of longer routes opens up more opportunity fordropped packets, and therefore it also may have loweredAPRL’s performance.
STARA’s message delivery ratio is the worst of thegroup. We attribute the algorithm’s poor performance inour experiment to an excessive amount of control traffic.Its continual probing of the network created overwhelm-ing congestion. Accordingly, we recognize that our imple-mentation of STARA needs additional flow restrictions onthe control traffic to constrain the unchecked propagationof control packets. Piyush Gupta proposes one possiblesolution to this excessive traffic problem by noting thatmultiple copies of an identical control packet arriving ata common node could be condensed into a fewer numberof packets before being rebroadcast [Gup00]. Gupta alsonotes that “extensive simulation study and protocol de-velopment [are] needed to make STARA a viable routingprotocol” [Gup00]. We agree, adding that our experiencewith STARA reinforces the importance of using detailedstochastic simulation to help validate and enhance routingprotocols during the design phase.
2.3.2 Communication efficiency
Figure2 shows, for each algorithm, the average number ofdata packets transmitted for each generated message. Wederive this number by dividing the total number of trans-
4
0
5
10
15
20
25
AODV APRL ODMRP STARA
Dat
a P
acke
ts P
er M
essa
ge
Algorithm
Figure 2: Average number of data packets transmitted foreach generated message.
mitted data packets by the total number of generated mes-sages. This metric approximates each algorithm’s datatransportation efficiency.
ODMRP once again dominates the plot. This is not un-expected, as ODMRP floods the network with data pack-ets when trying to locate a route. Accordingly, the numberof data packet transmitted for each message in ODMRP issignificant. If the size of the messages being transmittedis large, then the effect of this data packet load on avail-able bandwidth would be dramatic. In our experiment,however, the generated traffic size was relatively modest,approximately 423 outgoing bytes per laptop per second,allowing ODMRP to avoid excessive congestion. If, onthe other hand, the traffic had been concentrated on fewerdestinations, or if the network had been more static, wewould observe fewer ODMRP data packets as it wouldnot need to flood the network as often.
AODV transmitted 1.32 data packets per message,while APRL transmitted only .90. The difference betweenthese two values, though small in magnitude, is notable.As shown below in Section 2.3.3, APRL’s average hopcount for successfully received messages is larger thanAODV. Therefore, if both algorithms had equally accu-rate routing information, APRL’s data packets per mes-sage value should be larger than AODV, as its routes tendto require more hops. This is not, however, what we ob-serve. APRL’s smaller value of data packets per messageindicates a lack of quality routing information. As we ex-plore in more detail in the next section, APRL had a largenumber of messages dropped at their source (without gen-erating any data packets), because an active route couldnot be identified.
STARA transmitted the fewest data packets, which weattribute to the packet drops due to the congestion createdby the algorithms control traffic.
Figure3 shows, for each algorithm, the average num-ber of control packets transmitted for each generated mes-sage. We derive this number by dividing the total number
0
20
40
60
80
100
120
140
160
AODV APRL ODMRP STARA
Con
trol P
acke
ts P
er M
essa
ge
Algorithm
Figure 3: Average number of control packets transmittedfor each generated message.
Packets Per MessageAODV 7.50APRL 33.30
ODMRP 45.59STARA 150.67
Table 1: The average number of packets transmitted (datapackets and control packets) for each generated message.
of transmitted control packets by the total number of gen-erated messages. This allows us to compare each algo-rithm’s control traffic efficiency.
STARA clearly produced the most control traffic. Itgenerated, on average, 150 control packets for each mes-sage. This overwhelming result evidences the excessivecontrol traffic that we cite as causing STARA’s poor per-formance in our experiment.
APRL generated the next largest amount of control traf-fic, with 32 control packets, on average, for each message.AODV was the most efficient, generating only six con-trol packets, on average, for each message. These resultsdemonstrate that in our scenario, with light traffic and dy-namic connectivity, one of the costs of APRL’s periodicproactive route discovery, as oppose to AODV’s reactiveapproach, is a substantial increase in control traffic.
It is difficult to find comparative significance for theODMRP control traffic result, as control and data packetsare not clearly distinguished for this algorithm. In our ex-periment, we count packets not containing message dataas control packets. For ODMRP this would include onlythe reply traffic generated in response to the algorithm’sflooding of the network with data packets, even though, inmany ways, the flooded data packets are acting the roleof control packets. A more meaningful comparison ofODMRP’s communication efficiency can be found withthetotal packets per messagevalues that we present next.
Table1 shows, for each algorithm, the average number
5
of packets transmitted for each generated message. Thesevalues are calculated by adding the total number of con-trol packets and data packets transmitted, and then divid-ing this sum by the total number of generated messages.This measure approximates the overall communication ef-ficiency of each algorithm.
AODV is clearly the most efficient, requiring, on av-erage, only 7.5 packets for each message. Surprisingly,ODMRP does not fare much worse than APRL. Onemight assume that ODMRP’s aggressive network floodingapproach would lead to a more significant increase in traf-fic costs as compared to APRL’s periodic route advertise-ments. However, ODMRP’s 45.59 packets transmitted foreach message is not overwhelmingly larger than APRL’s33.30. It should, however, be noted that if the size of thedata packets being transmitted is large, APRL would gaina more noticeable lead over ODMRP, as the majority ofAPRL’s traffic is in the form of streamlined control pack-ets,7 whereas ODMRP includes copies of its data pack-ets with much of its traffic. Considering that AODV andODMRP are both reactive algorithms that flood the net-work to discover routes, it is also surprising to note howmany fewer packets per message are required by AODV.This result emphasizes that it is important for protocol de-signers to carefully consider the flow restrictions on theirroute discovery packets. Finally, we note that this exper-iment generated a modest amount of messages. BecauseAPRL’s control traffic should remain constant regardlessof the number of messages being sent, it might gain moreof an efficiency advantage over its reactive counterparts ifthe amount of data traffic was greatly increased.
2.3.3 Hop count
Figure4 shows, for each algorithm, the average numberof hops successfully received messages traveled to reachtheir destination. We limit our sample to successful mes-sages because we are interested in characterizing the typ-ical route selected by each algorithm.
For ODMRP, it is difficult to calculate hop count val-ues because messages can be received at their destinationmultiple times from multiple paths of varying length. Weavoid this problem in this plot by counting only the firstcopy of each message to successfully arrive.
STARA has the lowest average hop count for success-fully received packets. This result, however, is due to theexcessive control traffic congestion which made success-ful packet transmission difficult. In this environment, onlypackets being sent to a neighbor had a good chance of suc-ceeding. Therefore we cannot gain a good understandingof the typical route selected by this algorithm in more for-giving conditions.
7APRL uses only a simple binary indication of whether or not a routeexists in its routing table, leading to small control packets.
0
1
2
3
4
5
AODV APRL ODMRP STARA
Ave
rage
num
ber o
f hop
s
Algorithm
Figure 4: The average number of hops traveled for suc-cessfully received messages.
After STARA, AODV required the next fewest num-ber of hops for successful packets. It is expected thatAODV should have a lower average hop count than APRLor ODMRP, as the former finds routes on demand, andselects for a shorter path, whereas the latter two do notconsider hop count in selecting a route.
In fact, ODMRP has the largest average hop countvalue. Because many of its messages arrive randomlyat their destination during the undirected route discoveryphase, ODMRP is unable to always use the most efficientidentified route.
Figures5–7 provide a distribution of hop count valuesfor each algorithm. Specifically, they show the total num-ber of messages that traveled each number of hops. Theyalso include an independent bar for messages that weresuccessfully received, and an independent bar for mes-sages that failed. This allows a more detailed understand-ing of the relationship between route size and messagedelivery success. They also include a bar for zero hops,which represent failed message that never left the sendinghost, for lack of a route. We omit a detailed ODMRP hopcount distribution because its flooding approach to mes-sage delivery makes it impossible to define a comparablehop count value for failed messages.
Figure5 shows that the vast majority of STARA mes-sages never made it beyond their source node. Success-ful messages are clustered almost entirely in the one-hopbucket, with 513 of 598 total successful messages trav-eling one hop, 73 traveling two, and only 12 success-ful messages traveling any further. The maximum pathlength traveled by any successful message was 7 hops.There were also a small number of failed messages thattraveled unusually long routes before failing. The longestsuch route was 33 hops.
Considering that this algorithm was transmitting, on av-erage, 150 packets for each message, interference likelycaused the large number of 0-hop message failures that weobserve. Interference also limited the ability of the algo-
6
0
1000
2000
3000
4000
5000
6000
7000
8000
0 5 10 15 20 25 30 35
Num
ber o
f Mes
sage
s
Number of Hops
All MessagesFailed Messages
Successful Messages
Figure 5: Distribution of STARA hop count values for allmessages, successful messages, and failed messages.
rithm to maintain reliable routing information. The failedpackets observed to travel over unusually long paths arelikely caused by invalid and looping routes. This is notsuprising, as the congestion generated by STARA wouldmake it difficult for any node to maintain a full set of con-sistently valid routes.
Figure6 shows that APRL also had a significant num-ber of messages fail without leaving their source node.Successful messages are divided almost equally betweenone and two hops, and failed messages decrease regularlyfrom one to four hops in proportion to the decreasingnumber of total messages in each of those buckets. Thelongest path traveled by any message was 12 hops.
The large number of observed 0-hop message fail-ures reveals that the majority of generated messages weredropped because APRL could not identify a valid route tothe desired destination. The implication of this strikingresult is that APRL’s periodic route advertising schemewas unable to consistently maintain adequate routing in-formation in our experiment. While we admit that therewould be many situations in our experiment where a routephysically did not exist between two nodes, the large dif-ference between APRL’s and AODV’s 0-hop failures indi-cates this more serious problem (AODV is shown in Fig-ure 7). It is also interesting to note that failed messagesoutnumber successful messages in the 1-hop bucket. Thisrelationship is the opposite of what we see with AODV.The explanation for this behavior may involve APRL notselecting for shorter routes. With an average route lengthof 2.11 hops (as compared to AODV’s 1.61), APRL cre-ates more opportunity for invalid routes or collisions tocreate failed packet transmissions after one hop.
Figure7 shows that AODV has far fewer 0-hop mes-sage failures. The majority of its successful messagestraveled one hop, but those that traveled two, three, or fourhops significantly outnumber the failed messages in theirrespective buckets. This is especially noticeable in thetwo and three-hop buckets where almostall of the mes-
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Num
ber o
f Mes
sage
s
Number of Hops
All MessagesFailed Messages
Successful Messages
Figure 6: Distribution of APRL hop count values for allmessages, successful messages, and failed messages.
0
500
1000
1500
2000
2500
3000
3500
0 1 2 3 4 5 6 7 8 9
Num
ber o
f Mes
sage
s
Number of Hops
All MessagesFailed Messages
Successful Messages
Figure 7: Distribution of AODV hop count values for allmessages, successful messages, and failed messages.
sages which traveled that distance were successful. Thelongest path traveled by any message was 8 hops.
The implication of these results is that AODV’s on-demand approach to route discovery worked well in ourdynamic environment. When the algorithm could identifya route, which it did much more frequently than APRL,it was subsequently successful in delivering a message toits intended destination. This indicates quality route in-formation, and the advantage of AODV in finding moreone-hop paths than APRL, increasing its overall deliverysuccess, and conserving network resources.
2.3.4 End-to-end latency
Calculating end-to-end latency for ad hoc networks is dif-ficult. The main obstacle is a lack of synchronization be-tween the individual node clocks. This creates a situationin which a comparison of receiver and sender timestampsis not sufficient for generating accurate latency values.
In our experiment we did not run NTP. We made thisdecision to avoid extra computational overhead and band-width usage. Instead, we relied on the GPS units to pro-vide accuracy to our clocks. Specifically, we set each node
7
clock from the GPS units before the experiment, and resetthem from the units every 10 seconds during the exper-iment. Since we required regular GPS queries for otherpurposes (such as tracking node mobility), this approachdid not introduce significant extra overhead, and requiredno bandwidth usage. We found, however, that our nodeclocks still drifted from each other on the order of tens toa few hundreds of milliseconds. We attribute this to de-lays in reading the time from the GPS unit and invokingthe kernel time system calls.
Though relatively small, this clock drift is still signifi-cant as many of our calculated end-to-end latency valuesare within the same order of magnitude. In this study,we introduce a novel approach to better approximate timesynchronization in mobile ad hoc nodes. We take advan-tage of the fact that our nodes were configured to broad-cast a simple beacon at a regular interval (once every 3seconds), which provides us with a convenient set oftimesynchronization events.Specifically, if we want to syn-chronize the clocks between nodeA and nodeB at timet,we analyze our beacon logs to find a beacon that was sentby a third node,C, and that was received by bothA andB near timet. Assuming thatA andB should receive thebroadcast beacon more or less at the same instant, we cancalculate the skew between the two clocks around timetby comparing what time they each receivedC ’s beacon.This concept can be extended to find the clock skew be-tween all node pairs at all times by locating an appropri-ate time synchronization event for each (nodeA, nodeB ,timet) 3-tuple.
To be computationally efficient, however, we approxi-mated this calculation by splitting the duration of each al-gorithm’s run into a group of 13 equally-sized time buck-ets. For each bucket, we calculated the average skew valuefor every pair of nodes. We did so by performing the skewcalculation for every time synchronization event that oc-curred within the bucket’s time range, and then averagingthe skew values generated for each pair.
To subsequently calculate the end-to-end latency for agiven message sent fromA to B, we use the send timeto locate the appropriate bucket, and then use the averageskew value stored for (A, B) in that bucket to synchronizethe clocks. If there are no time synchronization eventsbetweenA andB during the relevant bucket duration, wethrow out the message, and do not include its latency valuein our metric.
We recognize that this approach is only approximatefor several reasons: 1) it is unrealistic to assume thattwo nodesA andB will receive and timestamp a broad-cast beacon at the same instant, as computational factorsunique to each node can affect how long it takes for theevent to actually be logged; 2) in a multi-hop routing en-vironment, it is possible that the sender and the receiverof a given packet are too distant to have recently received
0
0.5
1
1.5
2
2.5
3
3.5
4
AODV APRL ODMRP STARA
1
2
3
Ave
rage
end
-to-e
nd la
tenc
y (s
econ
ds)
Ave
rage
num
ber o
f hop
s
Algorithm
LatencyHop Count
Figure 8: A comparison of average corrected end-to-endlatency, plotted with average hop count for successfulmessages.
the same beacon; and 3) using the average observed timeskew calculated over a given bucket duration is not as ac-curate as always searching out the closest single time syn-chronization event.
Accordingly, we do not present our end-to-end latencyvalues as precise measurements. We do, however, main-tain that our corrected values are more accurate indica-tions of transit time than relying on uncorrected times-tamps. We use them here to provide meaningful insighton matters such as the relative ranking of the algorithms.We leave the refinement of this technique as future work.
Figure 8 shows the average corrected end-to-end la-tency value and the average hop count value for success-ful messages. We find the expected relationship betweenend-to-end latency and hop count. For AODV, APRL, andODMRP, the average end-to-end latency value increasesroughly proportionately to the average hop count value.STARA is an exception because it shows a low averagehop count, and a large average end-to-end latency. Thisabnormality can be explained by the large amount of com-putational overhead generated by the excess amount ofcontrol traffic. The notable volume of control packets inboth the receive and send queues of all nodes could signif-icantly increase the delay between the sender generatinga message and the receiver processing it.
2.4 Conclusions
Any conclusions we draw from this outdoor experimentmust be qualified by the conditions of our particular test-ing environment. A markedly different scenario couldproduce markedly different results. For example, ournodes were highly mobile, and our terrain was non-uniform (though there were few permanent obstructions),leading to a dynamic state of connectivity. This environ-ment may disadvantage an algorithm like APRL that doesnot seek routes on demand. Similarly, our traffic load was
8
relatively light,8 which may have advantaged an aggres-sive data-packet flooding algorithm like ODMRP, whichmay have failed under heavier traffic conditions.
With these qualifications in mind, we present the fol-lowing conclusions:
• AODV is efficient and effective. Though its mes-sage delivery ratio was not as high as ODMRP, itdelivered messages significantly better than APRLand STARA. More importantly, on all measures ofcommunication efficiency, AODV generated by farthe least amount of traffic for each message. Andin terms of route selection, AODV was successful inconsistently finding short paths, giving it the addi-tional advantage of having the lowest average end-to-end latency value. In an environment with limitedbandwidth, or limited energy resources, AODV is agood choice as a provider of low-cost, adaptable, re-liable, and fast communication.
• ODMRP is optimal for specific scenarios, bad forothers. This algorithm generates a lot of overheadtraffic. Its network flooding is bandwidth intensive,and if data packets are large, ODMRP could fail dueto congestion. At the same time, however, it hadthe highest message delivery ratio of all four algo-rithms. This indicates that in a situation in whichbandwidth and energy resources are plentiful, datapackets are small, and communication reliability iscrucial, ODMRP is a good choice.9
• APRL performed poorly in our environment. Itsmessage delivery ratio was low, its overhead waslarge, and it had a substantial percentage of packetsfail at their source. Our results indicate that APRLhad a hard time maintaining reliable routing infor-mation in our relatively dynamic environment. Inany scenario comparable to our experiment, APRLshows no clear advantage over a reactive algorithmsuch as AODV.
• Our STARA implementation emphasizes the im-portance of flow control. In their original paper,Gupta and Kumar validated STARA with a simplestochastic simulation that did not model collision orinterference effects [GK97]. Their analytic valida-tion demonstrated that STARA performs better thanother approaches because of its dynamic avoidanceof highly trafficked routes. Their analysis, however,
8Messages were generated, on average, only once every three sec-onds.
9Because we reduced ODMRP to the unicast case for our experiment,we can not specifically address its effectiveness as a provider of multi-cast communication. We hope, however, that our analysis of its generalcommunication efficiency and reliability can still act as useful guides forthose interested in effective multicast communication.
avoids the reality that would be clear in more detailedsimulation: If control traffic is not carefully con-trolled, it can destabilize the entire network throughexcessive congestion. Gupta later identifies the po-tential for this problem in his PhD thesis, where hebriefly suggests one possible solution [Gup00]. Hegoes on to suggest that more extensive simulationis necessary before the design could be consideredcomplete. We agree, and further recommend thatallprotocol designers integrate more detailed simulationinto their design process so as to more effectively ad-dress necessary practical concerns, such as flow con-trol, in their original protocol specifications.
• Reactive is better than proactive in dynamic envi-ronments. APRL and STARA’s poor performance,as compared to AODV and ODMRP’s relative suc-cess, highlights the general advantage of a reac-tive approach to routing in a dynamic environment.Our analysis of APRL shows an unnecessarily largenumber of messages dropped before leaving theirsource node, and STARA crippled itself with ex-cessive proactive discovery. It is a fair assumptionthat if we had restrained STARA’s control traffic toa reasonable level, it would have faced the samelack of quality routing information demonstrated byAPRL. Similarly, if we had decreased APRL’s routeadvertisement interval to increase the timeliness ofits routing information, it would have suffered froman excess amount of control traffic. This observa-tion underscores the perhaps unresolvable tensionbetween control traffic and message delivery successpresent in proactive algorithms operating in dynamicenvironments: If you make your algorithm efficient,its reliability drops; if you make your algorithm re-liable, its efficiency drops. Reactive approaches areclearly preferable for scenarios with variable connec-tivity.
3 Common assumptions in ad hocnetwork simulation
Now that we have described the behavior of a real ad hocnetwork, we can explore how close simulation comes toreproducing this reality. In this section, we demonstratehow the commonly used theoretical models of radio be-havior are far simpler than reality, and we codify thesesimplifications into a group of six assumptions commonlymade by simulation designers. Using data from our out-door experiment, we prove these six assumptions to befalse.
In the following section, we use a wireless networksimulator, configured to mimic our real world experiment,
9
to quantify and describe the impact of these assumptionson simulation results.
3.1 Radios in theory and practice
The top example in Figure9 provides a simple model ofradio propagation, one that is used in many simulationsof ad hoc networks; contrast it to the bottom example ofa real signal-propagation map, drawn at random from theweb. Measurements from an ad hoc network of BerkeleyMotes demonstrate a similar non-uniform non-circular be-havior [GKW+02]. The simple model is based on Carte-sian distance in an X-Y plane. More realistic models takeinto account antenna height and orientation, terrain andobstacles, surface reflection and absorption, and so forth.
Of course, not every simulation study needs to use themost detailed radio model available, nor explore everyvariation in the wide parameter space afforded by a com-plex model. The level of detail necessary for a given an-alytic or simulation study depends on the characteristicsof the study. The majority of results published to date usethe simple models, however, with no examination of thesensitivity of results to the (often implicit) assumptionsembedded in the model.
There are real risks to protocol designs based on overlysimple models of radio propagation. First, “typical” net-work connectivity graphs look quite different in realitythan they do on a Cartesian grid. An antenna placed topof a hill has direct connectivity with all other nearby ra-dios, for example, an effect that cannot be observed insimulations that represent only flat plains. Second, it isoften difficult in reality to estimate whether or not onehas a functioning radio link between nodes, because sig-nals fluctuate greatly due to mobility and fading as wellas interference. Broadcasts are particularly hard-hit bythis phenomenon as they are not acknowledged in typi-cal radio systems. Protocols that rely on broadcasts (e.g.,beacons) or “snooping” may therefore work significantlyworse in reality than they do in simulation.
Figure10depicts one immediate drawback to the over-simplified model of radio propagation. The three differentmodels in the figure, the Cartesian “Flat Earth” model, athree-dimensional model that includes a single hill, and amodel that includes (absorptive) obstacles, all produce en-tirely different connectivity graphs, even though the nodesare in the same two-dimensional positions. As all thenodes move, the ways in which the connectivity graphchanges over time will be different in each scenario.
Figure11 presents a further level of detail. At the top,we see a node’s trajectory past the theoretical (T) andpractical (P) radio range of another node. Beneath it wesketch the kind of change in link quality we might ex-pect under these two models. The theoretical model (T)gives a simple step function in connectivity: either one
Typical theoretical model
Source: Comgate Engineeringhttp://www.comgate.com/ntdsign/wireless.html
Figure 9: Real radios, such as the one at the bottom, aremore complex than the common theoretical model at thetop. Here different colors, or shades of gray, representdifferent signal qualities.
is connected or one is not. Given a long enough straightsegment in a trajectory, this leads to a low rate of changein link connectivity. As such, this model makes it easy todetermine when two nodes are, or are not, “neighbors” inthe ad hoc network sense.
In the more realistic model (P), the quality of the link islikely to vary rapidly and unpredictably, even when tworadios are nominally “in range.” In these more realis-tic cases, it is by no means easy to determine when twonodes have become neighbors, or when a link betweentwo nodes is no longer usable and should be torn down.In the figure, suppose that a link quality of 50% or betteris sufficient to consider the nodes to be neighbors. In thediagram, the practical model would lead to the nodes be-ing neighbors briefly, then dropping the link, then beingneighbors again, then dropping the link.
In addition to spatial variations in signal quality, a ra-dio’s signal quality varies over time, even for a station-ary radio and receiver. Obstacles come and go: peopleand vehicles move about, leaves flutter, doors shut. Linkconnectivity can come and go; one packet may reach aneighbor successfully, and the next packet may fail. Both
10
Flat Earth
3-D
Obstacles
Figure 10: The Flat Earth model is overly simplistic.
short-term and long-term changes are common in reality,but not considered by most practical models. Some, butnot all, of this variation can be masked by the physical ordata-link layer of the network interface.
Although the theoretical model may be easy to usewhen simulating ad hoc networks, it leads to an incor-rect sense of the way the network evolves over time. Forexample, in Figure11, the link quality (and link connec-tivity) varies much more rapidly in practice than in the-ory. Many algorithms and protocols may perform muchmore poorly under such dynamic conditions. In some,particularly if network connectivity changes rapidly withrespect to the distributed progress of network-layer orapplication-layer protocols, the algorithm may fail dueto race conditions or a failure to converge. Simple ra-dio models fail to explore these critical realities that candramatically affect performance and correctness. For ex-ample, Ganesan et al. measured a dense ad hoc networkof sensor nodes and found that small differences in theradios, the propagation distances, and the timing of colli-sions can significantly alter the behavior of even the sim-plest flood-oriented network protocols [GKW+02].
In summary,“good enough” radio models are quite im-portant in simulation of ad hoc networks. The Flat Earthmodel, however, is by no means good enough. In the fol-
Time
0%
100%
T P
TP
Link
Qua
lity
Node Trajectory Past Another Node
Figure 11: Difference between theory (T) and practice(P).
lowing sections we make this argument more precise.
3.2 Models used in research
We surveyed a set of MobiCom and MobiHoc proceed-ings from 1995 through 2003. We inspected the simula-tion sections of every article in which RF modeling issuesseemed relevant, and categorized the approach into one ofthree bins:Flat Earth, Simple, andGood. This catego-rization required a fair amount of value judgment on ourpart, and we omitted cases in which we could not deter-mine these basic facts about the simulation runs.
Figure 12 presents the results. Note that even inthe best years, the Simple and Flat-Earth papers signifi-cantly outnumber the Good papers. A few, such as Takaiet al. [TMB01], deserve commendation for thoughtfulchannel models.
Flat Earth models are based on Cartesian X–Y prox-imity, that is, nodesA andB communicate if and only ifnodeA is within some distance of nodeB.
Simple modelsare, almost without exception,ns-2models using the CMU 802.11 radio model [FV02]. Thismodel provides what has sometimes been termed a “real-istic” radio propagation model. Indeed it is significantlymore realistic than the “Flat Earth” model, e.g., it modelspacket delay and loss caused by interference rather thanassuming that all transmissions in range are received per-fectly. We still call it a “simple” model, however, becauseit embodies many of the questionable axioms we detail
11
95 96 97 98 99 00 01 02 03
0123456789
101112131415161718
GoodSimpleFlat Earth
Figure 12: The number of papers in each year of Mobicomand MobiHoc that fall into each category.
below. In particular, the standard release ofns-2 pro-vides a simple free-space model (often termed a “Friis-free-space” model in the literature) and a two-ray ground-reflection model. Both are described in thens-2 docu-ment package [FV02].
The free-space model is similar to the “Flat Earth”model described above, as it does not include effects ofterrain, obstacles, or fading. It does, however, model sig-nal strength with somewhat finer detail (1/r2) than just“present” or “absent.”
The two-ray ground-reflection model, which considersboth the direct and ground-reflected propagation path be-tween transmitter and receiver, is better, but not partic-ularly well suited to most MANET simulations. It hasbeen reasonably accurate for predicting large-scale sig-nal strength over distances of several kilometers for cel-lular telephony systems using tall towers (heights above50m), and also for line-of-sight micro-cell channels inurban environments. Neither is characteristic of typicalMANET scenarios. In addition, while this propagationmodel does take into account antenna heights of the twonodes, it assumes that the earth is flat (and there are oth-erwise no obstructions) between the nodes. This may bea plausible simplification when modeling cell towers, butnot when modeling vehicular or handheld nodes becausethese are often surrounded by obstructions. Thus it too isa “Flat Earth” model, even more so if the modeler doesnot explicitly choose differing antenna heights as a nodemoves.10
More recently,ns-2 added a third channel model—the“shadowing” model described earlier by Lee [Lee82]—to account for indoor obstructions and outdoor shad-
10See also Lundberg [Lun02], Sections 4.3.4–4.3.5, for additional re-marks on the two-ray model’s lack of realism.
owing via a probabilistic model [FV02]. The problemwith ns-2 ’s shadowing model is that the model doesnot consider correlations: a real shadowing effect hasstrong correlations between two locations that are closeto each other. More precisely, the shadow fading shouldbe modeled as a two-dimensional log-normal randomprocess with exponentially decaying spatial correlations(see [Gud91] for details). To our knowledge, only a fewsimulation studies include a valid shadowing model. Forexample, WiPPET considers using the correlated shad-owing model to compute a gain matrix to describe ra-dio propagation scenarios [KLM +00]. WiPPET, however,only simulates cellular systems. The simulation modelwe later use for this study considers the shadowing ef-fect as a random process that is temporally correlated; be-tween each pair of nodes we use the same sample from thelog-normal distribution if the two packets are transmittedwithin a pre-specified time period.11
Good models have fairly plausible RF propagationtreatment. In general, these models are used in paperscoming from the cellular telephone community, and con-centrate on the exact mechanics of RF propagation. Togive a flavor of these “good” models, witness this quotefrom one such paper [ER00]:
In our simulations, we use a model for the pathloss in the channel developed by Erceg et al.This model was developed based on extensiveexperimental data collected in a large numberof existing macro-cells in several suburban ar-eas in New Jersey and around Seattle, Chicago,Atlanta, and Dallas. . . . [Equation follows withparameters for antenna location in 3-D, wave-length, and six experimentally determined pa-rameters based on terrain and foliage types.]. . . In the results presented in this section, . . . theterrain was assumed to be either hilly with lighttree density or flat with moderate-to-heavy treedensity. [Detailed parameter values follow.]
Of course, the details of RF propagation are not al-ways essential in good network simulations; most criti-cal is the overall realism of connectivity and changes inconnectivity (Are there hills? Are there walls?). Alongthese lines, we particularly liked the simulations of well-known routing algorithms presented by Johansson et al.[JLH+99], which used relatively detailed, realistic sce-narios for a conference room, event coverage, and disas-ter area. Although this paper employed thens-2 802.11radio model, it was rounded out with realistic network ob-stacles and node mobility.
11A recent study by Yuen et al. proposes a novel approach to modelingthe correlation as a Gauss-Markov process [YLA02]. We are currentlyinvestigating this approach.
12
3.3 Common MANET axioms
For the sake of clarity, let us be explicit about some basic“axioms” upon which most MANET research explicitlyor implicitly relies. These axioms, not all of which areorthogonal, deeply shape how network protocols behave.We note that all of these axioms are contradicted by theactual measurements reported in the next section.0: The world is flat.1: A radio’s transmission area is circular.2: All radios have equal range.3: If I can hear you, you can hear me (symmetry).4: If I can hear you at all, I can hear you perfectly.5: Signal strength is a simple function of distance.
There are many combinations of these axioms seen inthe literature. In extreme cases, the combination of theseaxioms leads to a simple model like that in the top dia-gram in Figure9. Some papers assume Axioms 0–4 andyet use a simple signal propagation model that expressessome fading with distance; a threshold on signal strengthdetermines reception. Some papers assume Axioms 0–3and add a reception probability to avoid Axiom 4.
In this paper we address the research community in-terested in ad hoc routing protocols and other distributedprotocols at the network layer. The network layer restson the physical and medium-access (MAC) layers, and itsbehavior is strongly influenced by their behavior. Indeedmany MANET research projects consider the physical andmedium-access layer as a single abstraction, and use theabove axioms to model their combined behavior. We takethis network-layer point of view through the remainder ofthe section. Although we mention some of the individualphysical- and MAC-layer effects that influence the behav-ior seen at the network layer, we do not attempt to iden-tify precisely which effects cause which behaviors; suchan exercise is beyond the scope of this paper. We nextshow that the above axioms do not adequately describethe network-layer’s view of the world. Then, in Section 4,we show how the use of these axioms leads simulations toresults that differ radically from reality.
3.4 The reality of the axioms
Unfortunately, real wireless network devices are notnearly as simple as those considered by the axioms in thepreceding section. In this section, we use data collectedfrom the large MANET experiment described previouslyto examine the reality of radio behavior in an actual ad hocnetwork implementation. We demonstrate how this realityclearly differs from the behavior described by our axioms.
Before proceeding, it should be noted that the wirelesscards in our experiment operated at 2 Mb/s. This fixed ratemade it much easier to conduct the experiment, since wedid not need to track (and later model) automatic changes
to each card’s transmission rate. Most current wirelesscards are multi-rate, however, which could lead toAx-iom 6: Each packet is transmitted at the same bit rate.We leave the effects of this axiom as an area for futurework.
We also note that in the following analysis we do notuse data from the STARA portion of the outdoor experi-ment. We were concerned that the excessive control traf-fic generated by this algorithm might impede an accurateassessment of the observed radio behavior.
3.4.1 Axiom 0
The world is flat.
Common stochastic radio propagation models assumea flat earth, and yet clearly the Earth is not flat. Evenat the short distances considered by most MANET re-search, hills and buildings present obstacles that dramati-cally affect wireless signal propagation. Furthermore, thewireless nodes themselves are not always at ground level.A local researcher using Berkeley “motes” for sensor-network research notes the critical impact of elevation andground-reflection effects:
In our current experiments we just bought 60plastic flower pots to raise the motes off theground because we found that putting the moteson the ground drastically reduces their transmitrange (though not the receive range). Raisingthem a few inches makes a big difference.
Even where the ground is nearly flat, note that wirelessnodes are often used in multi-story buildings. Indeed twonodes may be found at exactly the samex, y location, buton different floors. (This condition is common among theWiFi access points deployed on our campus.) Any FlatEarth model would assume that they are in the same loca-tion, and yet they are not. In some tall buildings, we foundit was impossible for a node on the fourth floor to hear anode in the basement, at the samex, y location.
We need no data to “disprove” this axiom. Ultimately,it is the burden of all MANET researchers to either a) usea detailed and realistic terrain model, accounting for theeffects of terrain, or b) clearly condition their conclusionsas being valid only on flat, obstacle-free terrain.
3.4.2 Axioms 1 and 2
A radio’s transmission area is circular.
All radios have equal range.
The real-world radio map of Figure9 makes it clearthat the signal coverage area of a radio is far from simple.
13
Not only is it neither circular nor convex, it often is non-contiguous.
We combine the above two intuitive axioms into a moreprecise, testable axiom that corresponds to the way theaxiom often appears (implicitly) in MANET research.
Testable Axiom 1. The success of a transmission fromone radio to another depends only on the distance
between radios.
Although it is true that successful communication usu-ally becomes less likely with increasing distance, there aremany other factors: (1) All radios are not identical. Al-though in our experiment we used “identical” WiFi cards,there are reasonable applications where the radios or an-tennas vary from node to node. (2) Antennas are not per-fectly omnidirectional. Thus, the angle of the sender’santenna, the angle of the receiver’s antenna, and their rel-ative locations all matter. (3) Background noise varieswith time and location. Finally, (4) there are hills andobstacles, including people, that block or reflect wirelesssignals (that is, Axiom 0 is false).
From the point of view of the network layer, thesephysical-layer effects are compounded by MAC-layer ef-fects, notably, that collisions due to transmissions fromother nodes in the ad hoc network (or from third partiesoutside the set of nodes forming the network) reduce thetransmission success in ways that are unrelated to dis-tance. In this section, we use our experimental data toexamine the effect of antenna angle, sender location, andsender identity on the probability distribution of beaconreception over distance.
We first demonstrate that the probability of a beaconpacket being received by nearby nodes depends stronglyon the angle between sender and receiver antennas. Inour experiments, we had each student carry their “node,”a closed laptop, under their arm with the wireless inter-face (an 802.11b device in PC-card format) sticking outin front of them. By examining successive location ob-servations for the node, we compute the orientation of theantenna (wireless card) at the time it sent or received abeacon. Then, we compute two angles for each beacon:the angle between the sender’s antenna and the receiver’slocation, and the angle between the receiver’s antenna andthe sender’s location. Figure13illustrates the first of thesetwo angles, while the second is the same figure exceptwith the labels Source and Destination transposed. Fig-ure14shows how the beacon-reception probability variedwith both angles.
To compute Figure14, we consider all possible valuesof each of the two angles, each varying from[−180, 180).We divide each range into buckets of 45 degrees, such thatbucket 0 represents angles in[0, 45), bucket 45 representsangles in[45, 90), and so forth. Since we bucket both an-gles, we obtain the two-dimensional set of buckets shown
in the figure. We use two counters for each bucket, one ac-counting for actual receptions, and the other for potentialreceptions (which includes actual receptions). Each timea node sends a beacon, every other laptop is a potentialrecipient. For every other laptop, therefore, we add one tothe potential-reception count for the bucket representingthe angles between the sender and the potential recipient.If we can find a received beacon in the potential recipi-ent’s beacon log that matches the transmitted beacon, wealso add one to the actual-reception count for the appro-priate count. The beacon reception ratio for a bucket isthus the number of actual receptions divided by the num-ber of potential receptions. Each beacon-reception prob-ability is calculated without regard to distance, and thusrepresents the reception probability across all distances.In addition, for all of our axiom analyses, we consideredonly the western half of the field, and incremented thecounts only when both the sender and the (potential) re-cipient were in the western half. By considering only thewestern half, which is perfectly flat and does not includethe lower-altitude section, we eliminate the most obviousterrain effects from our results. Overall, there were 40,894beacons transmitted in the western half of the field, and af-ter matching and filtering, we had 275,176 laptop pairs, in121,250 of which the beacon was received, and in 153,926of which the beacon was not received.
Figure14 shows that the orientation of both antennaswas a significant factor in beacon reception. Of course,there is a direct relationship between the antenna anglesand whether the sender or receiver (human or laptop) isbetween the two antennas. With a sender angle of 180,for example, the receiver is directly behind the sender, andboth the sender’s body and laptop serves as an obstructionto the signal. A different kind of antenna, extending abovethe level of the participants’ heads, would be needed toseparate the angle effects into two categories, effects dueto human or laptop obstruction, and effects due to the ir-regularity of the radio coverage area.
Although the western half of our test field was flat, weobserved that the beacon-reception probability distribu-tion varied in different areas. We subdivided the westernhalf into four equal-sized quadrants (northwest, northeast,southeast, southwest), and computed a separate receptionprobability distribution for beacons sent from each quad-rant. Figure15 shows that the distribution of beacon-reception probability was different for each quadrant, byabout 10–15 percent for each distance. We bucketed thelaptop pairs according to the distance between the senderand the (intended) destination—the leftmost bar in thegraph, for example, is the reception probability for laptoppairs whose separation was in the range[0, 25). Althoughthere are many possible explanations for this quadrant-based variation, whether physical terrain, external noise,or time-varying conditions, the difference between distri-
14
Direction ofradio (andnode movement)
The anglebetween thesource radioand thedestination
Destination
Source
Figure 13: The angle between the sending laptop’s an-tenna (wireless card) and the destination laptop.
butions is enough to make it clear that the location of thesender is not to be ignored.
The beacon-reception probability in the western halfof the field also varied according to the identity of thesender. Although all equipment used in every node wasan identical model purchased in the same lot and con-figured identically, the distribution was different for eachsender. Figure16 shows the mean and standard devia-tion of beacon-reception probability computed across allsending nodes, for each bucket between 0 and 300 me-ters. The buckets between 250 and 300 meters were nearlyempty. Although the mean across nodes, depicted bythe boxes, is steadily decreasing, there also is substantialvariation across nodes, depicted by the standard-deviationbars on each bucket. This variation cannot be explainedentirely by manufacturing variations within the antennas,and likely includes terrain, noise and other factors, evenon our space of flat, open ground. It also is important tonote, however, that there are only 500-1000 data pointsfor each (laptop, destination bucket) pair. With this num-ber of data points, the differences may not be statisticallysignificant. In particular, if a laptop is moving away frommost other laptops, we might cover only a small portionof the possible angles, leading to markedly different re-sults than for other laptops. Overall, the effect of identityon transmission behavior bears further study with experi-ments specifically designed to test it.
In other work, Ganesan et al. used a network of Berke-ley “motes” to measure signal strength of a mote’s radiothroughout a mesh of mote nodes [GKW+02].12 The re-sulting contour map is not circular, nor convex, nor evenmonotonically decreasing with distance. Indeed, since the
12The Berkeley mote is currently the most common research platformfor real experiments with ad hoc sensor networks.
−180−135
−90 −45
0 45
90 135
180−180 −135 −90 −45 0 45 90 135 180
0.3
0.35
0.4
0.45
0.5
0.55
Sender to Destination
Destination to Sender
Probability of Beacon Reception by Angle
Pro
babi
lity
of B
eaco
n R
ecep
tion
Figure 14: The probability of beacon reception (over alldistances) as a function of the two angles, the angle be-tween the sender’s antenna orientation and the receiver’slocation, and the angle between the receiver’s antenna ori-entation and the sender’s location. In this plot, we dividethe angles into buckets of 45 degrees each, and includeonly data from the western half of the field. We also ex-press the angles on the scale of -180 to 180, rather than 0to 360, to better capture the inherent symmetry. -180 and180 both refer to the case where the sending antenna ispointing directly away from the intended destination, or,correspondingly, the receiving antenna is pointing directlyaway from the sending node.
coverage area of a radio is not circular, it is difficult toeven define the “range” of a radio.
3.4.3 Axiom 3
If I can hear you, you can hear me (symmetry).
More precisely,
Testable Axiom 3: If a message fromA to B succeeds, animmediate reply fromB to A succeeds.
This wording adds a sense of time, since it is clearlyimpossible (in most MANET technologies) forA andBto transmit at the same time and result in a successful mes-sage, and sinceA andB may be moving, it is importantto consider symmetry over a brief time period so thatAandB have not moved apart.
There are many factors affecting symmetry, from thepoint of view of the network layer, including the phys-ical effects mentioned above (terrain, obstacles, relativeantenna angles) as well as MAC-layer collisions. It is
15
0
0.2
0.4
0.6
0.8
1
0 25 50 75 100 125 150 175 200 225 250 275 300
Pro
babi
lity
of B
eaco
n R
ecep
tion
Distance In Meters
SWSE
NWNE
Figure 15: The probability of beacon reception variedfrom quadrant to quadrant within the western half of thefield.
0
0.2
0.4
0.6
0.8
1
0 25 50 75 100 125 150 175 200 225 250 275 300
Bea
con
Rec
eptio
n P
roba
bilit
y
Distance in Meters
Figure 16: The average and standard deviation of recep-tion probability across all nodes, again for the western halfof the field.
worth noting that the 802.11 MAC layer includes an inter-nal acknowledgment feature, and a limited amount of re-transmission attempts until successful acknowledgment.Thus, the network layer does not perceive a frame as suc-cessfully delivered unless symmetric reception was pos-sible. Thus, for the purposes of this axiom, we choseto examine the broadcast beacons from our experimentaldataset, since the 802.11 MAC has no internal acknowl-edgment for broadcast frames. Since all of our nodes senta beacon every three seconds, we were able to identifysymmetry as follows: whenever a nodeB received a bea-con from nodeA, we checked to see whetherB’s nextbeacon was also received by nodeA.
Figure17shows the conditional probability of symmet-ric beacon reception. Using the definition of symmetrydescribed above, we calculate each bar by dividing thenumber of observed symmetric relationships by the totalnumber of observed symmetricand asymmetric relation-ships for the given distance range. If the physical andMAC layer behavior was truly symmetric, this probabil-ity would be 1.0 across all distances. In reality, the prob-
0
0.2
0.4
0.6
0.8
1
0 25 50 75 100 125 150 175 200 225 250 275 300
Pro
babi
lity
of B
eaco
n R
ecep
tion
Distance in Meters
Figure 17: The conditional probability of symmetric bea-con reception as it varied with the distance between twonodes, again for the western half of the field.
ability was never much more than 0.8, most likely due toMAC-layer collisions between beacons. Since this graphdepends on the joint probability of a beacon arriving fromA to B and then another fromB to A, the lower receptionprobability of higher distances leads to a lower joint prob-ability and a lower conditional probability. The abnormalbump in the 200 to 225 meter distance bucket is explainedby the fact that the experimental field was roughly 225meters long on its north-south axis. We observed that itwas a common movement pattern to walk to the either thenorthern or southern terminus of the field, and then turn tohead toward another location. Therefore, there commonlyoccurred a situation where two nodes would be facingeach other from opposite ends of the field. In this orien-tation their reception probability was increased, bumpingup the overall probability observed for this range.
Figure18shows how the conditional probability variedacross all the nodes in the experiment. The probabilitywas consistently close to its mean 0.76, but did vary fromnode to node with a standard deviation of 0.029 (or 3.9%).Similarly, when calculated for each of the four quadrants(not shown), the probability also was consistently close toits mean 0.76, but did have a standard deviation of 0.033(or 4.3%). As mentioned in the discussion of Axioms1 and 2, there are many possible explanations for thesevariations, including physical terrain, external noise, anddifferent movement patterns. Regardless of the specificcauses, the fact that this variation exists evidences the in-validity of assuming equal symmetry among all nodes andlocations in a real environment.
In other work, Ganesan et al. [GKW+02] noted thatabout 5–15% of the links in their ad hoc sensor networkwere asymmetric. In that paper, an asymmetric link hada “good” link in one direction (with high probability ofmessage reception) and a “bad” link in the other direction(with a low probability of message reception). [They donot have a name for a link with a “mediocre” link in either
16
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30
Pro
babi
lity
Node
Figure 18: The conditional probability of symmetric bea-con reception as it varied across individual nodes, againfor the western half of the field.
direction.]Overall, it is clear that reception is far from symmetric.
Nonetheless, many researchers assume this axiom is true,and that all network links are bidirectional. Some do ac-knowledge that real links may be unidirectional, and usu-ally discard those links so that the resulting network hasonly bidirectional links. In a network with mobile nodesor in a dynamic environment, however, link quality canvary frequently and rapidly, so a bidirectional link maybecome unidirectional at any time. It is best to developprotocols that do not assume symmetry.
3.4.4 Axiom 4
If I can hear you at all, I can hear you perfectly.
Testable Axiom 4: The reception probability distributionover distance exhibits a sharp cliff; that is, under some
threshold distance (the “range”) the receptionprobability is 1 and beyond that threshold the reception
probability is 0.
Looking back at Figure16, we see that the beacon-reception probability does indeed fade with the distancebetween the sender and the receiver, rather than remain-ing near 1 out to some clearly defined “range” and thendropping to zero. There is no visible “cliff.” The com-monns-2 model, however, assumes that frame transmis-sion is perfect, within the range of a radio, and as long asthere are no collisions. Althoughns-2 provides hooks toadd a bit-error-rate (BER) model, these hooks are unused.More sophisticated models do exist, particularly those de-veloped byQualnet and the GloMoSim project13 that arebeing used to explore how sophisticated channel modelsaffect simulation outcomes.
Takai examines the effect of channel models on sim-ulation outcomes [TBTG01], and also concluded that
13http://www.scalable-networks.com/pdf/mobihocpreso.pdf
different physical layer models can have dramaticallydifferent effect on the simulated performance of proto-cols [TMB01], but lack of data prevented them from fur-ther validating simulation results against real-world ex-periment results, which they left as future work. In thenext section, we compare the simulation results with datacollected from a real-world experiment, and recommendthat simple models of radio propagation should be avoidedwhenever comparing or verifying protocols, unless thatmodel is known to specifically reflect the target environ-ment.
3.4.5 Axiom 5
Signal strength is a simple function of distance.
Rappaport [Rap96] notes that the average signalstrength should fade with distance according to a power-law model. While this is true, one should not underes-timate the variations in a real environment caused by ob-struction, reflection, refraction, and scattering. In this sec-tion, we show that there is significant variation for individ-ual transmissions.
Testable Axiom 5: We can find a good fit between asimple function and a set of (distance, signal strength)
observations.
To examine this axiom, we consider only received bea-cons, and use the recipient’s signal log to obtain the signalstrength associated with that beacon. More specifically,the signal log actually contains per-second entries, whereeach entry contains the single strength of the most recentpacket received from each laptop. If a data or routingpacket arrives immediately after a beacon, the signal-logentry actually will contain the signal strength of that sec-ond packet. We do not check for this situation, since thesignal information for the second packet is just as validas the signal information for the beacon. It is best, how-ever, to view our signal values as those observed withinone second of beacon transmission, rather than the valuesassociated with the beacons themselves.
As a starting point, Figure19 shows themeanbea-con signal strength observed during the experiment as afunction of distance, as well as best-fit linear and powercurves. The power curve is a good fit and validates Rap-paport’s observation. When we turn our attention to thesignal strength of individual beacons, however, as shownin Figures20 and 21, there clearly is no simple (non-probabilistic) function that will adequately predict the sig-nal strength of an individual beacon based on distancealone.
The reason for this difficulty is clear: our environment,although simple, is full of obstacles and other terrain fea-tures that attenuate or reflect the signal, and the cards
17
169
171
173
175
177
179
181
183
185
12.5 37.5 62.5 87.5 112.5 137.5 162.5 187.5 212.5 237.5
Distance (meters)
Sig
nal S
tren
gth
(dB
m +
255
)
ObservedPowerLinear
Figure 19: Linear and power-curve fits for the mean sig-nal strength observed in the western half of the field. Notethat we show the signal strength as reported by our wire-less cards (which is dBm scaled to a positive range byadding 255), and we plot the mean value for each distancebucket at the midpoint of that bucket.
themselves do not necessarily radiate with equal powerin all directions. In our case, the most common obsta-cles were the people and laptops themselves, and in fact,we initially expected to discover that the signal strengthwas better behaved across a specific angle range (per Fig-ure 14) than across all angles. Even for the seeminglygood case of both source and destination angles between0 and 45 degrees (i.e., the sender and receiver roughlyfacing each other), we obtain a distribution (not shown)remarkably similar to Figure20. Other angle ranges alsoshow the same distribution as Figure20.
Overall, noise-free, reflection-free, obstruction-free,uniformly-radiating environments are simply not real, andsignal strength of individual transmissions will never be asimple function of distance. Researchers must be care-ful to consider how sensitive their simulation results areto signal variations, since their algorithms will encountersignificant variation once deployed.
Summary. These axioms are often considered to be areasonable estimate of how radios actually behave, andtherefore they are frequently used in simulation withoutreservation. Our data, however, reveal the danger of sucha belief. These assumptions do not just simplify reality,in many cases they distort it. An algorithm that performswell in the calm and predictable physical environment de-scribed by our axioms may perform quite differently in theinconsistent and highly variable physical environment thatwe observe in the real world. These results should compelsimulation designers to carefully consider and condition
150
160
170
180
190
200
210
220
230
0 25 50 75 100 125 150 175 200 225 250 275 300
Sig
nal S
treng
th (d
Bm
+ 2
55)
Distance in Meters
Figure 20: A scatter plot demonstrating the poor correla-tion between signal strength and distance. We restrict theplot to beacons both sent and received on the western halfof the field, and show the mean signal strength as a heavydotted line.
160.5170.5
180.5190.5
200.5
2.5 37
.5 72.5
107.
5
142.
5
177.
5
212.
5
0
50
100
150
200
250
300
350
Number of Data Points
Signal Strength (dBm + 255) Distance
Figure 21: Same as Figure20 except that it shows thenumberof observed data points as a function of distanceand signal strength. There is significant weight relativelyfar away from the mean value.
the simplifications they integrate into their radio models.In the next section we quantitatively explore the impact ofthese axioms.
4 Computer simulation results
We demonstrate above that the axioms are untrue, but akey question remains: what is the effect of these axiomson the quality of simulation results? In this section, webegin by comparing the results of our outdoor experimentwith the results of a best-effort simulation model, and thenprogressively weaken the model by assuming some of theaxioms. To better understand the observed effects, wethen use a connectivity trace, derived from the outdoorexperiment data, to more generally validate and probe the
18
predictive power of our simplified models. We also exam-ine the role of two important parameters in generating theresults.
With the exception of the experiment represented byFigure 26, we exclude STARA from our simulation runsbecause of its excessive control traffic problems. This un-usual behavior makes it a poor choice for most of our realworld versus simulation comparisons. We include it inthis figure alone to validate our claim from Section 2 thatdetailed simulation would have revealed STARA’s trafficflow problems to its designers. As we expected, the de-tailed model came much closer than the two simpler mod-els in predicting the algorithm’s poor performance.14
The purpose of this study is not to claim that our simu-lator can accurately model the real network environment.Instead, we show quantitatively the impact of the axiomson the simulated behavior of routing protocols, and pro-vide detailed insight into the varied robustness of threepopular radio models.
We recognize that analytical or simulation research inwireless networking must work with an abstraction of re-ality, modeling the behavior of the wireless network be-low the layer of interest. Unfortunately, overly simplisticor malformed assumptions can lead to misleading or in-correct conclusions.
Our results provide a counter-example to the notion thatthe arbitrary selection and generic configuration of anypopular radio propagation model is sufficient for researchon ad hoc routing algorithms. We do not claim to validate,or invalidate, the results of any other published study. In-deed, our point is that the burden is on the authors of pastand future studies to a) clearly lay out the assumptionsmade in their simulation model, b) demonstrate whetherthose assumptions are reasonable within the context oftheir study, and c) clearly identify any limitations in theconclusions they draw.
4.1 The wireless network simulator
We used SWAN [LYN+04], a simulator for wirelessad hoc networks that provides an integrated, configurable,and flexible environment for evaluating ad hoc routingprotocols, especially for large-scale network scenarios.SWAN contains a detailed model of the IEEE 802.11wireless LAN protocol and a stochastic radio channelmodel, both of which were used in this study.
We used SWAN’s direct-execution simulation tech-niques to execute within the simulator thesameroutingcode that was used in the experiments from the previous
14The two simpler models double the message delivery ratio predictedby the most detailed model. The even simpler analytical model used bySTARA’s designers would have exaggerated this value even more. Onegood simulation run, using a good stochastic model, would have revealeda drastic performance gap between their simple predictions and reality.
section. We modified the real routing code only slightlyto allow multiple instances of a routing protocol imple-mentation to run simultaneously in the simulator’s singleaddress space.
We extended the simulator to read the node mobilityand application-level data logs generated by the real ex-periment. In this way, we were able to reproduce the samenetwork scenario in simulation as in the real experiment.To further increase the fidelity of our simulation, we fo-cused only on the 33 laptops that actually transmitted, re-ceived, and forwarded packets in the real experiments. Toreproduce a comparable traffic pattern in simulation, theapplication traffic generator on each of the 33 active nodesstill included the 7 crashed nodes as their potential packetdestinations. Moreover, by directly running the routingprotocols and the beacon service program, the simulatorgenerated the same types of logs as in the real experiment.These conditions allow a direct comparison of results.
In the next few sections, we describe three simulationmodels with progressively unrealistic assumptions, andthen present results to show the impact.
4.2 Our best model
We begin by comparing the results of the outdoor exper-iment with the simulation results obtained with our bestsignal propagation model and a detailed 802.11 protocolmodel. The best signal propagation model is a stochasticmodel that captures radio signal attenuation as a combi-nation of two effects: small-scale fading and large-scalefading. Small-scale fading describes the rapid fluctua-tion in the envelope of a transmitted radio signal overa short period of time or a small distance, and primar-ily is caused by multipath effects. Although small-scalefading is in general hard to predict, wireless researchersover the years have proposed several successful statisti-cal models for small-scale fading, such as the Rayleighand Ricean distributions. Large-scale fading describes theslowly varying signal-power level over a long time inter-val or a large distance, and has two major contributingfactors: distance path-loss and shadow fading. The dis-tance path-loss models the average signal power loss as afunction of distance: the receiving signal strength is pro-portional to the distance between the transmitter and thereceiver raised to a given exponent. Both the free-spacemodel and the two-ray ground reflection model mentionedearlier can be classified as distance path-loss models. Theshadow fading describes the variations in the receivingsignal power due to scattering; it can be modeled as azero-mean log-normal distribution. Rappaport [Rap96]provides a detailed discussion of these and other models.
For our simulation, given the light traffic used in thereal experiment, we used a simple SNR threshold ap-proach instead of a more computationally intensive BER
19
Experiment Simulation ErrorAODV 42.3% 46.8% 10.5%APRL 17.5% 17.7% 1.1%
ODMRP 62.6% 56.9% -9.2%
Table 2: Comparing message delivery ratios between realexperiment and simulation.
approach. Under heavier traffic, this choice might havesubstantial impact [TMB01]. For the propagation model,we chose 2.8 as the distance path-loss exponent and 6 dBas the shadow fading log normal standard. These values,which must be different for different types of terrain, pro-duce signal propagation distances consistent with our ob-servations from the real network. Finally, for the 802.11model, we chose parameters that match the settings of ourreal wireless cards.
Table2 shows the difference in the overall message de-livery ratio (MDR)—which is the total number of mes-sages received by the application layer divided by the totalnumber of messages generated—between the real exper-iment and the simulation. This propagation model pro-duced relatively good results: the relative errors in pre-dicted MDR were within 10% for all three routing pro-tocols tested. We caution, however, that one cannot ex-pect consistent results when generalizing this stochasticradio propagation model to deal with all network scenar-ios. After all, this model assumes some of the axioms wehave identified, including flat earth, omni-directional ra-dio propagation, and symmetry. In situations where suchassumptions are clearly mistaken—for example, in an ur-ban area—we should expect the model to deviate furtherfrom reality. Moreover, the real routing experiment pro-vides a single reference point, and we do not have suffi-cient data to assess the overall effectiveness of the modelunder different network conditions.
On the other hand, since the model produced good re-sults amenable to our particular outdoor experiment sce-nario, we use it in this study as the baseline to quantifythe effect of the axioms on simulation studies. As weshow, the axiom assumptions can significantly underminethe validity of the simulation results.
4.3 Simpler models
Next we weakened our simulator by introducing a sim-pler signal propagation model. We used the distancepath-loss component from the previous model, but dis-abled the variations in the signal receiving power intro-duced by the stochastic processes. Note that these vari-ations are a result of two distinct random distributions:one for small-scale fading and the other for shadow fad-ing. The free-space model, the two-ray ground reflection
model, and the generic distance path-loss model with agiven exponent—all used commonly by wireless networkresearchers—differ primarily in the maximum distancethat a signal can travel. For example, if we assume thatthe signal transmission power is 15 dBm and the receiv-ing threshold is -81 dBm, the free-space model has a max-imum range of 604 meters, the two-ray ground reflectionmodel a range of 251 meters, and the generic path-lossmodel (with an exponent of 2.8) a range of only 97 me-ters. Indeed, we found that the receiving range plays animportant role in ad hoc routing: longer distance short-ens the data path and can drastically change the routingmaintenance cost [LYN+04].
In this study, we chose to use the two-ray ground re-flection model since its signal travel distance matches ob-servations from the real experiment.15 This weaker modelassumes Axiom 4: “If I can hear you at all, I can hearyou perfectly,” and specifically the testable axiom “Thereception probability distribution over distance exhibits asharp cliff.” Without variations in the radio channel, allsignals travel the same distance, and successful receptionis subject only to the state of interference at the receiver.In other words, the signals can be received successfullywith probability 1 as long as no collision occurs duringreception.
Finally, we consider a third model that further weak-ens the simulator by assuming that the radio propagationchannel isperfect. That is, if the distance between thesender and the recipient is below a certain threshold, thesignal is received successfully with probability 1; other-wise the signal is always lost. The perfect-channel modelrepresents an extreme case where the wireless networkmodel introduces no packet loss from interference or col-lision, and the reception decision is based solely on dis-tance. To simulate this effect, we bypassed the IEEE802.11 protocol layer within each node and replaced itwith a simple protocol layer that calculates signal recep-tion based only on the transmission distance.
4.4 The results
First, we look at the reception ratio of the beacon mes-sages, which were periodically sent via broadcasts by thebeacon service program on each node. We calculate thereception ratio by inspecting the entries in the beaconlogs, just as we did for the real experiment. Figure22plots the beacon reception ratios during the execution ofthe AODV routing protocol. The choice of routing pro-tocol is unimportant in this study since we are comparing
15When we consider the full experiment field, which provides pos-sible reception ranges of over 500 meters, we see almost no receptionsbeyond 250 meters. The 251-meter range of the 2-ray model is computedfrom a well-known formula, using a fixed transmit power (15 dBm) andantenna height (1.0 meter).
20
0%
20%
40%
60%
80%
100%
0 50 100 150 200 250 300
Bea
con
rece
ptio
n ra
tio
Distance (in meters)
real experimentbest modelno variationsperfect channel
Figure 22: The beacon reception ratio at different dis-tances between the sender and the receiver. The proba-bility for each distance bucket is plotted as a point at themidpoint of its bucket; this format is easier to read thanthe boxes used in earlier plots.
the results between the real experiment and simulations.We understand that the control messages used by the rout-ing protocol may slightly skew the beacon reception ratiodue to the competition at the wireless channel.
Compared with the two simple models, our best modelis a better fit for the real experiment results. It does, how-ever, slightly inflate the reception ratios at shorter dis-tances and underestimate them at longer distances. Moreimportant for this study is the dramatic difference we sawwhen signal power variations were not included in thepropagation model. The figure shows a sharp cliff inthe beacon reception ratio curve: the quality of the radiochannel changed abruptly from relatively good receptionto zero reception as soon as the distance threshold wascrossed. The phenomenon is more prominent for the per-fect channel model. Since the model had no interferenceand collision effects, the reception ratio was 100% withinthe propagation range.
Next, we examine the effect of different simulationmodels on the overall performance of the routing proto-cols. Figures23–25show the message delivery ratios, forthe three ad hoc routing algorithms, as we varied the appli-cation traffic intensity by adjusting the average messageinter-arrival time at each node. Note the logarithmic scalefor thex-axes in the plots. The real experiment’s result isrepresented by a single point in each plot.
Figures23–25show that the performance of routing al-gorithms predicted by different simulation models varieddramatically. For AODV and APRL, both simple modelsexaggerated the message delivery ratiosignificantly. Inthose models, the simulated wireless channel was muchmore resilient to errors than the real network, since therewere no spatial or temporal fluctuations in signal power.Without variations, the transmissions had a much higherchance to be successfully received, and in turn, there were
0%
10%
20%
30%
40%
50%
60%
70%
80%
30 10 3 1 0.3
Pac
ket d
eliv
ery
ratio
Average packet inter-arrival time at each node (sec)
AODV
real experimentbest modelsno variationsperfect channel
Figure 23: Message delivery ratios for AODV.
0%
10%
20%
30%
40%
50%
60%
70%
80%
30 10 3 1 0.3
Pac
ket d
eliv
ery
ratio
Average packet inter-arrival time at each node (sec)
APRL
real experimentbest modelsno variationsperfect channel
Figure 24: Message delivery ratios for APRL.
fewer route invalidations, and more packets were able tofind routers to their intended destinations. The perfor-mance of the perfect-channel model remained insensitiveto the traffic load since the model did not include collisionand interference calculations at the receiver, explainingthe divergence of the two simple models as the traffic loadincreases. For ODMRP, we cannot make a clear distinc-tion between the performance of the best model and of theno-variation model. One possible cause is that ODMRPis a multicast algorithm and has a more stringent band-width demand than the strictly unicast protocols. A routeinvalidation in ODMRP triggers an aggressive route re-discovery process, and could cause significant packet lossunder any of the models.
In summary, the assumptions embedded inside thewireless network model have a great effect on the sim-ulation results. On the one hand, our best wireless net-work model assumes some of the axioms, yet the resultsdo not differ significantly from the real experiment results.On the other hand, one must be extremely careful whenassuming some of the axioms. If we had held our ex-periment in an environment with more hills or obstacles,the simulation results would not have matched as well.Even in this relatively flat environment, our study shows
21
0%
10%
20%
30%
40%
50%
60%
70%
80%
30 10 3 1 0.3
Pac
ket d
eliv
ery
ratio
Average packet inter-arrival time at each node (sec)
ODMRP
real experimentbest modelsno variationsperfect channel
Figure 25: Message delivery ratios for ODMRP.
that proper modeling of the lossy characteristics of the ra-dio channel has a significant impact on the routing pro-tocol behaviors. For example, using our best model, onecan conclude from Figure23 and Figure25 that ODMRPperformed better than AODV with light traffic load (con-sistent with real experiment), but that their performancewas comparable when the traffic was heavy. If we use themodel without variations, however, one might arrive at theopposite conclusion, that AODV performed consistentlyno worse than ODMRP. The ODMRP results are interest-ing by themselves, since the packet-delivery degradationas the traffic load increases is more than might be expectedfor an algorithm designed to find redundant paths (throughthe formation of appropriate forwarding groups). Bae hasshown, however, that significant degradation can occur asintermediate nodes move, paths to targets are lost, androute rediscovery competes with other traffic [BLG00]. Inaddition, the node density was high enough that each for-warding group could have included a significant fractionof the nodes, leading to many transmitted copies of eachdata packet. An exploration of this issue is left for futurework.
4.5 Further investigation
In the previous sections, we investigate the impact of as-suming the axioms. We demonstrate that certain assump-tions dramatically affect the results, and in some caseseven reverse the ranking of the algorithms being com-pared. Accordingly, we conclude that simulation design-ers should be wary of what assumptions they make in con-structing their models. In this section we extend our inves-tigation with a series of related simulation experiments.
Specifically, we combined three common radio prop-agation models with the connectivity trace derived fromthe outdoor experiment beacon logs, leading to six differ-ent radio propagation models in simulation: three usingthe connectivity traces and the other three not. In the firstthree cases, we used the connectivity trace to determine
whether a packet from a mobile station could reach an-other mobile station, and then we used the radio propaga-tion models to determine the receiving power for the in-terference calculation. Comparison of models with mea-sured connectivity with those without give us a means ofrefining a model’s power—if a model is seen to requireconnectivity information to work well, it is not a robustmodel because the power of prediction comes from mea-surements. On the other hand if a model without mea-sured connectivity information works about as well asdoes the version with it, then the model itself contains ac-curate predictive power for connectivity.
Recognizing that message delivery ratio is not the onlymetric of interest to routing protocol designers, we alsodescribe the performance of our six models in generatingaccurate hop count distributions.
We then investigate the sensitivity of two importantlarge-scale fading parameters: the distance path-loss ex-ponent, and the standard deviation value for shadow fad-ing. The choice of these parameters is often arbitrary insimulation studies, and we maintain that these values areimportant determinents of the environment being simu-lated, and therefore should be selected and accounted forwith care.
The three models used in this further investigation—generic propagation, two-ray ground reflection, and Friisfree-space—are similar, but not exactly the same as themodels used in quantifying the impact of the axioms.
Our generic propagation model is the same as the bestmodel from the axioms investigation. It uses the samelarge-scale and small-scale fading models, and the sameparameter values for the distance path-loss exponent andshadow-fading standard deviation. Similarly, the two-rayground reflection model is the same as the first weakeningof the axiom’s best model. It uses no shadow-fading orsmall-scale fading, and is configured with the same path-loss exponent as its axioms investigation equivalent.
The Friis free-space model, however, does differ fromthe perfect channel model used previously. The perfectmodel had no implementation of an 802.11 protocol, andinstead used only a simple distance threshold to modelpacket reception. The Friis free-space model, on the otherhand, retains an implementation of the 802.11 protocol,allowing for collisions and interference. And though itassumes an ideal radio propagation condition—the signalstravel in a vacuum space without obstacles—the powerloss is proportional to the square of the distance betweenthe transmitter and the receiver. Because the Friis modeluses signal strength to calculate collisions, this use of apath-loss exponent is a more complicated estimation ofradio behavior than the simple distance threshold used inthe perfect model.
We choose this slightly more complicated model forthis further investigation because the previous sections
22
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
AODV APRL ODMRP STARA
Pac
ket D
eliv
ery
Rat
io
real experimentgeneric model with connectivity
free-space with connectivitytwo-ray with connectivity
generic model no connectivityfree-space no connectivity
two-ray no connectivity
Figure 26: Comparing the message delivery ratio fromthe real experiment with various radio propagation mod-els. “With connectivity” means the connectivity trace wasused.
make it clear that the perfect model, without any compen-sation for collisions or interference, grossly overestimatesprotocol performance in all cases. The use of the Friisfree-space model in this experiment provides more inter-esting results by allowing us to compare three more re-lated approaches to estimating radio reception power overtime and distance.
It should also be noted that to further increase the sim-ilarity between the simulated environment and the realconditions, we modified the application traffic generatorto read the outdoor experiment application log and gen-erate the same packets as in the real experiment. Wewere unable to implement this feature in the axiom exper-iments because we ran those simulations at various ratesof packet generation.
Results. We first examine the message delivery ratio.Figure26 shows the message delivery ratio from the realexperiment and the simulation runs with six radio propa-gation models (three of which used the connectivity tracederived from the real experiment to determine the reacha-bility of the signals). Each simulation result is an averageof five runs; the variance is insignificant and therefore notshown.
These results verify many of the conclusions reachedin the previous simulation experiment. For example, thegeneric propagation model, with typical parameters, of-fers an acceptable prediction of the routing algorithm per-formance. Different propagation models predict vastlydifferent protocol behaviors, and these differences arenon-uniform across the algorithms tested. For three algo-rithms, the two-ray ground reflection and free-space mod-els both exaggerate the PDR, whereas ODMRP perfor-mance was underestimated.
More important is what we observe by comparing themodels with connectivity traces to those without. Thepropagation models that used the connectivity trace ingeneral lower the message delivery ratio, when compar-
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
0 1 2 3 4 5
Frac
tion
of T
otal
Dat
a P
acke
ts in
Tra
nsit
Hop Count
real experimentgeneric model with connectivity
free-space with connectivitytwo-ray with connectivity
generic model no connectivityfree-space no connectivity
two-ray no connectivity
Figure 27: The hop-count histogram of AODV in real ex-periment and in simulation.
ing with the propagation models that do not use the con-nectivity trace. This result is not surprising: the connec-tivity trace, to some degree, can represent the peculiar ra-dio propagation scenario of the test environment. In ourexperiment, there were significant elevation changes inthe test field that led to the obstruction of radio signalsbetween laptops that were close by in distance. With-out connectivity traces, the propagation model assumesan omni-directional path loss dependent only on the dis-tance, which resulted in a more connected network (fewerhops) and therefore better delivery ratio.
Of course, the message delivery ratio does not reflectthe entire execution environment of the routing algorithm.From the routing event logs, we collected statistics relatedto each particular routing strategy. Figure27 shows ahistogram of the number of hops that a data packet tra-versed in AODV, before it either reached its destination ordropped along the path. For example, a hop count of zeromeans that the packet was dropped at the source node;a hop count of one means the packet went one hop: ei-ther the destination was its neighbor or the packet failedto reach the next hop. The figure shows the fraction ofthe data packets that traveled in the given number of hops.As above, the free-space and two-ray models resulted infewer hops by exaggerating the transmission range. Wealso see that the connectivity trace was helpful in predict-ing the hop counts, which confirms that the problem withthe free-space and two-ray models using the connectivitytrace was that they did not consider packet losses due tothe variations in receiving signal power.
Finally, we take a look at the sensitivity of certain sim-ulation parameters in the generic propagation model. Theexponent for the distance path loss and the standard de-viation in log-normal distribution for the shadow fadingare heavily dependent on the environment under investi-gation. In the next experiment, we ran a simulation withthe same number of mobile stations and with the sametraffic load as in the real experiment. Figure28 showsAODV performance in packet delivery ratio, as we variedthe path-loss exponent from 2 to 4 and the shadow log-
23
2
2.5
3
3.5
4
02
46
810
12
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Path Loss ExponentShadow Stdev
Pac
ket D
eliv
ery
Rat
io
Figure 28: Sensitivity of the AODV protocol performanceto the parameters of the large-scale fading model.
normal standard deviation from 0 to 12 dB—the rangessuggested by [Rap96] for radio modeling out-of-doors.
The AODV behavior was more sensitive to the path-loss exponent than to the shadow standard deviation. Thatis, the signal propagation distance had a stronger effecton the algorithm’s performance. A shorter transmissionrange means packets must travel through more hops (vialonger routes) before reaching its destination, and there-fore has a higher probability to be dropped. A largershadow standard deviation caused the links to be moreunstable, but the effect varied. On the one hand, whenthe path-loss exponent was small, the signals had a longtransmission range, so the small variation in the receivingsignal strength did not have a significant effect on routing,causing only infrequent link breakage. On the other hand,when the exponent was large, most nodes were discon-nected. A variation in the receiving signal power helpedestablish some routes that were impossible if not for thesignal power fluctuation. Between the extremes, a largervariation in the link quality generally caused more trans-mission failures, and therefore resulted in slightly lowermessage delivery ratio.
The critical implication of this sensitivity study is thatwe cannot just grab a set of large-scale fading parameters,use them, and expect meaningful results for any specificenvironment of interest. On the one hand, pre-simulationempirical work to estimate path-loss characteristics mightbe called for, if the point of the experiment is to quantifybehavior in a given environment. Alternatively, one mayrequire more complex radio models (such as ray tracing)that include complex explicit representations of the do-main of interest. On the other hand, if the objective isto compare protocols, knowledge that the generic propa-gation model is good lets us compare protocols using arange of path-loss values. While this does notquantifybehavior, it may allow us to makequalitativeconclusionsabout the protocols over a range of environments.
To summarize, this further investigation reaffirms ourearlier conclusion that it is critical to choose a proper
wireless model that reflects a real-world scenario forstudying the performance of ad hoc routing algorithms. Incontrast to earlier studies [TBTG01], we found that usinga simple stochastic radio propagation model with param-eters typical to the outdoor environment can produce ac-ceptable results. We must recognize, however, the resultsare sensitive to these parameters. It is for this reason wecaution that the conclusions drawn from simulation stud-ies using simple propagation models should apply only tothe environment they represent. The free-space model andthe two-ray model, which exaggerate the radio transmis-sion range and ignore the variations in the receiving signalpower, can largely misrepresent the network conditions.
5 Conclusion
In recent years, dozens of Mobicom and Mobihoc papershave presented simulation results for mobile ad hoc net-works. The great majority of these papers rely on overlysimplistic assumptions of how radios work. Both widelyused radio models, “flat earth” andns-2 “802.11” mod-els, embody the following set of axioms: the world is twodimensional; a radio’s transmission area is roughly circu-lar; all radios have equal range; if I can hear you, you canhear me; if I can hear you at all, I can hear you perfectly;and signal strength is a simple function of distance.
Others have noted that real radios and ad hoc networksare much more complex than the simple models usedby most researchers [PJL02], and that these complexitieshave a significant impact on the behavior of MANET pro-tocols and algorithms [GKW+02]. In this study, we val-idate the importance of this problem, and present resultsand recommendations to help researchers generate morereliable simulations.
We present the results of an unprecedented large-scaleoutdoor experiment comparing in detail the performanceof four different ad hoc algorithms. We then enumeratethe set of common assumptions used in MANET research,and use data from our real-world experiment to stronglycontradict these “axioms.” Finally, we describe a seriesof simulation experiments that quantify the impact of as-suming these axioms and validate a group of radio modelscommonly used in ad hoc network research.
The results cast doubt on published simulation resultsthat implicitly rely on our identified assumptions, and pro-vide guidance for designing and configuring more reliablesimulation models for use in future studies.
We conclude with a series of recommendations,...for the MANET research community:
1. Choose your target environment carefully, clearly listyour assumptions about that environment, choose sim-ulation models and conditions that match those as-
24
sumptions, and report the results of the simulation inthe context of those assumptions and conditions.
2. Use a realistic stochastic model when verifying a pro-tocol, or comparing a protocol to existing protocols.Furthermore, any simulation should explore a range ofmodel parameters since the effect of these parametersis not uniform across different protocols. Simple mod-els are still useful for the initial exploration of a broadrange of design options, due to their efficiency.
3. Consider three-dimensional terrain, with moderatehills and valleys, and corresponding radio propagationeffects. It would be helpful if the community agreedon a few standard terrains for comparison purposes.
4. Include some fraction of asymmetric links (e.g., whereA can hearB but not vice versa) and some time-varying fluctuations in whetherA’s packets can bereceived byB or not. Here thens-2 “shadowing”model may prove a good starting point.
5. Use real data as input to simulators, where possible.For example, using our data as a static “snapshot” of arealistic ad hoc wireless network with significant linkasymmetries, packet loss, elevated nodes with highfan-in, and so forth, researchers could verify whethertheir protocols form networks as expected, even in theabsence of mobility. The dataset also may be helpful inthe development of new, more realistic radio models.
6. Recognize that connectivity is easily overestimated insimulation. Even the most realistic models used inour experiments overestimated network connectivity ascompared to our real-world results. This observation isespecially important when validating or comparing aprotocol that depends on a certain minimum thresholdof network connections to perform effectively.
7. Avoid simple models, such as free-space or two-rayground reflection, when validating or comparing a pro-tocol for which hop count is a vital component of itsperformance. These models significantly exaggeratetransmission range, and subsequently lower hop countsto an unrealistic level.
...for simulation and model designers:
1. Allow protocol designers to run the same code in thesimulator as they do in a real system [LYN+04], mak-ing it easier to compare experimental and simulationresults.
2. Develop a simulation infrastructure that encourages theexploration of a range of model parameters.
3. Develop a range of propagation models that suit differ-ent environments, and clearly define the assumptionsunderlying each model. Models encompassing bothphysical and data-link layer need to be especially care-ful.
4. Support the development of standard terrain and mo-bility models, and formats for importing real terraindata or mobility traces into the simulation.
...for protocol designers:
1. Consider carefully your assumptions of lower layers.In our experimental results, we found that the successof a transmission between radios depends on many fac-tors (ground cover, antenna angles, human and phys-ical obstructions, background noise, and competitionfrom other nodes), most of which cannot be accuratelymodeled, predicted or detected at the speed necessaryto make per-packet routing decisions. A routing proto-col that relies on an acknowledgement quickly mak-ing it from the target to the source over the reversepath, that assumes that beacons or other broadcast traf-fic can be reliably received by most or all transmission-range neighbors, or that uses an instantaneous measureof link quality to make significant future decisions,is likely to function significantly differently outdoorsthan under simulation or indoor tests.
2. Develop protocols that adapt to environmental condi-tions. In our simulation results, we found that the rel-ative performance of two algorithms (such as AODVand ODMRP) can change significantly, and even re-verse, as simulation assumptions or model parameterschange. Although some assumptions may not signif-icantly affect the agreement between the experimen-tal and simulation results, others may introduce radi-cal disagreement. For similar reasons, a routing proto-col tested indoors may work very differently outdoors.Designers should consider developing protocols thatmake few assumptions about their environment, or areable to adapt automatically to different environmentalconditions.
3. Explore the costs and benefits of control traffic. Bothour experimental and simulation results hint that thereis a tension between the control traffic needed to iden-tify and use redundant paths and the interference thatthis extra traffic introduces when the ad hoc routingalgorithm is trying to react to a change in node topol-ogy. The importance of reducing interference versusidentifying redundant paths (or reacting quickly to apath loss) might appear significantly different in realexperiments than under simple simulations, and pro-tocol designers must consider carefully whether extracontrol traffic is worth the interference price.
4. Use detailed simulation as a tool to aid the proto-col design process. Modeling the effects of collisionsand highly variant transmission strengths may providesome guidance for tailoring your protocol design tomore effectively avoid or adapt to destabilizing envi-ronmental conditions.
25
Availability. We will make our simulator and ourdataset available to the research community upon com-pletion of a conference version of this paper. The dataset,including the actual position and connectivity measure-ments, would be valuable as input to future simula-tion experiments. The simulator contains several radio-propagation models.
Acknowledgments. This thesis contains some text,data, and analysis adapted from two of our related studies.Accordingly, I would like to explicitly thank and acknowl-edge all of the relevant co-authors. Section 2.2, Section 3,Section 4, and Section 5 use material derived fromThemistaken axioms of wireless-network research, which isan update of an existing technical report [KNE03], andis currently under conference submission. The co-authorsfor this work are: David Kotz, Robert Gray, Chip Elliott,Calvin Newport, Jason Liu, and Yougu Yuan. Section 2.1and Section 4.5 use material derived fromSimulation Val-idation Using Direct Execution of Wireless Ad-Hoc Rout-ing Protocols [LYN+04], which was be published in theproceedings of the PADS ’04 conference. The co-authorsfor this work are: Jason Liu, Yougu Yuan, David Nicol,Robert Gray, Calvin Newport, David Kotz, and Luiz Fe-lipe Perrone.
I would also like to thank my thesis committee of DavidKotz, Bob Gray, and Chris McDonald for their insightfulcomments. In particular, I thank Bob Gray for his help inbringing together the outdoor experiment section. And Iwant to especially thank my advisor David Kotz for all ofhis support and guidance over the last two years.
This work was supported in part by the Dartmouth Cen-ter for Mobile Computing, DARPA (contract N66001-96-C-8530), the Department of Justice (contract 2000-CX-K001), the Department of Defense (MURI AFOSRcontract F49620-97-1-03821), and the Office for Domes-tic Preparedness, U.S. Department of Homeland Security(Award No. 2000-DT-CX-K001). Additional fundingprovided by the Dartmouth College Dean of Faculty of-fice in the form of a Presidential Scholar UndergraduateResearch Grant, and a Richter Senior Honors Thesis Re-search Grant. Points of view in this document are those ofthe author(s) and do not necessarily represent the officialposition of the U.S. Department of Homeland Security,the Department of Justice, the Department of Defense, orany other branch of the U.S. Government.
References
[BLG00] Sang Ho Bae, Sung-Ju Lee, and MarioGerla. Unicast performance analysis of theODMRP in a mobile ad-hoc network testbed.In ICCCN 2000, October 2000.
[ER00] Moncef Elaoud and Parameswaran Ra-manathan. Adaptive allocation of CDMAresources for network-level QoS assurances.In Proceedings of the Sixth Annual Interna-tional Conference on Mobile Computing andNetworking, pages 191–199. ACM Press,2000.
[FV02] Kevin Fall and Kannan Varad-han. The ns Manual, April 142002. www.isi.edu/nsnam/ns/ns-documentation.html.
[GK97] P. Gupta and P. R. Kumar. A system and traf-fic dependent adaptive routing algorithm forad hoc networks. InProceedings of the 36thIEEE Conference on Decision and Control,pages 2375–2380, December 1997.
[GKW+02] Deepak Ganesan, Bhaskar Krishnamachari,Alec Woo, David Culler, Deborah Estrin,and Stephen Wicker.Complex behavior atscale: An experimental study of low-powerwireless sensor networks. Technical ReportUCLA/CSD-TR 02-0013, UCLA ComputerScience, 2002.
[Gra00] Robert S. Gray. Soldiers, agents and wirelessnetworks: A report on a military application.In Proceedings of the Fifth InternationalConference and Exhibition on the PracticalApplication of Intelligent Agents and Multi-Agents (PAAM 2000), Manchester, England,April 2000.
[Gud91] M. Gudmundson. Correlation modelfor shadow fading in mobile radio sys-tems. Electronics Letters, 27(23):2145–2146, November 1991.
[Gup00] Piyush Gupta. Design and PerformanceAnalysis of Wireless Networks. PhD the-sis, Department of Electrical Engineering,University of Illinois at Urbana-Champaign,2000.
[JLH+99] Per Johansson, Tony Larsson, NicklasHedman, Bartosz Mielczarek, and MikaelDegermark. Scenario-based performanceanalysis of routing protocols for mobile ad-hoc networks. In Proceedings of the FifthAnnual International Conference on MobileComputing and Networking, pages 195–205.ACM Press, 1999.
[KK98] B. Karp and H. T. Kung. Dynamic neigh-bor discovery and loopfree, multi-hop rout-
26
ing for wireless, mobile networks. HarvardUniversity, May 1998.
[KLM +00] O. E. Kelly, J. Lai, N. B. Mandayam, A. T.Ogielski, J. Panchal, and R. D. Yates. Scal-able parallel simulations of wireless net-works with WiPPET: Modeling of radiopropagation, mobility and protocols.Mo-bile Networks and Applications, 5(3):199–208, September 2000.
[KNE03] David Kotz, Calvin Newport, and ChipElliott. The mistaken axioms of wireless-network research. Technical ReportTR2003-467, Dartmouth College, ComputerScience, Hanover, NH, July 2003.
[Lee82] W. .C. Y. Lee.Mobile communications engi-neering. McGraw-Hill, 1982.
[LGC02] S. J. Lee, M. Gerla, and C. C Chiang. On-demand multicast routing protocol in multi-hop wireless mobile networks.ACM/KluwerMobile Networks and Applications, spe-cial issue on Multipoint Communicationsin Wireless Mobile Networks, 7(6):441–453,December 2002.
[Lun02] David Lundberg.Ad hoc protocol evaluationand experiences of real world ad hoc net-working. Master’s thesis, Department of In-formation Technology, Uppsala University,Sweden, 2002.
[LYN +04] Jason Liu, Yougu Yuan, David M. Nicol,Robert S. Gray, Calvin C. Newport, David F.Kotz, and Luiz Felipe Perrone. Simulationvalidation using direct execution of wirelessad-hoc routing protocols. In18th Work-shop on Parallel and Distributed Simulation(PADS’04), May 2004.
[PJL02] K. Pawlikowski, H.-D.J Jeong, and J.-S.R.Lee. On credibility of simulation studies oftelecommunication networks. IEEE Com-munications, 40(1):132–139, January 2002.
[PR99] C. E. Perkins and E. M. Royer. Ad hoc on-demand distance vector routing. InProceed-ings of the 2nd IEEE Workshop on MobileComputing Systems and Applications, pages90–100, February 1999.
[Rap96] T. S. Rappaport.Wireless Communications,Principles and Practice. Prentice Hall, Up-per Saddle River, New Jersey, 1996.
[TBTG01] M. Takai, R. Bagrodia, K. Tang, andM. Gerla. Efficient wireless network sim-ulations with detailed propagation mod-els. Wireless Networks, 7(3):297–305, May2001.
[TMB01] Mineo Takai, Jay Martin, and Rajive Bagro-dia.Effects of wireless physical layer model-ing in mobile ad hoc networks. In Proceed-ings of MobiHoc 2001, pages 87–94. ACMPress, 2001.
[YLA02] W. H. Yuen, H. Lee, and T. Andersen. A sim-ple and effective cross layer networking sys-tem for mobile ad hoc networks.13th IEEEInt’l Symposium on Personal, Indoor andMobile Radio Communications, September2002.
[ZL02] Yongguang Zhang and Wei Li. An integratedenvironment for testing mobile ad-hoc net-works. In Proceedings of MobiHoc 2002,pages 104–111. ACM Press, 2002.
27
