Modeling and Simulation of Data CenterNetworks

Kashif Bilal, Samee U. Khan, Marc Manzano, Eusebi Calle, Sajjad A. Madani,Khizar Hayat, Dan Chen, Lizhe Wang and Rajiv Ranjan

1 Data Centers and Cloud Computing

Cloud computing is projected as the major paradigm shift in the Information andCommunication Technology (ICT) sector [1]. In recent years, cloud market has ex-perience enormous growth and adoption. The cloud adoption is expected to increasein coming years. As reported by Gartner [2], Software as a Service (SaaS) market isexpected to rise to $ 32.3 billion in 2016 ($ 13.4 billion in 2011). Similarly, Platformas a Service (PaaS) and Infrastructure as a Service (IaaS) are projected to rise from$ 7.6 billion in 2011 to $ 35.5 billion in 2016. Cloud computing has been adoptedin almost all of the major sectors, such as business, research, health, agriculture,e-commerce, and social life.

A data center is a repository to hold computation and storage resources inter-connected to each other using network and communication infrastructure [3]. Datacenters constitute the foundations and building blocks of cloud computing. Con-tinuous evolution of cloud services, as well as their increased demand mandategrowth in data center resources to deliver the expected services and required Qualityof Service (QoS). Various cloud service providers already host hundreds of thou-sands of servers in their respective data centers. Google is estimated to host around

K. Bilal (�) · S. U. KhanNorth Dakota State University, Fargo, ND, USAe-mail: kashifbilal@ciit.net.pk

M. Manzano · E. CalleUniversity of Girona, Girona, Spain

S. A. Madani · K. HayatCOMSATS Institute of Information Technology, Islamabad, Pakistan

D. Chen · L. WangChinese Academy of Sciences, Beijing, China

R. RanjanAustralian National University, Canberra, Australia

© Springer Science+Business Media New York 2015 945S. U. Khan, A. Y. Zomaya (eds.), Handbook on Data Centers,DOI 10.1007/978-1-4939-2092-1_31

946 K. Bilal et al.

0.9 million servers in their data centers. Similarly, Amazon is reported to have around0.45 million servers to support Amazon Web Services (AWS). The number of serversin Microsoft data centers double every 14 months [4].

The projected number of resources required to accommodate future service de-mands within data centers are mounting. The “scale-out” or “scale-up” approachesalone cannot deliver a viable solution for escalating resource demands. Scale-outis the common approach adopted by data centers designers by adding inexpensivecommodity hardware to the data center resources pool. Scale-up approach focuses onimproving and adding more power and complexity to the enterprise-level equipment,which is expensive and power-hungry [6]. Increasing the computational and storageresources is currently not a major challenge in the data center scalability. However,how to interconnect these commodity resources together to deliver the required QoSis the major challenge. Besides scaling-out the data centers, energy consumptionand resultant Operational expenses (OpEx) of data centers also pose serious chal-lenges. Environmental aspects, enormous amount of Green House Gases (GHG)emissions by data centers, and increasing energy costs are worsening the problem.These aforementioned problems mandate the revisions in design and operation ofdata centers.

Data Center Networks (DCNs) play a pivotal role in asserting the performancebounds and Capital Expenditure (CaPex) of a data center. Legacy ThreeTier DCNarchitecture is unable to accommodate the growing demands and scalability withindata centers [3]. Various novel DCN architectures have been proposed in the recentpast to handle the growth trend and scalability demands within data centers [4]. Be-sides, electrical network technology, optical, wireless, and hybrid DCN architecturesare also proposed [16, 17]. Moreover, intra-network traffic within the data center isgrowing. It has been estimated that around 70 % of the network traffic will flowwithin the data centers [7]. Various cloud and data center applications follow severalcommunication patterns, such as one to one, one to many, many to many, and all-to-all traffic flows [8]. The traffic patterns within data centers are fairly different fromthe traffic patterns observed in other type of telecommunication networks. There-fore, the traffic optimization techniques proposed for such networks are inapplicablewithin data centers. Finally, it has been observed that the main DCN architectureshave a low capacity to maintain an acceptable level of connectivity under differenttype of failures [5].

All of the aforementioned challenges require detailed analysis and quantifica-tion of various issues within a data center. In this particular case, simulation is anappropriate solution for detailed analysis and quantification of the aforementionedproblems, since experiments comprising realistic DCNs scenarios are economicallyunviable. Simulation can help to quantify and compare the behavior of a networkunder a presented workload and traffic pattern. Unfortunately, network models andsimulators to quantify the data center network and varying traffic patterns at a de-tailed level are scarce, currently. Moreover, current network simulators, such as ns-2,ns-3, or Omnet++ lack the data center architectural model and simulation capability.Therefore, we implemented the state-of-the-art DCN architectures in ns-3 simulatorto carry out the DCN simulations and comparative analysis of major DCNs [3]. We

Modeling and Simulation of Data Center Networks 947

implemented the three major DCN architectures namely, (a) legacy ThreeTier [10],(b) DCell [8], and (c) FatTree [19]. We implemented six traffic patterns to observethe behavior of the three DCN architectures under a specified workload. We carriedout extensive simulations to perform a comparative analysis of the three consideredDCN architectures.

2 DCN Architectures

Based on the packet routing model, the DCN architectures can be classified in twomajor categories, namely: (a) switch-centric and (b) server-centric networks. Theswitch-centric networks rely on network switches and routers to perform networkpacket forwarding and routing. The ThreeTier, FatTree, VL2, and JellyFish DCNarchitectures are the examples of the switch-centric networks [4]. The server-centricnetworks utilize computational servers to relay and route the network packets. Theserver-centric network may be pure server-based or hybrid (using an amalgam ofnetwork switches and computational server for traffic routing). The CamCube is apure server based DCN architecture that relies solely on computational server forpacket forwarding [9]. The DCell, BCube, and FiConn are examples of the hybridserver-centric DCN architectures [8].

The legacy ThreeTier architecture is the most commonly deployed network topol-ogy within data centers, currently. The ThreeTier architecture is comprised of a singlelayer of computational servers and a three layered hierarchy of network switches androuters (see Fig. 1) [10]. The computational servers are grouped in racks. Typically,around 40 servers within a rack are connected to a Top of the Rack (ToR) switch [11].The ToRs connecting the servers within individual racks make the first layer of thenetwork hierarchy called access layer. Multiple access layer switches are connectedto the aggregate layer switches. The aggregate layer switches make the second layerof switches within the ThreeTier network hierarchy. A single access layer switch isconnected to multiple aggregate layer switches. The high-end enterprise-level coreswitches make the topmost layer of the ThreeTier network hierarchy called corelayer. A single core layer switch is connected to all of the aggregate layer switcheswithin the data center. The intra-rack traffic flow is controlled by the access layerswitches. The traffic flow among the racks with the ToRs connected to the sameaggregate layer switch passes through the aggregate layer switches. The inter-racktraffic flow where the ToRs of the source and destination rack are connected to dif-ferent aggregate layer switch passes through the core layer switches. Higher layersof the ThreeTier network architecture experiences higher oversubscription ratios.Oversubscription ratio is the worst-case available bandwidth among the end hostsand the total bisection bandwidth of the network topology [19].

The FatTree DCN architecture is a clos based arrangement of commodity networkswitches to deliver 1:1 oversubscription ratio [19]. The computational servers andcommodity network switches are arranged in a hierarchical manner similar to the

948 K. Bilal et al.

Access Network

Aggregation Network

Core Network

Fig. 1 ThreeTier DCN architecture

Pod 0 Pod 1 Pod 2 Pod 3

Edge Layer

Aggrega�on Layer

Core Layer

Fig. 2 FatTree DCN architecture

ThreeTier architecture. However, the number of the network devices and the inter-connection topology is different from the ThreeTier architecture (please see Fig. 2).The number of pods (or modules) represented by ‘k’decides the number of devices ineach layer of the topology. There are total (k/2)2 number of switches in the core layerof the FatTree architecture. The aggregate and access layers each contain k/2 numberof switches in each pod. Each access switch is used to connect k/2 computationalservers. Each pod in the FatTree contains k number of switches (arranged in twolayers) and (k/2)2 number of computational servers. The FatTree DCN architectureexhibit better scalability, throughput, and energy efficiency compared to the Three-Tier DCN. The FatTree architecture uses a custom addressing and routing scheme[AiL08].

The DCell is a hybrid server-centric DCN architecture [8]. DCell follows a re-cursively built topology where a server is directly connected to multiple servers inunits called dcells (see Fig. 3). The dcell0 constitutes the building block of the DCellarchitecture; where n servers are interconnected to each other using a commoditynetwork switch (n is a small number usually less than eight). n+ 1 dcell0 cells build

Modeling and Simulation of Data Center Networks 949

DCell1

DCell0

DCell2

Dcell1[0] Dcell1[1]

Dcell1[2]

Dcell1[3]

Dcell1[4]Dcell1[5]

Dcell1[6]

Dcell0[0] Dcell0[1]

Dcell0[2]

Fig. 3 DCell DCN Architecture

a level-1 cell called dcell1. A dcell0 is connected to all other dcell0’s within a dcell1.

Similarly, multiple lower layer dcell(L−1) cells constitute a higher level dcell(L) cell.DCell is an extremely scalable DCN architecture that may scale to millions of serversby having a level-3 DCell with only six servers in each dcell0.

3 DCN Graph Modeling

DCN architectures can be represented as multilayered hierarchical graphs [13]. Thecomputational servers, storage devices, and network devices represent the verticesof the graph. The network links connecting the devices represent the edges of thegraph. Table 1 presents the variables used in DCN models.

950 K. Bilal et al.

Table 1 Variables used in the DCN modeling

Variable Represents

ν Set of vertices (servers, switches, and routers) in the graph

ε Network links connecting various devices

Pi A pod/module in topology representing set of servers and middle layers switches

C Core layer switches

δ Servers

α Access layer switch

γ Aggregate layer switch

k Total number of pods/modules in the topology

n Total number of servers connected to a single access layer switch

s Total number of servers connected to a switch in a dcell0

m Total number of the access layer switches in each pod

q Total number of the aggregate layer switches in each pod

r Total Number of the core layer switches in topology

3.1 ThreeTier DCN Model

The ThreeTier DCN can be represented as

DCNTT = (ν, ε ), (1)

where ν represents the nodes in the ThreeTier graph (computational servers, networkswitches, and routers), and ε represent network links interconnecting the devices.The servers, access, and aggregate layer switches are arranged in k modules/pods(P ki ) and a single layer of the core Cri switches (see Fig. 1 with four modules and acore layer)

v = P ki ∪ Cri , (2)

Each module or pod Pi is organized in three distinct layers of nodes, namely: (a)aggregate layer ( lg), (b) access layer ( lc), and (c) server layer ( ls). The nodes ineach layer within a pod can be represented as

Pi = {lsmα ×nδ ∪ lcmα ∪ ls qγ }, (3)

where δ represents the servers, α represents the access layer switches, and the aggre-gate layer switches are represented by γ . |Pi | represents the total number of nodeswithin a pod

|Pi | =(

m∑1

n +m+ q)

, (4)

Modeling and Simulation of Data Center Networks 951

Total number of nodes in a Threetier architecture having n pods can be calculated as

|ν| ={

k∑i=1

|Pi | + |C|}

, (5)

There are three layers of edges (network links) interconnecting four layers of theThreeTier architecture nodes

ε = {§, α, C| }, (6)

where § represent the edges that connect servers to access switches, α are the edgesused to connect aggregate and access layers switches, and C| represent the edges usedto connect aggregate and core switches. Aggregate switches are also connected toeach other within a pod, represented by ′γ . The set of edges within the ThreeTierarchitecture can be represented by

ε = {§(∀δ,α) , α(∀α, ∀γ ),′γ (∀γ , ∀γ ), C| (∀γ ∀C)}. (7)

Total number of edges within a ThreeTier DCN can be calculated as

|ε| =k∑1

(m∑1

n+m∑1

q + q (q − 1)

2+

q∑1

r

). (8)

3.2 FatTree DCN Model

As discussed in Sect. 2, the FatTree is also a multi-layered DCN architecture similar tothe ThreeTier architecture. However, the number of devices and the interconnectionpattern among the devices in various layers varies largely in both of the architectures.The FatTree architecture follows a Clos topology for network interconnection. Thenumber of nodes in each layer within the FatTree topology is fixed and is based onthe number of the pods ‘k’

n = m = q = (k/2), (9)

r = (k/2)2 (10)

Similar to the ThreeTier architecture, the FatTree DCN can be modeled as

DCNFT = (ν, ε ), (11)

where ν, Pi , |Pi |, and |ν| can be modeled by using Equation 2–5, respectively.However, the aggregate layers switches within a FatTree are not connected to eachother. Moreover, the contrary to the ThreeTier architecture, each of the core layerswitch is connected to a single aggregate layer switch from each pod

ε = {§(∀δ,α) ∪ α(∀α,∀γ ) ∪C| (∀,γi )}

, (12)

952 K. Bilal et al.

and the total number of edges can be calculated as

|ε| =k∑1

(m∑1

n+m∑1

q

)+

R∑1

k. (13)

3.3 DCell DCN Model

DCell uses a recursively built topology, where a single server in a dcell is connectedto servers in other dcells for server-based routing (see Fig 3). The graph model ofthe DCell DCN architecture can be represented as:

DCNDC = (ν, ε ), (14)

ν = {∂i , ∂i+1 , . . . , ∂L} , (15)

where 0 ≤ i ≤ L. ∂0 represents dcell0 and L denotes highest level.

∂0 ={δ ∪ α

}, (16)

where δ represents the set of ‘s’ servers with dcell0 and α presents a single switchconnecting the servers.

∂l = {xl.∂l−1}, (17)

where 1 ≥ l ≤ L, and xl is the total number of ∂l−1 in ∂l .A dcell1 can be represented by

∂1 = {x1.∂0} , (18)

x = s + 1 (19)

similarly, for l ≥ 2:

xl =(l−1∏i=1

xi × s + 1

). (20)

A 3-level DCell can accommodate around 3.6 million servers with s= 6. Total numberof node in a 3-level DCell can be calculated as:

∣∣ν30

∣∣ =(x3∑1

x2∑1

x1∑1

(s + 1)

), (21)

and the total number of edges in a 3-level DCell can be calculated as:

∣∣ε30

∣∣ =x3∑1

(x2∑1

((x1∑1

s

)+ (x1(x1 − 1)/2)

)+ (x2(x2 − 1)/2)

)+ (x3(x3 − 1)/2)

(22)

Modeling and Simulation of Data Center Networks 953

The total number of vertices in the l-level DCell are:

|ν| =(

n∏i=1

(xi∑1

(s + 1)

)/(s + 1)(l−1)

), (23)

and the total number of edges can be calculated as:

|ε| =(

l∏i=1

(xi∑1

(s)

)/s(l−1)

)+ 1/2

⎡⎣ l∑j=1

⎛⎝⎛⎝ l∏y=jxy

⎞⎠ (xj − 1)

⎞⎠⎤⎦ (24)

4 DCNs Implementation in ns-3

We implemented three major DCN architectures namely: (a) ThreeTier, (b) FatTree,and (c) DCell (see Sect. 2 for details). We used ns-3 discrete-event simulator to im-plement the three DCN architectures. We implemented (a) interconnection topology,(b) customized addressing scheme, and (c) customized routing logic for the threeconsidered DCN architectures. Moreover, we implemented six traffic patterns to ob-serve the behavior of the considered DCNs under various network conditions andtraffic loads.

In the year 2003–2004, ns-2 was the most used network simulator for networkresearch [14]. However, to address the outdated code design and scalability of ns-2, a new simulator called ns-3 was introduced. The ns-3 is a new simulator (not anevolution of ns-2) written from scratch. Some of the salient features of ns-3 simulatorare as follows [3, 14], and [18]. The ns-3 simulator offers the modeling of realisticnetwork scenarios. The ns-3 uses the implementation of real Internet Protocol (IP).Moreover, the ns-3 offers implementation of Berkeley Socket Distribution (BSD)sockets interface and installation of multiple network interfaces on a single node.Simulated packets in the ns-3 contain real network bytes. Furthermore, the ns-3 offersto capture the network traces, which can be analyzed by using various network tools,such as WireShark. We implemented our DCN models using the ns-3.13 release.Currently, ns-3.18 is the stable release of the ns-3 simulator [18].

One of the major drawbacks of the ns-3 simulator is that it does not provide Eth-ernet switch implementation. The BridgeNetDevice is the closely related bridgeimplementation that can be used to simulate an Ethernet switch. However, theBridgeNetDevice is used for simulating CSMA devices and does not work for thePoint-To-Point devices. Therefore, we implemented a Point-To-Point based switchfor simulations in ns-3 [18].

4.1 ThreeTier DCN Implementation Details

We offer a customizable implementation of the ThreeTier architecture. The simula-tion parameters can be configured to simulate the ThreeTier architecture with devices

954 K. Bilal et al.

arranged in four layers having different oversubscription ratio. Users can define thenumber of pods/modules in the topology. Each pod contains (a) servers arrangedin racks, (b) number of access layers switches, and (c) number of aggregate layerswitches. Users can specify the required number of servers in each rack. All of theservers within a rack are connected by an access layer (ToR) switch. The bandwidthof the network link interconnecting the servers with the ToR switch can be config-ured. The default bandwidth for the links connecting servers to a ToR is 1Gbps. Allof the ToRs, within a single pod/module are connected to all of the aggregate layerswitches. The default bandwidth of the network links interconnecting the ToRs tothe aggregate layer switch is 10Gbps, which is configurable. The number of devicesand bandwidth of the links in each of the layers (server, access, and aggregate layer)of the ThreeTier architecture remains same in all of the pods.

The core layer of the ThreeTier architecture is the topmost layer that is used toconnect various pods to each other. Users can specify the number of core switchesin the topology. Each core layer switch is connected to all of the aggregate layerswitches. The network switch used in the aggregate and core layer of the ThreeTierarchitecture are often high-end enterprise layer switches. One of the major featuresof the high-end switches is the support of Equal Cost Multi Path (ECMP) routing[15]. We have added the support for ECMP for the aggregate and core layer switchesfor realistic results.

We used ns-3 Ipv4GlobalRoutingHelper class for routing with the ECMP supportin the ThreeTier architecture. It is worth mentioning that the performance and resultsof the ThreeTier architecture are heavily dependent on the oversubscription ratioat each layer and use of the ECMP. The throughput and network delay fluctuatessubstantially by varying the oversubscription ratio and use of ECMP routing. Weconfigured each device with real IP address for simulation. The IP addressing schemeis also customizable and users can assign the network addresses of choice to devices.Figure 4a depicts the topology setup for the ThreeTier architecture with k = 4.

4.2 FatTree DCN Implementation Details

FatTree DCN is based on the Clos interconnection topology. The number of devicesin each layer of the FatTree architecture is fixed based on the number of pods ‘k’.Contrary to the ThreeTier architecture, the user only needs to configure the totalnumber of pods for the FatTree simulation. The implementation of the FatTree ar-chitecture creates k pods and the required devices and interconnection within eachpod. The value of k must be (a) greater than or equal to four and (b) an even number.The network bandwidth of the interconnecting links is configurable and the defaultvalue is 1Gbps. The entire network links uses same bandwidth value contrary to theThreeTier architecture, where the bandwidth value of the network links connectingservers to ToR and the links connecting ToRs to aggregate layer switch is usuallydifferent.

Modeling and Simulation of Data Center Networks 955

Fig. 4 DCN Topologies in ns-3 a ThreeTier Topology b DCell Topology

Fig. 5 Simulation of a k= 4 FatTree in ns-3

FatTree architecture uses a custom network addressing scheme. The networkaddress of a server or a node is dependent on the location of the node. The networkaddress of a server is based on the pod-number that contains the server, and theaccess switch number that connects the server. We have implemented the customnetwork addressing scheme of the FatTree architecture and each of the nods withinthe FatTree is assigned the addressing scheme as specified. Figure 5 and 6 depictsthe assignment of the network addresses within each pod of a FatTree with k= 4.

The FatTree architecture uses a two-level routing scheme for packet forwarding.The packet forwarding decision is based on two-level prefix lookup scheme. TheFatTree uses a primary prefix routing table and a secondary suffix table. Firstly, thelongest prefix match is checked in the primary prefix table. If a match is found for the

956 K. Bilal et al.

10.0.0.2 10.0.0.3 10.0.1.2 10.0.1.3

10.0.0.1 10.0.1.1

10.0.2.1 10.0.3.1

10.1.0.2 10.1.0.3 10.1.1.2 10.1.1.3

10.1.0.1 10.1.1.1

10.1.2.1 10.1.3.1

10.2.0.2 10.2.0.3 10.2.1.2 10.2.1.3

10.2.0.1 10.2.1.1

10.2.2.1 10.2.3.1

10.3.0.2 10.3.0.3 10.3.1.2 10.3.1.3

10.3.0.1 10.3.1.1

10.3.2.1 10.3.3.1

10.4.1.1 10.4.1.2 10.4.2.1 10.4.2.2

Pod 0 Pod 1 Pod 2 Pod 3

Access Layer

AggregationLayer

Core Layer

Fig. 6 Assignment of IP addresses in a k= 4 FatTree

destination address of the packet, then the packet is forwarded to the port specified inrouting table. If the longest prefix does not match, then the longest matching suffixin the secondary table is found, and the packet is forwarded to the port specifiedin the secondary routing table. The switches in the access and aggregate layers usesame algorithm for routing table generation (please see Algorithm 1 in [12]). Thecore layer switches uses a different algorithm (please see Algorithm 2 in [12]) forcore switch routing table generation. We implemented both of the algorithms for ourFatTree DCN implementation.

4.3 DCell DCN Implementation Details

The DCell is a recursively built, highly scalable, and server-based hybrid DCNarchitecture. As detailed in Sect. 2 and 3, the building block of the DCell architectureare presented. We offer a customizable implementation of the DCell architecture. Wehave implemented 3-level DCell topology that can accommodate millions of serversby having less than eight nodes in dcell0. The user can configure the number ofserver in dcell0. The number of servers in the DCell increases exponentially with theincrease in number of servers in dcell0. Table 2 presents the total number of serversin the DCell topology. As can be observed, by having only eight servers can lead toa DCell comprised of 27 million servers. Because of the exponential increase in thenumber of server and upper level dcells, we enabled the configuration of the numberof the dcells at each level. The user can configure the number of servers, the numberof dcell0 in dcell1, number of dcell1 in each dcell2, and number of dcell2 in dcell3 tocontrol the number of servers in resulting DCell.

Each dcell0 contains a Ethernet switch that is used for packet forwarding amongthe servers within the dcell0. The traffic forwarding among the servers in different

Modeling and Simulation of Data Center Networks 957

Table 2 Number of Servers in DCell

Number of servers in dcell0 Total number of serversin 3-level DCell

Total number of nodes (includingswitch) in 3-level DCell

2 1806 2709

3 24492 32656

4 176,820 221,025

5 865,830 1,038,996

6 3,263,442 3,80349

7 10,192,056 11,648,064

8 27,630,792 31,084,641

dcells is performed by servers, that are interconnected to each other. Each servernode is equipped with multiple interface cards to directly connect switch and otherservers. The default bandwidth of the network links is 1Gbps that is also configurable.We used a realistic IP address assignment to each server in the DCell architectureimplementation.

The DCell does not specify any custom addressing scheme. However, the DCellrouting scheme takes into account the placement of the server within the dcells. Forinstance, the NodeId (3, 1, 2) specifies the server number 2, in dcell0 number 1,and dcell1 number 3. We implemented programming routines that can find the IPaddress of a specified server number and vice versa. The DCell uses a custom routingprotocol called DCell Fault-tolerant Routing (DFR) protocol for packet forwarding.The DFR is a recursive and source based routing protocol. When a node wants toinitiate communication with some other node, the DFR is invoked to calculate theend-to-end path for the flow. The DFR first calculates the link connecting the dcells ofthe source and destination node. Then the DFR calculates the path from source to thelink and from link to the destination. The combination of all paths is the end-to-endpath. We implemented the DFR protocol. The output path from the DFR provides theNodeIds instead of the IP address. We use a custom programmed routine to convertthe NodeIds based path to IP based path. We place the complete source to destinationpath in an extra header in each packet, as the DFR is a source based routing protocol.Each intermediate node parses the header and decides the forwarding port/link forthe next hop.

Unfortunately, the algorithm listing for the DCell in the original paper was in-complete and erroneous (please see Fig. 3 in [8]). In Sect. 4.1.1 of the original paper[8], it is mentioned that if (Sk−m< dk−m), then the link interconnecting the sub-dcellscan be calculated as ‘([Sk−m, dk−m − 1], [dk−m, Sk−m])’. The ‘else clause’ for theaforementioned ‘if statement’ is not given in the original paper. Therefore, the im-plementation of the DFR was erroneous and incomplete. We figured the else clausefor the aforementioned scenario and implemented the complete DFR algorithm fortraffic routing. Moreover, the example for the path calculated using DFR also had atypographical mistake in the original paper.

958 K. Bilal et al.

References

1. IBM, IBM Data Center Networking Planning for Virtualization and Cloud Computing, 2011.Online: http://www.redbooks.ibm.com/redbooks/pdfs/sg247928.pdf

2. Gartner, Market Trends: Platform as a Service, Worldwide, 2012–2016, 2H12 Update, 2012.3. K. Bilal, S.U. Khan, L. Zhang, H. Li, K. Hayat, S.A. Madani, N. Min-Allah, L. Wang,

and D. Chen, “Quantitative Comparisons of the State of the Art Data Center Architectures,”Concurrency and Computation: Practice and Experience, (DOI:10.1002/cpe.2963).

4. K. Bilal, S. U. R. Malik, O. Khalid, A. Hameed, E. Alvarez, V. Wijaysekara, R. Irfan, S.Shrestha, D. Dwivedy, M. Ali, U. S. Khan, A. Abbas, N. Jalil, and S. U. Khan, “A Taxon-omy and Survey on Green Data Center Networks,” Future Generation Computer Systems.(Forthcoming.)

5. M. Manzano, K. Bilal, E. Calle, S. U. Khan, “On the connectivity of Data Center Networks”,IEEE Communication Letters. (Forthcoming)

6. K. Yoshiaki, and M. Nishihara. “Survey on Data Center Networking Technologies.” IEICEtransactions on communications, Vol. 96, No. 3, 2013.

7. P. Mahadevan, P. Sharma, S. Banerjee, and P. Ranganathan, “Energy aware networkoperations,” IEEE INFOCOM Workshops 2009, pp. 1–6.

8. C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, and S. Lu, “DCell: A Scalable and Fault-tolerantNetwork Structure for Data Centers.” ACM SIGCOMM Computer Communication Review,Vol. 38, No. 4, 2008, pp. 75–86.

9. H. Abu-Libdeh, P. Costa, A. Rowstron, G. O’Shea, and A. Donnelly, “Symbiotic Routing inFuture Data Centers,” ACM SIGCOMM 2010 conference, New Delhi, India, 2010, pp. 51–62.

10. Cisco, Cisco Data Center Infrastructure 2.5 Design Guide, Cisco press, 2010.11. A. Greenberg, J. Hamilton, N. Kandula, C. Kim, and S. Sengupta, “VL2: a scalable and flexible

data center network,” ACM SIGCOMM Communication Review, Vol. 39, No. 4, 2009, pp.51–62.

12. M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable, commodity data center network archi-tecture,” ACM SIGCOMM 2008 conference on Data communication, Seattle, WA, 2008, pp.63–74.

13. K. Bilal, M. Manzano, S. U. Khan, E. Calle, K. Li, and A. Y. Zomaya, “On the Characteri-zation of the Structural Robustness of Data Center Networks,” IEEE Transactions on CloudComputing. (Forthcoming.)

14. G. Carneiro, ns-3, Network Simulator 3, 2010. Online: http://www.nsnam.org/tutorials/NS-3-LABMEETING-1.pdf

15. C. Hopps, Analysis of an Equal-Cost Multi-Path Algorithm. RFC 2992, Internet EngineeringTask Force, 2000.

16. C. Kachris and L. Tomkos, “A Survey on Optical Interconnects for Data Centers,” Communi-cations Surveys & Tutorials, IEEE, Vol. 14, No. 4, 2012, pp. 1021–1036

17. X. Zhou, Z. Zhang, Y. Zhu, Y. Li, S. Kumar, and A. Vahdat, “Mirror Mirror On The Ceil-ing: Flexible Wireless Links For Data Centers,” ACM SIGCOMM Computer CommunicationReview, Vol. 42, No. 4, pp. 443–454, 2012.

18. ns-3 Simulator, online: http://www.nsnam.org/.19. Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat, “A Scalable, Commodity Data

Center Network Architecture,” ACM SIGCOMM, Seattle, Washington, USA, August 17–22,2008, pp: 63–74.