1

Traffic Engineering for Service-Oriented 5GNetworks with SDN-NFV Integration

Kaige Qu, Student Member, IEEE, Weihua Zhuang, Fellow, IEEE, Qiang Ye, Member, IEEE,Xuemin (Sherman) Shen, Fellow, IEEE, Xu Li, and Jaya Rao

Abstract—Network slicing is a promising approach to provi-sioning service-level virtual network customization, based on theintegration of software-defined networking (SDN) and networkfunction virtualization (NFV) technologies. Although differentnetwork slices are logically independent, they are physicallyoperated over a common infrastructure, resulting in challengesfor quality-of-service (QoS) guarantee among slices in the pres-ence of traffic dynamics. In this article, a traffic engineering(TE) framework is proposed for efficient resource managementamong slices, to avoid congestion and prevent the consequent QoSperformance degradation. An NFV architecture integrated withtwo-level SDN controllers located in tenant and infrastructuredomains can support the proposed TE framework. A case studyis presented to evaluate the effectiveness of the proposed TEframework, in terms of QoS performance guarantee, improvedresource utilization, and reduced reconfiguration overhead.

Index Terms—SDN, NFV, service function chain, networkslicing, traffic engineering, end-to-end delay, load balancing,reconfiguration overhead, state transfer.

I. INTRODUCTION

The fifth generation (5G) communication networks are en-visioned to support a broad range of new services. There arethree typical 5G use case families: enhanced mobile broadband(eMBB), massive machine-type communication (mMTC), andultra-reliable and low latency communication (uRLLC), whosedisparate performance requirements are difficult to be satisfiedby the legacy one-size-fits-all network architecture. Instead,network slicing is required on a per-service basis, to provideservice-level performance guarantees. Multiple network sliceswith diverse performance requirements are embedded over acommon physical infrastructure [1]. This requires a flexibleand programmable network architecture, with abstraction onboth the plane and layer dimensions [2]. Software-definednetworking (SDN) brings the plane-dimension abstraction bydecoupling the data and control planes. With a global net-work view and flow awareness brought by SDN, end-to-end(E2E) data delivery paths can be dynamically established andresources are explicitly allocated to different paths by an SDNcontroller. Network function virtualization (NFV) provides the

This work was financially supported by research grants from HuaweiTechnologies Canada and from the Natural Sciences and Engineering ResearchCouncil (NSERC) of Canada.

Kaige Qu, Weihua Zhuang, and Xuemin (Sherman) Shen are withthe Department of Electrical and Computer Engineering, University ofWaterloo, Waterloo, ON, Canada, N2L 3G1 (emails: {k2qu, wzhuang,sshen}@uwaterloo.ca).

Qiang Ye is with the Department of Electrical and Computer Engineeringand Technology, Minnesota State University, Mankato, MN 56001 USA (e-mail: qiang.ye@mnsu.edu). Corresponding author: Qiang Ye.

Xu Li and Jaya Rao are with Huawei Technologies Canada Inc., Ottawa,ON, Canada, K2K 3J1 (emails: {Xu.LiCA, jaya.rao}@huawei.com).

layer-dimension abstraction, by abstracting physical resourcesto virtual resources with a virtualization layer and realizingservice-level functionalities, referred to as virtual networkfunctions (VNFs) [2], [3]. Traditionally, service providersrely on dedicated hardware middleboxes to realize networkfunctions as in-path packet processing units required by aservice. Compared with middleboxes, VNFs are more cost-efficient and flexible for deployment and management. Severalframeworks have been proposed for SDN-NFV integration,to fully exploit their advantages and provide an integratedarchitecture with abstractions in both the plane and layerdimensions for customized service provisioning [2], [4].

With SDN-NFV integration, a tenant such as a serviceprovider requests network services in the form of servicefunction chains (SFCs). An SFC is composed of multipleVNFs in a predefined order, to fulfill a composite servicewith certain processing and transmission resource demands,according to service-level agreements (SLAs) negotiated withan infrastructure provider (InP). The resource demands areusually static and estimated from long-term traffic statisticsand quality-of-service (QoS) requirements [5]–[7]. The InPcustomizes network services over the physical infrastructure,generating network slices tailored for each service. However,with traffic load fluctuations during the operation of networkslices, the states of both processing and transmission resourcesalternate between overutilized and underutilized, causing tem-poral mismatch between traffic load and resource availability.Also, the imbalanced load distribution over both processingand transmission resources can result in local performancebottlenecks and inefficient resource usage at the same time,causing spatial mismatch between traffic load and resourceavailability. The temporal-spatial mismatch between trafficload and resource availability is detrimental to both serviceperformance and resource utilization. Therefore, how to over-come traffic load fluctuations for consistent service-level QoSguarantee requires further investigation [8], [9].

In this article, we propose a traffic engineering (TE) frame-work for efficient resource management among slices, bytaking into account traffic load dynamics. The goal of TEis to guarantee the E2E delay performance of each service,to achieve load balancing for long-term efficient resourceutilization, and to reduce the reconfiguration overhead. TheTE problem is formulated as a multi-objective optimizationproblem, and a heuristic algorithm is proposed to obtain a time-efficient solution. To support the proposed TE framework inservice-oriented 5G networks, the extended functionalities ofdifferent blocks in a reference network architecture with SDN-

2

Infrastructure SDN Controller

Virtualized

Infrastructure

Manager

(VIM)

NFV Infrastructure (NFVI) Domain

VNF Manager

(VNFM)

VNF 1 VNF 2 VNF 3

eMBB service

mMTC service

uRLLC service

Tenant SDN Controller

(VNF)

Tenant Domain

Service-level

forwarding rules

Operation/Business Support System

(OSS/BSS)

NFV Orchestrator

(NFVO)

Network Service Descriptor

NFV Management and Orchestration (MANO)

Resource Virtualization,

Allocation, and Abstraction

Physical network(Physical resource pool)

Edge switch

Edge switch

Edge switch

Virtual resource pool

Infrastructure-level

forwarding rules

Physical linkSDN switch NFV node with multiple VNFs Logical linkNFV node

Source Destination

Fig. 1: Network slicing and traffic engineering supported by an extended NFV MANO architecture with SDN integration.

NFV integration are discussed. A case study is presented forperformance evaluation of the proposed TE framework.

II. SYSTEM MODEL

A time-slotted system is considered, in which the timeline isdivided into time intervals with equal length. TE is performedfor a time interval with predicted QoS violations. We describethe system model in three parts, as illustrated in Fig. 1.

A. NFV Infrastructure Domain

1) Physical Network: A physical network consists of SDNswitches and NFV nodes interconnected by physical links.Switches forward traffic from incoming physical links tooutgoing physical links. Some switches act as edge switchesfor service access. NFV nodes, such as commodity serversand data centers (DCs), have both forwarding and processingcapabilities. The physical network contains a physical resourcepool, including transmission resources on physical links andprocessing resources at NFV nodes. A path in the physicalnetwork, i.e., a physical path, is composed of a series ofphysical links and SDN switches between two NFV nodes orbetween one NFV node and one edge switch. The maximumtransmission rate supported by a physical path is allocated bya central orchestrator.

2) Virtual Resource Pool: We call a logical abstraction ofall physical paths with pre-allocated transmission resourcesbetween two different NFV nodes or between one NFVnode and one edge switch as a logical link. The maximumtransmission rate supported by a logical link is the aggregatemaximum transmission rate over all its underlying physicalpaths. Transmission resources on logical links are seen asvirtual resources, since the mapping between logical links andphysical paths are transparent to service flows traversing thelogical links. With the consideration that processing resourceson NFV nodes can be distributed among several VNFs throughvirtualization, we introduce the concept of virtual resourcepool. A virtual resource pool with a certain topology isrepresented as a directed graph, in which vertices includeall NFV nodes and edge switches, and edges include alllogical links. The virtual resource pool is abstracted from thephysical resource pool, which ignores composition details ofthe physical paths associated with the logical links. It makesboth SDN switches and physical links fully transparent toservice flows on the logical links. A path in the virtual resourcepool, i.e., a virtual path, is composed of a series of logicallinks and NFV nodes between two edge switches. It is possiblethat the virtual resource pool is not a fully connected graph,i.e., not every two NFV nodes or edge switches are directlyconnected by a logical link. Assume that there are sufficienttransmission resources available in the physical network. We

3

can scale up/down resources on existing logical links, removean existing logical link, and find physical paths with sufficientresources for extra logical links. In this way, the topology ofthe virtual resource pool can be updated, which enables flexiblelogical link provisioning among the fixed NFV nodes.

3) Infrastructure SDN Controller: With SDN, packet for-warding rules are configured in SDN switches by an infrastruc-ture SDN controller to route traffic flows over a physical path.For logical link provisioning, the infrastructure SDN controlleris responsible for 1) configuring forwarding rules on physicalpaths associated with each logical link, and 2) enforcing atraffic splitting ratio among corresponding physical paths foreach logical link. When a topology update for the virtualresource pool is required, the infrastructure SDN controller isresponsible for (re-)configuring forwarding rules on physicalpaths for the scaled and extra logical links, and removing thoseassociated with the removed logical links.

B. Tenant Domain

1) Services: A service request is represented as an SFCwith specified QoS requirements, including average E2E delayrequirement and maximal tolerable downtime in one serviceinterruption. There are two levels of connectivity in an SFC,namely, service-level and infrastructure-level. The service-level connectivity requires that VNFs be chained in a pre-defined order between the source and destination nodes (fixedat edge switches), to facilitate the E2E service delivery. Theservice-level connectivity is achieved by mapping an SFCto a virtual path between the source and destination nodes.For two neighboring VNFs in an SFC, packets processedby the upstream VNF are transmitted to the downstreamVNF, generating traffic between consecutive VNFs, i.e., inter-VNF subflows. The infrastructure-level connectivity requiresthat each subflow be routed over at least one physical path,if its upstream and downstream VNFs are not co-located.The infrastructure-level connectivity is achieved by mappingeach subflow to a logical link which is provisioned via theinfrastructure SDN controller.

2) Tenant SDN Controller: The tenant SDN controllerconfigures service-level forwarding rules at edge switches andNFV nodes to guide packets belonging to a flow traversingan SFC (i.e., an SFC flow) through a virtual path, thusenabling the service-level connectivity. In the presence oftraffic variations, an SFC flow can be rerouted to an alternativevirtual path via the tenant SDN controller, according to TEdecisions made by a central orchestrator.

C. SDN-NFV Integration

An NFV management and orchestration (MANO) architec-ture can efficiently manage the life cycle of network functions,services, and their constituent resources in a common NFVinfrastructure (NFVI) [1]. The architecture is extended withSDN integration to realize service function chaining andprovide TE architectural support [1]. In the following, wediscuss the main functional blocks in the architecture and theirinteractions with the tenant and infrastructure SDN controllers.

1) Virtualised infrastructure manager (VIM) is responsiblefor managing resources in the NFVI. Specifically, the VIM

deals with resource virtualization and allocation, and maintainsthe mapping between the virtual resource pool and physicalresource pool. The VIM is also in charge of logical linkprovisioning via an infrastructure SDN controller;

2) VNF manager (VNFM) is in charge of the life cyclemanagement of VNFs, including instantiation, configuration,and scaling. In addition to VNFs serving as network servicecomponents, the tenant SDN controller is regarded as a VNF;

3) NFV orchestrator (NFVO), which is responsible forcentral orchestration, contains a resource orchestrator (RO) anda network service orchestrator (NSO). The RO is responsiblefor orchestrating NFVI resources. For TE, the RO containsan engine to make dynamic TE decisions. Specifically, itdetermines the rerouted virtual paths for SFC flows, includingboth the VNF to NFV node remapping and the consequentsubflow to logical link remapping. It also determines thelogical link to physical path remapping, to facilitate logicallink provisioning. The NSO is responsible for the life cyclemanagement of network services, including service instantia-tion and dynamic network service capacity scaling. For TE, ittriggers TE requests to the RO when potential QoS violationsare predicted, due to traffic load fluctuations.

III. TRAFFIC ENGINEERING FOR SFCS WITHIN VIRTUALRESOURCE POOL

A. Overview

Traffic engineering (TE) has been extensively investigated intraditional networks, to find paths for traffic transmission fromsource to destination within link capacity. However, traditionalTE methods cannot be directly applied in service-oriented5G networks due to the following reasons. First, an SFCflow requires two-dimensional (processing and transmission)resource provisioning, and the potential mismatch betweenthe two-dimensional resources should be addressed. Second,the transfer of VNF states should be considered, since simplyrerouting in-progress flows on a state-dependent VNF (i.e., aVNF in which states are stored and updated locally togetherwith packet processing) to an alternative NFV node introducesstate inconsistency and processing inaccuracy [10]. Here, weconsider TE for SFCs in a virtual resource pool, and focus onthe service-level connectivity. How to map a logical link tophysical paths is not the focus of this work. We rely on theNFVO to perform such tasks, using typical TE methods suchas solving a multi-commodity flow problem. A TE decisionwithin the virtual resource pool determines the remappingbetween VNFs and NFV nodes. Since the logical link to whicha subflow is mapped is uniquely determined by the locationsof the corresponding upstream and downstream VNFs, thesubflow to logical link remapping is a byproduct of a TE deci-sion. However, to maintain the infrastructure-level connectivityof SFCs, the potential topology update requirements for thevirtual resource pool is considered as an overhead for TE.

B. Elastic VNF Provisioning

1) Traffic Rate, Packet Processing Rate, and Processing Re-sources: With traffic load fluctuations, the processing resourcedemand of each VNF should be determined to guaranteeat least an average E2E processing delay for an SFC flow.

4

(a) (b)

Edge switch 1

Edge switch 3

Edge switch 2A B

CSFC 1

SFC 2

Edge switch 1

Edge switch 3

Edge switch 2A B

CSFC 1

SFC 2

NFV node with multiple VNFs

Unchanged logical link

Extra logical link

Scaled logical link

Total amount of resources available at an NFV node

Amount of resources allocated to SFC 2

Amount of resources allocated to SFC 1

Fig. 2: An illustration for traffic engineering in a virtual resource pool with topology update: (a) before traffic engineering; (b) after traffic engineering.

Consider that the E2E processing delay requirement is initiallydecomposed into per-hop delay requirements on each VNF,and that TE is performed in a time scale much larger thanpacket inter-arrival time of a traffic flow. Packet arrivals ofan SFC during a sufficiently large time interval are modeledas a Poisson process, with different rates (in packet/s) acrossdifferent time intervals. Assume that VNF packet processingtime is exponentially distributed. Then, packet processing at aVNF is modeled as an M/M/1 queue1, with different arrivaland service rates across different time intervals. Hence, theper-interval packet processing (service) rate demand of a VNFcan be calculated based on the Little’s Law. Given processingresources, the maximum packet processing rate supportedby a VNF depends on both the service (e.g., packet size,service type, function type) and platform (e.g., packet I/Oand virtualization technology at an NFV node). We use aprocessing density (in cycle/packet) to represent the processingresource demand (in cycle/s) of a VNF at an NFV node,corresponding to one packet/s in processing rate. With thequeueing and processing density models, the relationshipsamong the traffic rate, processing rate, processing resources,and the VNF processing delay requirement can be established.Then, the time-varying processing resource demand of a VNFat an NFV node can be determined, to satisfy the delayrequirement with a time-varying traffic rate.

2) Processing Resource Sharing: Consider multiple-to-onemapping between VNFs and NFV nodes. To achieve pro-cessing resource sharing among multiple VNFs from differentSFCs at an NFV node, a CPU polling scheme is employed.Each VNF gets a portion of CPU processing resources, whichis linear with the allocated CPU time share in a polling period.Based on the generalized processor sharing (GPS) discipline,the VNFs are guaranteed minimum processing resources (incycle/s). The percentage of the total allocated CPU time in aCPU polling period is defined as the NFV node loading factor.

3) Joint VNF Migration and Vertical Scaling: NFV enableselastic scaling of processing resources allocated to VNFs ina cost-effective manner, which facilitates agile service provi-sioning and management. Dynamic VNF operations, includinghorizontal scaling, vertical scaling, and migration are widelyemployed to provide elastic VNF provisioning [11]. With

1The M/M/1 queue model is used for simplicity of the presentation.

horizontal scaling, the number of instances for a VNF isscaled in/out, with constant amount of processing resources foreach instance. With vertical scaling, the amount of processingresources for a VNF instance is scaled up/down. With VNFmigration, a VNF instance migrates to an alternative NFVnode, without changing the amount of processing resources.To avoid overloading and achieve load balancing, we considerjoint VNF migration and vertical scaling, under the assump-tion that a VNF is instantiated once, which means that aVNF can migrate to an alternative NFV node with sufficientresources to satisfy its processing resource scaling demand.Fig. 2 illustrates joint VNF migration and vertical scalingfor two SFCs, with each SFC composed of two VNFs. Weuse rectangle height to represent the amount of processingresources. Specifically, one VNF of SFC 1 migrates from NFVnode B to NFV node C, and all VNFs are vertically scaled.

4) Redistribution of VNF Processing Delay Requirements:Another dimensionality of elasticity comes from redistributionof VNF processing delay requirements, since the E2E pro-cessing delay requirement of an SFC is satisfied as long asthe aggregation of all VNF processing delays in an SFC doesnot exceed a specified upper bound.

C. Flexible logical link Provisioning

Assume that there are sufficient transmission resources andno queueing on logical links. For a subflow, if its upstreamor downstream VNF migrates to an alternative NFV node, itshould be re-mapped to an alternative logical link accordingly,and resources on the original logical link should be released.However, it is possible that the virtual resource pool is notfully connected and extra logical links are required. As shownin Fig. 2, a subflow of SFC 1 is released from logical linkA → B, and re-mapped to an extra logical link A → C.Accordingly, the transmission rate over logical link A → Bcan be scaled down. In this way, logical links are flexiblyprovisioned, which addresses the potential mismatch betweenprocessing resources at fixed NFV nodes and transmissionresources on existing logical links. The extra logical links forflow rerouting incur signaling overhead between the infrastruc-ture SDN controller and SDN switches, due to forwarding rulereconfiguration along the underlying physical paths.

5

D. Parallel VNF State Transfer

Typical examples of state-dependent VNFs include networkaddress translators that store mappings between ports andhosts, and intrusion detection systems that keep track of patternmatchings for accurate attack detection. Some frameworkssuch as OpenNF are proposed to solve the state inconsistencyproblem, by not only moving packets of the rerouted SFCflow but also transferring the associate VNF states [10].Packet processing is halted during state transfer, causing aservice downtime. For a VNF state transfer, the product ofstate transfer time and transmission rate is equal to the statesize [12]. For a remapped SFC with multiple state transfers, weuse parallel state transfer in which all state transfers take placesimultaneously in the data plane, thus reducing the transferringlatency at the cost of transmission resource overhead [13]. Theservice downtime with parallel state transfer is the maximumstate transfer time along the E2E path, instead of the total timefor sequential state transfer. Under the assumption that a TEtime interval is sufficiently long, the service downtime is muchshorter than the stable service operation time for any service.

A state transfer belonging to one service can share a logicallink with subflows from other services. For example, as shownin Fig. 2, a state transfer from SFC 1 and a subflow from SFC2 both happen on logical link B → C. Generally, since ser-vices experience different downtimes, transmission resourcesallocated to subflows can be opportunistically used by statetransfers before the subflows resume packet forwarding. How-ever, for the example in Fig. 2, transmission resources allocatedto the subflow cannot be used by the state transfer, since SFC2 experiences no service downtime. Thus, the transmissionrate over logical link B → C should be scaled up. In theworst case, all state transfers happen on logical links where nosubflows are mapped or no transmission resources allocated tosubflows can be opportunistically used. In this case, dedicatedlogical links should be established for state transfers, or thetransmission resource capacity of existing logical links shouldbe scaled up to support state transfers. Therefore, we considerthe total amount of transmission resources required by statetransfers as a TE overhead.

IV. PROBLEM DEFINITION AND HEURISTIC SOLUTION

A. Problem Definition

We consider multiple SFCs in a processing resource limitednetwork. Given predicted traffic rate variations, a delay-awareTE problem within the virtual resource pool is to 1) find theremapping between VNFs and NFV nodes, and 2) verticallyscale the amount of processing resources allocated to VNFs,to satisfy the average E2E (processing) delay requirementswithout violating the maximal tolerable service downtime.

1) Objective: As discussed in Subsections III-C and III-D,the reconfiguration overhead in TE consists of two parts: thesignaling overhead for configuring extra logical links requiredfor flow rerouting, and the transmission resource overheadincurred by state transfers. Assume that the total signalingoverhead has a linear relation with the number of extra logicallinks. Besides, under the assumption that all VNF stateshave the same size and all services have the same downtime

limits, the amount of transmission resources required by theparallel state transfers linearly varies with the total number ofstate transfers, i.e., the number of VNF migrations. Hence,the minimization of reconfiguration overhead in TE leadsto limited modification to the original VNF to NFV nodemapping. In this case, the loads on different NFV nodescan be imbalanced, which can result in more migrations inthe subsequent time intervals. A balanced load distributionmakes the network more tolerant of future demand changes,which is beneficial for achieving long-term efficient resourceutilization [14]. Therefore, we formulate the TE problem asan optimization problem, by jointly considering the reconfig-uration overhead and load balancing. The VNF to NFV noderemapping is denoted by a set of binary decision variables,to represent whether a VNF is mapped to an NFV nodeor not. The VNF processing resource allocation is denotedby a set of continuous decision variables, to represent theamount of resources allocated to a VNF at an NFV node.The average processing delay for the VNFs at the NFV nodesare represented by a set of continuous decision variables. Theobjective function to be minimized is a weighted sum of thenumber of extra logical links, the number of VNF migrations,and the maximum NFV node loading factor. Both the numbersof extra logical links and VNF migrations are dependent onthe VNF to NFV node remapping decision variables. Themaximum NFV node loading factor is dependent on the VNFprocessing resource allocation decision variables.

2) Constraints: The multiple-to-one mapping betweenVNFs and NFV nodes is enforced by a constraint on the binaryVNF to NFV node remapping decision variables. The loadingfactors of all NFV nodes should not be beyond an upperlimit ηU , e.g., 0.95. There is also a relationship constraint onthe VNF to NFV node remapping decision variables and theVNF processing resource allocation decision variables, sincethe amount of resources allocated to a VNF at an NFV nodeshould be zero if the VNF is not mapped to the NFV node.The average processing delay for a VNF at an NFV node isexpressed based on M/M/1 queueing model. Then, the averageE2E (processing) delay of an SFC, can be represented by asummation of all average VNF processing delays in the SFC,which should not exceed the average E2E delay requirement.

B. Heuristic Algorithm

The formulated TE problem is an NP-hard mixed inte-ger non-convex optimization problem. For time tractability,a heuristic algorithm is proposed to obtain a sub-optimalsolution, with a flowchart given in Fig. 3. Consider that theE2E (processing) delay requirement for each SFC is initiallydecomposed into a set of per-hop processing delay require-ments at the VNFs (i.e., hop delay bounds). We first calculateNFV node loading factors with predicted traffic variations andthe initial hop delay bounds, based on the M/M/1 delay modeland processing density model, for the initial VNF to NFVnode mapping. The calculated NFV node loading factors canbe even larger than 1. By comparing the calculated NFV nodeloading factors with threshold ηth (initial value set as ηU ), a setof overloaded NFV nodes is identified as potential bottlenecks.

6

Input: predicted traffic variation

Initialize: VNF to NFV node mapping, hop delay bounds, NFV node loading factor threshold

Redistribution of hop delay bounds

Migration required?

Threshold reaching precision?

Output: sub-optimal traffic engineering solution

Threshold updating via binary search

Reconfiguration overhead aware

migration decision

Y

N

Y

N

Fig. 3: A flowchart of the proposed heuristic algorithm.

1) Redistribution of hop delay bounds: Even if potentialbottlenecks are identified, migration may not be necessary.For a given threshold, ηth, how an E2E delay requirementis decomposed into hop delay bounds affects the number ofoverloaded NFV nodes. By making hop delay bounds lessstringent on overloaded NFV nodes and more stringent onunderloaded ones, it is possible to reduce the number ofoverloaded NFV nodes. The basic idea is as follows: if anSFC traverses both overloaded and underloaded NFV nodes,loading factors of the underloaded ones are increased to ηth,by reducing corresponding hop delay bounds, and loadingfactors of the overloaded ones are decreased, by increasingcorresponding hop delay bounds. This strategy is referred toas delay scaling, which is performed iteratively until there is noSFC traversing both overloaded and underloaded NFV nodes.The iterative delay scaling procedure with given threshold, ηth,is referred to as the redistribution of hop delay bounds.

2) Reconfiguration overhead reduction: A redistribution ofhop delay bounds with the initial threshold (ηth = ηU )is performed after the initialization step. If the number ofoverloaded NFV nodes is reduced to zero, no migration isrequired. Otherwise, migration is necessary to overcome trafficoverloading. Migration decisions are made sequentially, eachfollowed by a redistribution of hop delay bounds with theinitial threshold, until no more migration is required. Withalternate migration decision and redistribution of hop delaybounds, reconfiguration overhead is greedily reduced in twoways. One is the potential reduction of overloaded NFV nodes.The other is the consideration of reconfiguration overhead inmigration decision. A migration decision includes three steps,i.e., identification of a bottleneck NFV node, selection of anSFC to migrate, and selection of a target NFV node. First, themost heavily loaded NFV node is identified as the bottleneck.Next, an SFC to migrate from the bottleneck NFV node and atarget NFV node to accommodate the migrated SFC are jointlyselected to minimize the reconfiguration overhead, i.e., 1 plusthe number of extra logical links for flow rerouting. If thereare multiple choices, an SFC with the largest resource demandis migrated to the closest target NFV node.

3) Load balancing: After the sequential migration decisionprocedure, all NFV node loading factors are less than or equalto the initial threshold ηU . The gap between the smallestand largest NFV node loading factors can be large, whichis undesired in terms of load balancing. Actually, if one SFCtraverses two NFV nodes with different loading factors, its hopdelay bound can be relaxed at the NFV node with the largerloading factor and be shrunk on the other NFV node, to reducethe gap between the two loading factors. Based on this idea,an iterative procedure with alternate threshold updating andredistribution of hop delay bounds is performed to graduallyreduce the gap and to balance the NFV node loading factors.The threshold, ηth, is first reduced stepwise from the initialvalue ηU , with an initial step size, until some overloaded NFVnodes are detected, after redistribution of hop delay boundswith the updated threshold. The emergence of overloaded NFVnodes indicates that the latest step of threshold reduction istoo aggressive. Then, a binary search between the latest twothreshold values is performed, until no overloaded NFV nodesare detected and a sufficient precision is reached.

V. A CASE STUDY

A case study is presented to evaluate the performance ofthe proposed TE heuristic algorithm, within a 64-node meshvirtual resource pool in which logical links exist only betweenneighboring nodes. We consider homogeneous processing den-sities and a maximum NFV node processing rate of 1000packet/s. There are 3 SFCs, each with 4 VNFs, initiallymapped to the virtual resource pool. SFC 3 shares two NFVnodes with SFC 1 and one NFV node with SFC 2. The averageE2E delay requirement and the maximal tolerable servicedowntime for each SFC are 20ms and 5ms, respectively. TheVNF state size is a constant, equal to 10 bytes. All SFCs havean initial traffic rate of 200 packet/s during the previous timeinterval (k−1). Denote the predicted traffic load during currenttime interval k for SFC s as λ(s)(k). We have λ(2)(k) = 200packet/s, and vary both λ(1)(k) and λ(3)(k) from 200 packet/sto 780 packet/s. The initial step size and the precision forthreshold updating are set to 0.1 and 0.0001, respectively.

A. Load Balancing and Reconfiguration Overhead Trade-off

Fig. 4 shows the TE performance with the increase ofλ(1)(k) and λ(3)(k). Performance metrics include the maxi-mum NFV node loading factor, η(k), the number of migra-tions, N(k), and the number of extra logical links, S(k), forflow rerouting. We see a zigzag trend for η(k) and step-wise in-creasing trends for both N(k) and S(k). In Fig. 5, the relation-ships among the three performance metrics with the increaseof λ(3)(k) are illustrated, for λ(1)(k) = 280, 480, 700 packet/s,respectively. We observe that η(k) drops sharply when N(k)or S(k) is increased by 1. When N(k) and S(k) are stable,η(k) shows either a linear increasing trend or a flat trend. Thelinear increasing trend indicates the dominance by λ(3)(k),while the flat trend indicates the dominance by λ(1)(k). Therelationships among η(k), N(k), and S(k) demonstrate thetrade-off between load balancing and reconfiguration overhead.

7

0.7800

0.8

800

(k)

600

0.9

(3)(k)

600

(1) (k)

1

400 400200 200

(a)

0800

1

800

N(k

)

600

2

(3)(k)

600

(1) (k)

3

400 400200 200

(b)

0800

800

0.5

S(k

)

600

(3)(k)

600

(1) (k)

1

400 400200 200

(c)

Fig. 4: Performance metrics of the proposed TE framework with respect to λ(1)(k) and λ(3)(k): (a) η(k); (b) N(k); (c) S(k).

200 300 400 500 600 700

(3)(k)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(k)

(1)(k)=280 packet/s

0

1

2

3

4

5

6

7

8

N(k

) o

r S

(k)

(k)

N(k)

S(k)

(a)

200 300 400 500 600 700

(3)(k)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(k)

(1)(k)=480 packet/s

0

1

2

3

4

5

6

7

8

N(k

) o

r S

(k)

(k)

N(k)

S(k)

(b)

200 300 400 500 600 700

(3)(k)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(k)

(1)(k)=700 packet/s

0

1

2

3

4

5

6

7

8

N(k

) o

r S

(k)

(k)

N(k)

S(k)

(c)

Fig. 5: Performance trade-off with the increase of λ(3)(k) for: (a) λ(1)(k)=280 packet/s; (b) λ(1)(k)=480 packet/s; (c) λ(1)(k)=780 packet/s.

TABLE I: Average E2E delay performance

Traffic arrival Poisson MMPP

Rate (packet/s) 300 400 500 600 700 R1 R2 R1 R2 R1 R2 R1 R2

350 250 450 150 500 100 550 50

Average delay after TE (ms) 19.3 19.7 19.5 19.8 19.7 20.6 22.7 23.3 25.7

B. Delay and Impact of Traffic Burstiness

We carry out packet-level simulations using network simula-tor OMNeT++ to evaluate the E2E delay after TE. To verify theeffectiveness of our TE model, not only Poisson packet arrivalsbut also MMPP packet arrivals are simulated. We use a two-state MMPP model with a same transition rate between thetwo states and an average traffic rate of R1+R2

2 packet/s, withR1 and R2 being the individual average packet arrival rates atthe two states. A larger gap between R1 and R2 indicates ahigher level of traffic burstiness. The average delay of Poissontraffic arrivals with average rates of 300, 400, 500, 600, 700packet/s and MMPP traffic arrivals with an average rate ofR1+R2

2 = 300 packet/s after TE are given in Table I. Weobserve that the E2E delay requirement (20ms) is satisfiedfor all Poisson traffic arrivals. However, with the increase oftraffic burstiness for the MMPP traffic arrivals, the E2E delayperformance degrades. How to incorporate traffic burstiness inthe TE framework remains our future work.

VI. CONCLUSION

In this article, we present a traffic engineering frameworkwithin an NFV/SDN architecture in service-oriented 5G net-works, to achieve consistent QoS provisioning for multipleservice-level network slices in the presence of traffic variations.

We consider services in the form of SFCs in which VNFs arechained to fulfill a composite service delivery. The TE problemis formulated as a multi-objective mixed integer optimizationproblem, and a time-efficient heuristic algorithm is presented.In the case study, the performance of the proposed TE heuristicalgorithm is evaluated, demonstrating a trade-off betweenload balancing and reconfiguration overhead. A packet-levelsimulation is performed to show the delay performance ofdifferent traffic arrivals with the proposed TE model.

REFERENCES

[1] J. Ordonez-Lucena, P. Ameigeiras, D. Lopez, J. J. Ramos-Munoz,J. Lorca, and J. Folgueira, “Network slicing for 5G with SDN/NFV:Concepts, architectures, and challenges,” IEEE Commun. Mag., vol. 55,no. 5, pp. 80–87, May 2017.

[2] Q. Duan, N. Ansari, and M. Toy, “Software-defined network virtual-ization: an architectural framework for integrating SDN and NFV forservice provisioning in future networks,” IEEE Netw., vol. 30, no. 5, pp.10–16, Sept. 2016.

[3] C. Lorenz, D. Hock, J. Scherer, R. Durner, W. Kellerer, S. Gebert,N. Gray, T. Zinner, and P. Tran-Gia, “An SDN/NFV-enabled enterprisenetwork architecture offering fine-grained security policy enforcement,”IEEE Commun. Mag., vol. 55, no. 3, pp. 217–223, Mar. 2017.

[4] ETSI NFV ISG, “NFV-EVE005: SDN usage in NFV architecturalframework,” Oct. 2015.

[5] O. Alhussein, P. T. Do, J. Li, Q. Ye, W. Shi, W. Zhuang, X. Shen, X. Li,and J. Rao, “Joint VNF placement and multicast traffic routing in 5Gcore networks,” in Proc. IEEE GLOBECOM, Dec. 2018, pp. 1–6.

8

[6] Q. Ye, W. Zhuang, X. Li, and J. Rao, “End-to-end delay modeling forembedded VNF chains in 5G core networks,” IEEE Internet of ThingsJ., vol. 6, no. 1, pp. 692–704, Feb. 2019.

[7] Q. Ye, J. Li, K. Qu, W. Zhuang, X. Shen, and X. Li, “End-to-end qualityof service in 5G networks – Examining the effectiveness of a networkslicing framework,” IEEE Veh. Technol. Mag., vol. 13, no. 2, pp. 65–74,June 2018.

[8] K. Qu, W. Zhuang, Q. Ye, X. Shen, X. Li, and J. Rao, “Delay-awareflow migration for embedded services in 5G core networks,” in Proc.IEEE ICC, May 2019, pp. 1–6.

[9] Y. Jia, C. Wu, Z. Li, F. Le, A. Liu, Z. Li, Y. Jia, C. Wu, F. Le, andA. Liu, “Online scaling of NFV service chains across geo-distributeddatacenters,” IEEE/ACM Trans. Netw., vol. 26, no. 2, pp. 699–710, Apr.2018.

[10] A. Gember-Jacobson, R. Viswanathan, C. Prakash, R. Grandl, J. Khalid,S. Das, and A. Akella, “OpenNF: Enabling innovation in networkfunction control,” in Proc. ACM SIGCOMM, Aug. 2014, pp. 163–174.

[11] M. Ghaznavi, A. Khan, N. Shahriar, K. Alsubhi, R. Ahmed, andR. Boutaba, “Elastic virtual network function placement,” in Proc. IEEECloudNet, Oct. 2015, pp. 255–260.

[12] J. Xia, D. Pang, Z. Cai, M. Xu, and G. Hu, “Reasonably migrating virtualmachine in NFV-featured networks,” in Proc. IEEE Conf. Computer andInformation Technology, Dec. 2016, pp. 361–366.

[13] B. Zhang, P. Zhang, Y. Zhao, Y. Wang, X. Luo, and Y. Jin, “Co-scaler:Cooperative scaling of software-defined NFV service function chain,” inProc. IEEE Conf. Network Function Virtualization and Software DefinedNetworks, Nov. 2016, pp. 33–38.

[14] L. Guo, J. Pang, and A. Walid, “Dynamic service function chaining inSDN-enabled networks with middleboxes,” in Proc. IEEE ICNP, Nov.2016, pp. 1–10.

Kaige Qu (S’19) received her B.S. degree in com-munication engineering from Shandong University,Jinan, China, in 2013. She received her M.S. degreesin integrated circuits engineering and electrical en-gineering from Tsinghua University, Beijing, China,and KU Leuven, Leuven, Belgium, respectively, in2016. She is currently pursuing her Ph.D. degreewith the Department of Electrical and ComputerEngineering, University of Waterloo, Waterloo, ON,Canada. Her research interests include resource al-location in SDN/NFV-enabled networks, 5G and

beyond, and machine learning for future networking.

Weihua Zhuang (M’93−SM’01−F’08) has beenwith the Department of Electrical and ComputerEngineering, University of Waterloo, Canada, since1993, where she is a Professor and a Tier I CanadaResearch Chair in wireless communication networks.She was a recipient of the 2017 Technical Recogni-tion Award from the IEEE Communications SocietyAd Hoc and Sensor Networks Technical Committee,and several best paper awards from IEEE confer-ences. Dr. Zhuang is a Fellow of the Royal Societyof Canada, Canadian Academy of Engineering, and

Engineering Institute of Canada. She is an Elected BoG Member and VPPublications of the IEEE Vehicular Technology Society.

Qiang Ye (S’16−M’17) has been an assistant pro-fessor with the Department of Electrical and Com-puter Engineering and Technology, Minnesota StateUniversity, Mankato, MN, USA, since September2019. He received his Ph.D. degree in electricaland computer engineering from the University ofWaterloo, Waterloo, ON, Canada, in 2016. He hadbeen with the Department of Electrical and Com-puter Engineering, University of Waterloo, as a post-doctoral fellow and then a research associate fromDecember 2016 to September 2019. His current

research interests include 5G networks, SDN/NFV, network slicing, AI andmachine learning for future networking.

Xuemin (Sherman) Shen (M’97−SM’02−F’09) iscurrently a University Professor with the Departmentof Electrical and Computer Engineering, Universityof Waterloo, Canada. His research focuses on net-work resource management, wireless network secu-rity, social networks, 5G and beyond, and vehicularad hoc networks. Dr. Shen is a Canadian Academyof Engineering Fellow, a Royal Society of CanadaFellow, a Chinese Academy of Engineering ForeignFellow. He received the R.A. Fessenden Award in2019 from IEEE, Canada, James Evans Avant Garde

Award in 2018 from the IEEE Vehicular Technology Society, Joseph LoCiceroAward in 2015 and Education Award in 2017 from the IEEE CommunicationsSociety.

Xu Li is a senior principal researcher at HuaweiTechnologies Canada. He received a PhD (2008) de-gree from Carleton University, an MSc (2005) degreefrom the University of Ottawa, and a BSc (1998)degree from Jilin University, China, all in computerscience. Prior to joining Huawei, he worked as aresearch scientist (with tenure) at Inria, France. Hiscurrent research interests are focused in 5G andbeyond. He contributed extensively to the develop-ment of 3GPP 5G standards through 90+ standardproposals. He has published 100+ refereed scientific

papers and is holding 40+ issued US patents.

Jaya Rao received his B.S. and M.S. degrees inElectrical Engineering from the University of Buf-falo, New York, in 2001 and 2004, respectively, andhis Ph.D. degree from the University of Calgary,Canada, in 2014. He is currently a Senior ResearchEngineer at Huawei Technologies Canada, Ottawa.Since joining Huawei in 2014, he has worked onresearch and design of CIoT, URLLC and V2Xbased solutions in 5G New Radio. He has contributedfor Huawei at 3GPP RAN WG2, RAN WG3, andSA2 meetings on topics related to URLLC, network

slicing, mobility management, and session management.