Refactoring Network Functions Modules to Reduce Latencies ...

0733-8716 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JSAC.2018.2869965, IEEE Journalon Selected Areas in Communications

1

Refactoring Network Functions Modules to ReduceLatencies and Improve Fault Tolerance in NFV

Muhammad Taqi Raza, Member, IEEE, Songwu Lu, Fellow, IEEE, Mario Gerla, Fellow, IEEE, and Xi Li

Abstract—Network Functions Virtualization (NFV) allowsservice providers to deliver new services to their customersmore quickly by adopting software centric network functionsimplementation over commercial, off-the-shelf hardwares. ThisNFV based software-centric approach cannot use dedicatedmechanisms implemented over custom built boxes to reducelatencies and tolerate faults. We present a case study of IPMultimedia Subsystem (IMS), which is the most complex NFVinstance, requires extremely low end-to-end latency (40 msec),and demands system availability as high as five nines. Throughan empirical study, we discover that highly modular IMSnetwork functions implementation over virtualized platform (1)incurs latencies, and (2) does not tolerate faults. NFV-based IMSmodules incur high latencies by creating a feedback loop amongeach other while executing delay sensitive data-plane traffic.These IMS modules are also susceptible to failure, causing thecontrol-plane to terminate the application session while keepingthe data-plane to forward data packets. To address these issues,we propose to refactor network function modules. We reducelatencies by pipelining the IMS modules, and recover failedmodules by reconfiguring their neighboring modules. We buildour system prototype of open source IMS over OpenStackplatform. Our results show that our scheme reduces latenciesand failure recovery time upto 12× and 10×, respectively, whencompared to the stat-of-the-art virtualized IMS implementation.

Index Terms – Network Functions Virtualization, LTE, IPMultimedia Subsystem, Fault Tolerance, Software Defined Net-working.

I. INTRODUCTION

TO meet exponentially increasing service demands and toeven launch a new network service, a service provider

often requires installing a new dedicated appliance that bringscomplexity of integrating and deploying it in a network.Moreover, purpose-built appliances rapidly reach end of lifebecause dedicated hardwares life cycles are becoming shorteras innovation accelerates, reducing the return on investment ofdeploying new services [1]. Network Functions Virtualization

Manuscript received May 01, 2018; revised August 15, 2018; acceptedAugust 28, 2018; approved by IEEE Journal on Selected Areas in Com-munications – 2018 Special Issue on Network Softwarization & Enablers– Series Editor Tarik Taleb. An earlier version of this paper appeared inACM/IEEE/IFIP CNSM 2017.

M. Raza, S. Lu, and M. Gerla are with the Department of Computer Science,University of California, Los Angeles, CA 90095 USA (e-mail: [email protected]; [email protected]; [email protected]); X. Li is with the School ofSoftware Engineering, University of Science and Technology of China (e-mail: [email protected]).

The work of Mr. Muhammad Taqi Raza and Dr. Songwu Lu was partiallysupported by NSF grants 1423576 and 1526985; Dr. Xi Li acknowledgesthe funding support of NSFC grants 61772482 and 61272131, and is thecorresponding author; M. Raza is a student author of this work.

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

(NFV) addresses these problems by implementing networkfunctions (NFs) in a software that can run on general-purposehardware servers [2]. NFV allows network service providers toscale services up or down quickly to address changing servicedemands while reducing capital and operational expenditures(CAPEX and OPEX) [3] [4].The move away from proprietary hardwares means virtualizednetwork systems cannot take advantage of special mechanismsto tolerate faults and reduce packet processing latencies. Thespecial mechanisms in proprietary hardware include (1) mod-ule redundancy by providing strong coupling between softwareand hardware [5] [6]; and (2) performing functional checksthrough proprietary software platforms [7] [8]. This motivatesus to study how virtualized network functions (VNFs) handlelatencies of growing number of users requests, and toleratefaults in the absence of vendor specific mechanisms.We consider Virtualized IP Multimedia Subsystem (vIMS) asa case study of NFV based network system implementation.There are several reasons behind our choice of vIMS as NFVimplementation use case. First, there are a variety of IMSbased applications (such as Voice over LTE (VoLTE), Videoover LTE (ViLTE), Conference Call, to name a few) used bymillions of subscribers in operational LTE networks [9]. Sec-ond, IMS based applications stipulate stringent requirementson both low latency and high fault tolerance. For example,VoLTE call requires end-to-end voice packets latency to be lessthan 100 msec while continuing voice call during system faults[10]. Third, IMS procedure is well standardized by 3GPP [11]standard body restricting all IMS implementations to followthe standard. Fourth, through open source IMS implementation(OpenIMS [12]), we can test the working of standardized NFsand their modules.We prototype vIMS by implementing OpenIMS over opensource cloud platform (OpenStack [13]), and we ensure thatvIMS implementation is in accordance to the IMS standard.Our prototype acts as a baseline vIMS implementation thatwe call state-of-the-art vIMS. We conduct an empirical studyon our vIMS implementation and record the observations asfollows:

1) IMS NFs are highly modular; one NF has many functionalmodules. During media plane processing, different modulesimplemented in three different NFs interact with each otherin a loop; where a module in one NF uses delegation modelto delegate the processing of data packets to a module indifferent NF, while retaining the overall control to itself.Similarly, policies are defined at a module in one NF, butare enforced at a module in different NF. As a result, a loop



2

is created between processor, controller and policy modulesin different NFs that require realtime packet processingbehavior to update control information and policies. Forexample, a controller module requires available bandwidth,and packets arrival rate from a processor module to adjustvoice codec [14]. This results in packet processing laten-cies.

2) During media session setup, both control and data planescreate device session states and transition from one stateto the other as a call progresses. Device states at controland data planes remain synchronized during call life cycle.However, we discover that a particular module failurecauses device states to be desynchronized; the control-plane terminates the call but data-plane keeps device stateas Connected. This hanging state machine phenomenonemerges when control-plane detects the failure and changesdevice state to Morgue before terminating the control-planeconnection; whereas, the failure goes undetected at data-plane that keeps receiving downlink data packets (whosecontrol-plane connection is aborted).

To address these issues, we propose to refactor IMS NFsmodules by (1) pipelining data packets processing and fetchingits control instructions, and (2) reconfiguring modules torecover from a failure. To reduce media plane latencies, ourdesign predicts future packets and prefetches control instruc-tions. Once future packets arrive, these control instructions areused to steer media plane execution. For prediction, we use ex-ponential smoothing model [15] that weighs past observationsusing exponentially decreasing weights, i.e. recent observa-tions are given relatively more weight in forecasting than theolder observations. Because control instructions remain samefor a range of metadata values (that represent media behavior),our design fetches correct future control instruction even fora small prediction error.To improve fault tolerance, our design provides configurationsfor each module during the system setup phase by addingthe back-up paths to their one-hop neighboring module. Atruntime, neighboring module detects a module failure andassumes the role of failed module by loading its executionlogic as provided in the configurations, then connects withrest of the failed module’s neighbors through a back-up linkand resumes failed operation.

We evaluate our design and gather results from our Open-IMS [12] implementation over Openstack [13]. Our resultsshow that (1) our system reduces media latencies upto 12×,and (2) resumes failed operation within 2.5 seconds, which is10× better than current state-of-the-art vIMS design.

II. MIGRATING CARRIER GRADE NFS TO NFV

Network systems heavily rely on advanced in-network pro-cessing for critical functions ranging from traffic managementto VoLTE service. These Network Functions (NFs) are im-plemented over dedicated hardware boxes which are spreadwithin the network. Recently, the motivation to reduce sys-tems’ operational and capital expenditures with exponentiallyincreasing traffic growth inspire building network systems(NFs) using commodity compute/storage/network resources

while reducing (or removing) dependency on specialized plat-forms. Migrating carrier grade boxes functionality towardsvirtualized middleboxes is called Network Function Virtual-ization, NFV. Such software only implementation of NF(s)decoupled from hardware also lead to more agile and flexibledeployment models.However, implementing task specific network systems overvirtualized platform incurs latencies; The time critical appli-cations suffer the most whose latencies were previously con-tained by dedicated box mechanisms. Latencies in NFV mainlycome from virtualization layer, lack of system support forproviding enough resources to time critical applications, ser-vice chaining between different NFs, and more. Furthermore,failures are common in virtualized environment [16], whichare caused by single server or node failure, misconfiguredNFs, software patch or upgrade glitches, and due to networkcongestion or overload [17].To address these issues, both industry and researchers pro-posed very generic solutions, such as VNF placement ac-cording to application need [18], using baremetal approachto reduce latencies [19], adding 1:N redundancy [17] andothers. However, we argue that such generic solutions donot guarantee application-specific QoS in terms of enduredlatencies and expected system performance. We advocate toapply domain specific knowledge in improving a particularsystem and deliver application requirements. Therefore, inthis work, we focused on an NFV system by considering itsassociated application needs, the interactions between differentNFs during the system operation, and system requirementsstipulated by the standard.

III. MOTIVATION OF USING VIRTUALIZED IMS AS NFVIMPLEMENTATION CASE-STUDY

NFV is keenly followed by the telecom industry and itsproof of concept implementations are already in process [20].Most telecom operators are considering to support their IPMultimedia Subsystem (IMS) implementation over NFV [21].IMS is an architectural framework for delivering IP multi-media services, such as VoLTE, ViLTE, software as a service(SaaS), social sites, navigation, and many more. Operators canbenefit from virtualized IMS (vIMS) implementation; wherethey can quickly scale up virtualized NFs (VNFs) to supportgrowing demand of multimedia-rich applications and services(e.g. VoLTE and ViLTE). These multimedia services havestringent requirements on latency (end-to-end latency goes aslow as 40 msec [10]) and fault tolerance (operator networkswant to achieve 5 nines availability [22]). This motivates usto study how IMS can achieve such low latency by toleratingfaults over virtualized infrastructure. We are also motivatedto use IMS as a case-study because it already has an opensource implementation (OpenIMS [12]), and its functionalitiesare well documented by 3GPP specifications [11]. Lastly,like LTE, IMS supports both control-plane and data-planefunctions. That means, the lesson learned in IMS use casestudy can also be used in virtualizing LTE core.



3

IV. IMS BACKGROUND

IMS runs on top of LTE that provides best effort serviceto the users, with no guarantee on the amount of bandwidtha user gets for a connection and the delay experienced bythe packets. Therefore, IMS is the preferred choice of mobileoperators to support real-time multimedia services. IMS usesInternet protocols and brings multiple media, multiple pointof access and multiple modes of communication into a singlenetwork, enabling simultaneous voice and multimedia servicesfor end users [23].IMS Architecture: IMS operations are categorized intocontrol-plane and data-plane operations, as shown in Fig. 1.

Serving-CSCF

Media Control Model

Session Controller . . .

Media Resource ControllerMedia Control

Resource Controller . . .

Media Resource Processor

Media Processor

Media Update . . .

Application ServerMedia Policy

Application Logic . . .

Proxy-CSCF

Registrar

Authentication . . .PDN-Gateway

(LTE)

DeviceData-plane Control-plane

Fig. 1: IMS architecture: An overview

Control-Plane supports media sessions control through CallSession Control Function (CSCF) NFs, and Application Server(AS) NF. The CSCF performs all the signaling operations,manages Session Initiation Protocol (SIP) sessions and coor-dinates with other NFs for session control, service control andresource allocation. It consists of two main NFs: the Proxy-CSCF (P-CSCF) and Serving-CSCF (S-CSCF). The AS, onthe other hand, implements multimedia execution logic andpolicies, and coordinates with both CSCFs and data-plane NFs.LTE device (IMS client) first registers with LTE core networkand then initiates IMS signalling over IMS control-plane. TheP-CSCF is an access point for IMS and acts as a SIP proxy forall the user equipments. P-CSCF simply forwards all traffic toS-CSCF. S-CSCF is the core of the IMS and it is the point ofcontrol within the network that enables operators to controlthe entire service delivery process and all the sessions. S-CSCF forwards the request to AS that applies service logicin accordance to defined policy and replies back the modifiedsession to S-CSCF for its delivery to destination network.

Data-Plane includes Media Resource Function (MRF) thatprocesses, stores data and generates services for the sub-scribers. MRF functionality is further split into two NFsthat perform its control-plane, and data-plane actions. Onceuser session has been established, the user data-plane trafficis sent to Media Resource Function Processor (MRFP) –performing data-plane action of MRF. The MRFP connectsLTE core domain (via PDN gateway – PGW) with IMSdomain for multimedia service and converts between differenttransmission and coding techniques as controlled by MediaResource Function Controller (MRFC) – performing control-plane action of MRF. Moreover, MRFC employs monitoringschemes to determine policy rules in real-time.Note that, in order to improve the readability of the paper, weuse the term MRF when discussing the data-plane functionalityas a whole.

IMS NFs are highly modular where different modules han-dle different functionality such as execution logic, processing,policy, security, session states, resource control and more. Fig.1 shows different modules (rectangular shaped) implementedwithin different NFs (rounded rectangular shaped).

V. EMPIRICAL STUDY

Through an empirical study, we understand (1) how dif-ferent VNFs’ modules interaction adds latencies in media-plane (data-plane), and (2) key module failure impacts on-going multimedia traffic. We prototype open source vIMSimplementation by making significant changes into OpenIMS[12] platform. We develop various software modules afterstudying tens of IMS standard documents [24] [11] andobserve these modules’ interaction with each other for on-going media flow. The implementation details of our prototypeis discussed in Section VII. We use VoLTE as multimediaapplication, because VoLTE is a premier IMS applicationwidely used in operational LTE networks worldwide [25].Below, we describe each finding, its impact and then provideanalysis of the finding.

A. Media-plane latencies exponentially increase with call rate

We find that media-plane latencies significantly rise as weadd moderate number of simultaneous calls into the vIMSsystem. In reality, LTE network operators receive hundreds ofthousands of multimedia requests (including VoLTE calls) persecond [26]. Each of these media requests has stringent latencyrequirement as defined by LTE standard (refer to Table 6.1.7:Standardized QCI characteristics in [10]). For example, voice,video, push to talk voice, and IMS signaling have latencybounds of 100 msec; whereas interactive gaming and missioncritical jobs have latency bounds of 50 msec and 60 msec,respectively. For testing latency requirements, we launchedsimultaneous call requests by varying call rate from 2 callsper second to 300 calls per second. Fig. 2(left) shows that thepackets delay remain under 30 msec for first 100 calls persecond. The latency doubles to 60 msec when we add as fewas 25 more simultaneous calls; and reach upto upper bound ofVoLTE call’s acceptable value (100 msec) for only 180 callsper second. The delay increases up to 4 times of VoLTE call’sacceptable value (400 msec) by adding 0.5 times the existingnumber of calls (270 calls/sec). In our experiment, we did notterminate the already established calls because in reality IMSsystem should accept more number of calls on top of alreadyon-going calls. vIMS reaches its capacity to provide seamlessvoice service under 15,000 active calls in total, as shown inFig. 2(right).Impact: This experiment explains that state-of-the-art vIMSdesign fails to meet QoS requirements on media traffic [27] inoperational LTE network. Therefore, operators are required toinstall many more NFs instances to meet current subscribersdemand. This will increase their capital and operational ex-penditures (CAPEX and OPEX) which is against the NFVphilosophy. High latencies may also cause failures, whenvIMS completely stops responding because of system overload(by throwing too busy error) [28] and requires NFs reboot.



4

Delay in secondsRegistration Rate Probability10 0.01 1 999 0.00100110 0.01 2 0.00200210 0.01 3 0.00300310 0.01 4 0.00400410 0.01 5 0.00500510 0.01 6 0.00600610 0.01 7 0.00700710 0.01 8 0.00800820 0.02 9 0.00900920 0.02 10 0.0100120 0.02 11 0.01101120 0.02 12 0.01201220 0.02 13 0.01301320 0.02 14 0.01401420 0.02 15 0.01501520 0.02 16 0.01601620 0.02 17 0.01701720 0.02 18 0.01801820 0.02 19 0.01901920 0.02 20 0.0200220 0.02 21 0.02102120 0.02 22 0.02202220 0.02 23 0.02302320 0.02 24 0.02402420 0.02 25 0.02502520 0.02 26 0.02602620 0.02 27 0.02702720 0.02 28 0.02802820 0.02 29 0.02902920 0.02 30 0.0300320 0.02 31 0.03103120 0.02 32 0.03203220 0.02 33 0.033033

0100200300400500600

0 50 100 150 200 250 300

Traf

fic D

elay

(mse

c)

Call Rate (calls/sec)

Upper bound to avoid jitter

0

10000

20000

30000

40000

50000

0 50 100 150 200 250 300

Activ

e Ca

lls


Fig. 2: Even under the moderate call rate, the packets latenciesare beyond acceptable value. These latencies exponentially grow byadding fewer number of more calls and potentially clog the wholedata-plane

AS

Script Document

MRFC

Interpreter

DocumentRequest

XML Interpreter Context

MRFP

Script execution

Implementation Platform

Media plane loops – 2

(a) Script fetching loop

AS

Media Policy

MRFC

Media Control

Media updatePolicing input

Low level media focus

MRFP

Mixer

RTP/RTCP flow

Notification

Floor CTR

Processor

. . .

Media plane loops – 3

(b) multimedia execution loop

Fig. 3: Frequent interactions between different modules in data-planeexecution

Analysis: Frequent loops between media plane modulesThe root cause of above issues is due to frequent interactionsbetween different modules residing in different NFs. Theseinteractions form a loop and packet processing latencies soarto the level where handling of media packets is no longermeaningful (i.e. voice jitters with large packet delays). Evenworse, further increase in call rate cause packets congestionat different modules and render these module non-responsive.IMS media-plane functionality is divided among 3 NFs (AS,MRFC and MRFP). When AS NF receives originating callnotification, it sets-up media policy and informs MRFC toprepare network resources at MRFP. MRFC fetches mediaexecution script documents, located at AS and forwards it toMRFP script execution engine.When media (VoLTE data packets) starts arriving at MRFP,media packets are processed and call meta data (e.g. call arrivalrate, bandwidth usage, available buffer size etc.) is generated.This meta data is fed back to MRFC that adjusts call executionlogic (e.g. codec bit rate, codec sample size and codec intervaletc.) and informs MRFP. MRFP adjusts the number of packetsthat need to be transmitted every second (i.e. PPS = (codec bitrate) / (voice payload siz)) and the bandwidth (total packet size* PPS). This loop between different modules of MRFC andMRFP continues, as shown in Fig. 3b during media execution.

We find that on top of media execution loop, there existsscript fetching loop, as shown in Fig. 3a. This is mainly be-cause AS retains full media execution control by dynamicallygenerating XML scripts [29]. IMS employs web model wheremedia application behavior is defined in terms of markuplanguages/scripts (e.g. VoiceXML, SCXML, CCXML, and

others)1 [30] [31]. These scripts are located on the AS andretrieved by the MRFC using HTTP protocol. Scripts runningon the MRFP provide media behavior notifications to AS(via MRFC), and receive media control updates from AS.Furthermore, AS can exercise more fine-grained control of themedia behaviour by defining smaller scripts for the MRFC toexecute and requiring MRFC to retrieve further scripts fromAS. These scripts are dynamic where techniques (such as JSP,ASP and Servlets) are used to dynamically generate scriptdocuments and are transmitted to MRFC over HTTP protocol.In short, the media flow and presentation state is delegated tothe MRFC and MRFP, while the AS retains overall controlsince scripts are defined (and can be dynamically generated)by the AS. This delegation model generates the loop betweenAS, MRFC and MRFP for media execution.

B. Media packets keep forwarded to device whose control-plane is aborted

Certain modules act as bridges between NFs. Failure ofthese bridging modules result in control-plane termination.However, data-plane being decoupled from control-plane staysconnected via different set of NFs. We dial originating VoLTEcalls and slowly increase the call rate (adding 1 call persecond). We then trigger S-Incoming Leg Control module (abridging module between S-CSCF and the AS) failure at 25thsecond (i.e. on setting up 25th call), as shown in Fig. 4. Onmodule failure, P-CSCF does not accept any new call andmakes 5 retries (with retry interval of 1 second) to receivea response from S-CSCF. On 5th unsuccessful retry, P-CSCFdrops all calls by sending SIP BYE message to originatingdevice (as shown in Fig. 4(left)). However, this SIP BYEmessage from P-CSCF was not forwarded to terminatingdevices because of the failure of only bridging module (S-Incoming Leg Control) between originating and terminatingdevices. Also, MRF does not receive this call disconnectmessage from AS (via failed S-Incoming Leg Control module)and maintains data-plane connection with LTE PGW.Terminating devices keep generating UL media packets thatare forwarded to MRF through data-plane. Fig. 4(right) showsnumber of media packets received at different time interval.As shown, MRF forwards the terminating devices packetstowards originating device, but does not receive any packetfrom originating device. In short, terminating devices keep theVoLTE call intact and experience speech mute issue, wherethey do not receive a response from originating device.Impact: The impact of S-Incoming Leg Control module failureis quite severe, because media plane packets from terminatingdevices towards originating device keep flowing and systemdoes not have any mechanism (e.g. timers) to fully terminatethe media connections. During this failure, IMS relies onhuman intervention to stop the call (expecting the called partyhangs-up the call after speech mute issue).We find that the impact of this issue can be reduced (upto30 seconds) when originating and terminating devices are

1Voice Extensible Markup Language (VoiceXML), State Chart extensibleMarkup Language (SCXML), and Call Control eXtensible Markup Language(CCXML)



5

Time #ofCalls1 12 23 34 45 56 67 78 89 9

10 1011 1112 1213 1314 1415 1516 1617 1718 1819 1920 2021 2122 2223 2324 2425 2526 2527 2528 2529 2530 2531 032 033 0

05

101520253035

0 5 10 15 20 25 30 35

Calls

Time (sec)

Failure triggered No newcall added

0

1000

2000

3000

4000

5000

0 5 10 15 20 25 30 35

Pack

ets

Time (sec)

UE to MRFMRF to UE

MRF keeps receiving packets from destination

Fig. 4: All calls are dropped when S-Incoming Leg Control modulefailure is triggered (left). However, MRF keeps forwarding datapackets to device even though the device does not maintain anyconnection (right)

located behind the NAT. In this case, STUN (Session TraversalUtilities for NAT) Binding Requests are used by these devicesas a keep-alive mechanism to maintain NAT bindings forsignalling and media flows [32]. If a device does not receivea STUN reply (within 30 seconds), it considers the flowand any associated security associations invalid and performsthe initial registration procedures. However, operational LTEnetwork operators are using IPv6 and do not install NAT intheir network [33]; as a result, our finding has greater impactfor operational IMS systems. Furthermore, in accordance to3GPP requirements [32], device does not implement keep-alive mechanism when a NAT is not present (i.e. given batteryconsiderations for wireless devices).

Analysis: Control-plane termination does not stopdata-plane flow The root cause of above issue is the hangingstate machine at MRF. On S-Incoming Leg Control modulefailure, P-CSCF updates device state to Mortal but MRFstays in Connected state.We now explain how dialogue states are created at controland data planes and how these states transition that leadsto hanging state at data-plane on module failure. Once thedevice is registered, it initiates multimedia request (i.e.originating VoLTE call) by sending SIP INVITE requesttowards P-CSCF (via LTE PGW). The P-CSCF creates adialogue for requested multimedia service with a Start state(shown in Fig. 5a) and forwards the request to S-CSCF bytransitioning to Preparative state. Through dialogue, P-CSCFkeeps track that which SIP message is for which device andfor which multimedia session. On receiving the INVITEmessage from P-CSCF, S-CSCF creates device session andstates for incoming leg at its S-Incoming Leg Control moduleand then forwards the INVITE request to AS-Incoming LegControl module. AS then processes the INVITE request andprepares MRF to handle imminent multimedia traffic. MRFalso creates a dialogue for requested multimedia service,with a Start state (shown in Fig. 5c), and prepares requiredmultimedia resources for media application. Note that bothP-CSCF and MRF maintain dialogue states to keep track ofuser states in control-plane and data-plane, respectively. MRFtransitions to Created state by sending an acknowledgementregarding data-plane dialogue setup to AS. AS-Outgoing LegControl module modifies the INVITE request and forwardsthe request to S-Outgoing Leg Control module of S-CSCF,which then forwards it to terminating device and waitsfor a provisional response from terminating device (suchas 180 RINGING). Once S-CSCF receives the provisionalresponse, it forwards it to MRF (via AS) and P-CSCF.

Both P-CSCF and MRF move to Early and Progressingstates, respectively. Similarly, P-CSCF and MRF transition toMoratorium and Connected states, respectively, on receivingsuccessful multimedia setup response (such as 200 OK) fromterminating device.Connected state at MRF means that control-plane connectionhas been established between originating and terminatingdevices and multimedia traffic can flow at any time instance.However, P-CSCF requires an acknowledgement from deviceto transition from Moratorium state (a substate of theConfirmed state) to Established state.S-Incoming Leg Control module failure coupled with devicefailure at this point (when P-CSCF has not yet transitionedto Established state) results into hanging state machinescenario 1. Because of failure, P-CSCF is not able to receivean acknowledgement from device and transitions to Mortalstate by sending call clear (SIP BYE) message towardsS-CSCF. However, MRF does not receive the call clearmessage from S-CSCF via AS and does not terminate thedata-plane dialogue. We assume that there are no devicefailures and originating and terminating devices are successfulto establish multimedia connection. At later stage, S-IncomingLeg Control module failure happens that results into hangingstate machine scenario 2. On detecting failure, P-CSCF firsttries to reconnect by sending Reconnect control messagetowards S-CSCF, where S-CSCF tries to forward the messageto S-Incoming Leg Control module for message delivery toAS. Meanwhile, P-CSCF times-out and declares S-CSCFto be busy by generating 600 BUSY EVERYWHEREmessage code, as shown in Fig. 6. P-CSCF then acts as aUser Agent (UA) and generates SIP BYE message towardsoriginating device and S-CSCF, and transitions into Mortalstate. When originating device receives SIP BYE message, itterminates both control and data plane connections; whereasS-CSCF tries to forward SIP BYE to AS so that the ongoingmultimedia connection with the terminating device and MRFbe stopped. S-CSCF drops the packet because it could notforward it to AS due to constant S-Incoming Leg Controlmodule failure. As a result, terminating device does notreceive SIP BYE message and keeps its control and dataplanes sessions. When terminating device generates its mediapackets, it then forwards them to MRF. MRF finds originatingdevice dialogue state as Connected and forwards the receiveddata packets to PGW. However, these packets are dropped atPGW because PGW cannot reach the originating device. Notethat S-Incoming Leg Control module failure goes undetectedby AS because S-CSCF replies layer 3 keep-alive message toAS NF [34].

C. DiscussionWe discuss how latencies are controlled and fault tolerance

is provided in current dedicated boxes, and why virtualizationplatforms cannot match the performance of carrier gradesolutions.

Purpose-built hardware platforms have been developedwhich can tolerate faults and reduce latencies. They con-tinue to provide the required functioning despite occasional



6State Machine at P-CSCF

183 Session in Progress

202 Accepted

ACK

ByeTimeout

Confirmed

Terminated

StartPreparative Early

Established Moratorium

MortalMorgue

200 OK

(a) Dialogue state machine at P-CSCF

AS and S-CSCF failure

Application Server

S-CSCF

Incoming Leg Outgoing LegP-CSCF

MRFPGW

Device

Data PlaneControl Plane

AS-Incoming Leg Control

AS-Outgoing Leg Control

Incoming Leg State Model

S-Incoming Leg Control

S-OutgoingLeg Control

Incoming Leg State Model

Outgoing Leg State Model

(b) ILCM failure causes hanging state machine

State machine at MRF

<Created>

Connection progressing

Connectionconnected

Start Connected

Created Progressing

(c) Dialogue state machine at MRF

Fig. 5: When S-Incoming Leg Control module fails, the control-plane communication between originating and terminating device breaks.The originating device connection is aborted by P-CSCF (where P-CSCF state transitions to Mortal state). However, this abort control signaldoes not reach to terminating device. As a result, terminating device keeps forwarding media packets to media-plane (MRF)

internal components and modules failures, either transient orpermanent. Examples of hardware platforms are Ericsson’sBlade Systems (EBS) [5], Alcatel-Lucent’s Element Manage-ment System (EMS) [35], and Huawaei’s ATCA [36] thatemploy internal redundant hardware modules and provide NFavailability even during failures and maintenance at any timewithout disturbing traffic. At software side, NF equipmentvendors provide strong coupling between their software andhardware. Software fault tolerance is achieved by softwaredesign that ensures redundancy, both for error detection anderror recovery. As the system operates, functional checksare made on the acceptability of the results generated byeach piece of software component. Software platforms in-clude Ericsson’s ERLANG [5], Alcatel-Lucent’s NVP [8], andHuawaei’s Fusion [37] that use various software techniques forscalable real-time systems with requirements on concurrency,distribution and fault tolerance.On the other hand, vIMS is implemented over cloud platforms(such as OpenStack [13]), and relies on cloud platform’srecovery procedure for fault tolerance that uses heart-beatmechanism to detect failures. These cloud systems can onlydetect fail-stop failures at NF level, and not at the module-levelgranularity. Moreover, these system takes tens of seconds toreallocate a NF [38].

Fig. 6: Wireshark logs at P-CSCF: P-CSCF tries to connect to S-CSCF and times out because S-CSCF is unable to forward the requestto AS. P-CSCF updates S-CSCF status as busy and sends a BYEmessage to device by acting as an user agent (UA). Thereafter, anymodification to the dialogue is not allowed

VI. DESIGN

We put forward two design goals: (1) reduce media latency,and (2) improve system fault tolerance. At high level, werefactor NFs modules by (1) pipelining media plane processingand media control commands, and (2) quickly isolate faultymodule by reconfiguring its neighboring modules. Fig. 7 gives

an overview of our design. To reduce media plane latencies,Media Update module receives media meta-data from MediaProcessor module and predicts future meta-data values. Itthen requests control information for these predicted meta-datavalues from MRFC. In other words, MRFC prefetches controlinformation from MRFP for future purpose and steers MediaProcessor accordingly. This prefetching of control informa-tion and processing of media packets are done in parallel,unlike serially in the state-of-the art vIMS implementation.This is achieved because MRFC can likely predict futuremedia behavior as the media processing conditions change.Similarly, MRFC prefetches media execution scripts from ASby predicting media execution conditions.

Our design reconfigures each module by adding back-uppath with each of one-hop neighbouring module. As shown inFig. 7, Session Controller module at S-CSCF adds a back-uplink to its one-hop neighboring module (Incoming Leg Controlmodule) at AS. When Incoming Leg Control module at S-CSCF fails, then Session Controller assumes the role of failedIncoming Leg Control module and connects with IncomingLeg Control module of AS through back-up link. SessionController loads failed module’s execution logic and devicesession states upto to saved check-point. It then replays someof the already executed commands to resume failed operation.

Serving-CSCF

Incoming Leg Control

Session Controller

Media Resource ControllerMedia Control

Resource Controller

Media Resource Processor

Media Processor

Media Update

Application ServerMedia Policy

Incoming Leg Control

Proxy-CSCF

Registrar

AuthenticationPDN-Gateway

(LTE)

Design Overview

Backup LinkPrimary Link

Pre-fetch control info

Update Pre-fetchUpdate

Fig. 7: Design overview

A. Pipelining control instructions with media plane execution

MRFP processes media packets based on instructions itreceives from MRFC and AS in real time. Ideally, theselatencies do not go beyond few µseconds, but they significantlyincrease under high call rate and fine-grained script executioncontrol. To address this issue, we propose pipelining mediaprocessing and its control instructions request. This allows usto convert serial operations of processing of media packetsat MRFP and fetching control instructions from MFRC into



7

parallel operations. We achieve this by pipelining next controlinstructions for future media packets while processing receivedmedia packets. The control instructions are provided basedon media behavior which can be determined through mediametadata. When MRFP processes the packets, it also calculatesthe metadata (e.g. voice payload size, packets arrival rate orpackets per second, availble and consumed bandwidths andmore) for these packets. To pipeline control instructions, weare required to predict future metadata generated by futuremedia packets and then prefetching control instructions forthese packets. We use a simple prediction algorithm, where wefirst calculate the deviation of received metadata value fromits previous value and add this deviation into current value, asexplained by algorithm 1. These new metadata values becomeour predicted medtadata for which we request control instruc-tions from MRFC. Because control instructions remain samefor a range of metadata values, our prediction does not prefetchwrong control instructions for most of the cases. For example,voice codec G.711 is applied for all packets with voice payloadsize in between 160 to 240 Bytes, and jitter rate in the rangeof 30 to 50 packets per seconds [14]. In other words voiceprocessing instructions have built-in tolerance range that weexploit in our favour. However, as actual metadata comescloser to tolerance range, our algorithm may prefetch wrongcontrol instruction by predicting wrong metadata. We addressthis issue by predicting batch of metadata that also optimizesour prefetching algorithm.

Algorithm 1 Prefetching algorithm with/without optimization1: procedure PREFETCHING2: Predict:3: metadata← receive metadata from Media Processor4: εi ← calculate prediction error from receivedmetadata5: for all metadata values m do6: ∆m = m - mprevious

7: mnew = ∆m + m + εi8: metadatapredicted += mnew

9: end for10: Predict with Optimization:11: metadatah ← historical values of metadata12: αi ← smoothing constant,where 0 < α≤113: for all metadatah values m do14: mnew,t = αmprevious,t−1 + (1 - α)mnew,t−1

15: metadatapredicted += mnew

16: end for17: Send(metadatapredicted)18: ReceiveFrom()← receive control info from MRFC19: Update(Control) . UpdateMediaProcessor20: end procedure

Optimization using batch prefetching: Prefetching futurecontrol instruction, although, helps to run packet processingin parallel; it does not reduce the control instruction fetchingloop. We propose generating batch of metadata by takinghistorical metadata measurements into account, and then re-questing their control instruction. To achieve this, we useexponential smoothing model [15], described in algorithm 1,to forecast series of future metadata values. This model weighs

past observations using exponentially decreasing weights,i.e. recent observations are given relatively more weight inforecasting than the older observations. Using exponentialsmoothing model, we generate a batch of predicted metadata.This batch contains 5 (also configurable) sets of metadata,where each metadata set contains several metadata values.Then this batch is forwarded to MRFC and respective controlinstructions are received. Similarly, MRFC provides scriptdocuments against predicted metadata sets that it fetches fromAS. Thereafter, when Media Update module receives actualmetadata values from Media Processor module, it immediatelyreplies respective control instruction (as closest as possi-ble) from prefetched control instructions set. Media Updatemodule also observes the deviation of predicted and actualmetadata values and requests or delays prefetching futurecontrol instructions by generating new batch of metadata. Thisprocedure helps in reducing request/reply loop between MRFCand MRFP.

Complexity: The algorithm 1 contains two parts: (1) In thefirst part, the prediction of metadata value, metadatapredicted,is done by iterating through all metadata values m. Thisiteration is performed once which is linear with complexityof O(m). (2) The prediction with optimization part considershistorical values, metadatah, into account and iterates throughall metadata values m. This iteration is also done once and itis linear in time. Because, either part 1 or part 2 is executed,thus, the time complexity of the algorithm is linear in m.

Algorithm 2 Module Reconfiguration algorithm1: procedure RECOVERY PROCEDURE2: Start:3: SIP ← ReceiveFrom()4: Start timer A5: SendTo (SIP)6: if Timer A expires And SendTo(SIP) is Failed then7: goto FastRetry()8: else if SendTo(SIP) is Success then9: Break

10: end if11: FastRetry():12: Start timer B13: SendTo (SIP)14: if Timer B expires And SendTo(SIP) is Failed then15: goto Failover()16: else if SendTo(SIP) is Success then17: Break18: end if19: Failover():20: Announce(): failure to failed module’s neighbors21: Reconfigure(): links with failed module’s neighbors22: goto Start()23: end procedure

B. Fault isolation and module reconfiguration

At the heart of failure recovery in our design is fault detec-tion, its isolation and module reconfiguration. We show failure



8

State-transition in design

1. Start timer A

SIP Received

SIP Forwarded

Fast Retry

2. SIP Sent

3. Timer A expires

5. SIP Resent

Failover

4. Start timer B

6. Timer B expires

7. Modules reconfigured

Replay

Fig. 8: Failure detection and recovery procedure through statetransition diagram

recovery procedure through finite state machine diagram, asshown in Fig. 8, and capture the steps through algorithm 2.In the following, we explain failure detection and recoveryprocedure through SIP INVITE message example. However,note that our design can detect and recover from failure whenfailure occurs during call setup phase, (such as during INVITE,RINGING, TRYING SIP message failure), or after call setupphase (such as NOTIFY, UPDATE, CANCEL SIP messagefailure).

Failure detection procedure: When Session Controllerreceives the INVITE request from P-CSCF, it enters into SIPReceived state and starts the timer A, as shown in Fig. 8. TimerA is configurable timer, which is set to be 1 second in ourimplementation. Thereafter, INVITE message is forwarded tothe next hop, i.e. towards S-Incoming Leg Control module, andSession Controller moves its state to SIP Forwarded sate. IfS-Incoming Leg Control module does not reply to SIP INVITEmessage and Timer A expires, then Session Controller moduleenters into fast-retransmit stage – which is failure detectionstage. The rationale of fast-retransmit is to quickly recoverfrom failure by reducing the retry interval. Session Controllermodule starts Timer B, and resends SIP INVITE message aftermoving to Fast Retry state. In Fast Retry, SIP INVITE messageis resent at an interval of 200 msec until Timer B expires,which is set to be 1 second in our implementation. In otherwords, we propose 5 retries of the failed SIP message. Assume,S-Incoming Leg Control still does not reply to SIP request; as aresult Timer B expires and S-Incoming Leg Control is declaredto be failed.

We should also highlight the rationale of not using keep-alive mechanism to detect the failure. First, such mechanismcan burden internal NFs modules by unnecessarily pingingall neighboring modules. Second, IMS system capable ofhandling hundreds of thousands of subscribers voice requestsper day [26] is processing at least tens of SIP messages perseconds. These SIP messages are enough to detect modulefailure. Third, even though the failure is not detected for alonger period, it will not have any impact on media planeexecution and let users exchange media packets even duringfailures.

Failover procedure: After detecting failure, we performfail-over procedure when Session Controller module declaresS-Incoming Leg Control module out-of-service and takescharge of non-responding module. When S-Incoming LegControl does not respond and triggers Timer B expiration,

we first deactivate the link between Session Controller andS-Incoming Leg Control. Then Session Controller moduleloads the failed module’s executable through preloaded con-figurations and configures it by activating the link betweenitself and newly loaded module (acting as S-Incoming LegControl). In other words, Session Controller connects with S-Incoming Leg Control through a local loop. Next, it announcesmodule failure to failed module’s neighbors via backup linkand declares itself being in-service module serving the failedmodule’s execution. The recipient neighboring modules updatetheir routing path and connects to Session Controller. Once theconnection is setup, then Session Controller module replaysthe failed messages (i.e. INVITE SIP message in our example)at newly setup module and resumes the control-plane opera-tion.

To proceed with message replay and resume the service,in-service module should have access to the session states ofthe failed module. However, such session states are also lostduring module failure. To address this, we exploit the fact thatboth Session Controller and S-Incoming Leg Control modulescommunicate in a feedback loop of request and response. Therequester can always know the session states at responderwhen it receives the reply. For example, device initiates callrequest by sending INVITE SIP message which ultimatelyreaches at S-Incoming Leg Control module of S-CSCF whichthen forwards it to AS-Incoming Leg Control module of AS.On receiving SIP message, AS creates device session thatincludes user identities, charging function address, and deviceauthentication information etc.; and modifies the INVITE SIPmessage and then sends the modified message to S-CSCF. Onreceiving modified SIP INVITE message, S-CSCF modulesstore updated session along with checkpoint. Now assume,the failure occurred during next SIP message transmission(i.e PROGRESSING). On this failure, the in-service module(which takes charge of failed module) replay the SIP messagesstarting from stored checkpoint. That is replaying all the SIPmessage upto the checkpoint over newly launched module(residing locally).

When a timeout is not a failure: It is possible thatproposed, but configurable, timeout value does not representactual module failure. Such rare case occurs when the linkbetween two module is severely congested or S-Incoming LegControl module goes through random failure not impactingthe functionality of S-Incoming Leg Control. We still definesuch case as a failure that impacts user Quality of Service(QoS), and performs failover procedure. Indeed, such failoverprocedure helps reducing link congestion. This is becauseduring failover procedure links (interfaces) between modulesare reconfigured (changed) and congested links are removedfrom execution path.

Complexity: The algorithm 2 contains three parts: (1)check for failure, (2) if failure happens then do fast retry,(3) if failure persists then start failover. As a worst casescenario, the failover procedure kicks-in that re-configures theinterfaces. To reconfigure all interfaces, the algorithm needsto iterate through all interfaces / links. Hence, the complexityof algorithm 2 is O(n), where n represents the number ofinterfaces to be re-configured.



9

VII. IMPLEMENTATION

We use open source IMS platform (OpenIMS [12]) and opensource cloud operating system (OpenStack [13]) to implementthe functionalities of IMS protocol and NFV, respectively.OpenIMS provides basic implementation of IMS NFs and isdeployed over OpenStack platform. OpenStack provides fullflexibility on how IMS NFs are managed on cloud platform byproviding abstraction of common hardware resources throughvirtualization and meets compute, networking and storagedemands of different IMS applications. We spent significantefforts to modify source code in both platforms to suite ourneeds.

A. State-of-the-art implementation of IMS

OpenIMS has coupled all IMS NFs by implementing themover single virtual machine (e.g. VMware [39]) that providesoptimal performance when hundreds of users are accessingIMS network at the same time. For NFV deployment, we firstdecouple IMS NFs into separate VMs. Then these VMs arebridged through virtual network interface. These stand-aloneVMs are deployed over OpenStack to achieve state-of-the-artvIMS implementation. We also provide 1:1 redundant copyof IMS NFs to achieve minimum industry requirement forNFV [40] [41]. We use default timers as specified by IMS andOpenStack documents [42] [38]. Our system configurationsare based on the guidelines and parameters provided by 3GPP[42], OpenStack [38], and CISCO’s IMS [43]. We considerthis implementation as state-of-the-art vIMS with which wecompare our design.

B. Implementation of proposed vIMS

We exploits OpenIMS modular structure and adopt itsimplementation to our needs. We describe our efforts as below:Defining modules: OpenIMS provides basic IMS implementa-tion where it does not implement all of the modules as definedby 3GPP specification [24]. We modify OpenIMS sourcecode to add many more modules (such as breaking incomingand outgoing connection through Incoming/Outgoing SessionControl Leg modules). We achieve transition between thesemodules when one functional module calls other functionalmodule in a chain of SIP messages.Pipelining control instructions with media plane processingTo support pipelining, we first break the dependency betweenMedia Processor and Media Update modules and setup twointerfaces among them. Media Processor uses first interfaceto send metadata towards Media Update, and receives controlinstructions over second interface. It implements callbackfunction to receive control instructions from Media Updatemodule.Irrespective of packet processing job, Media Update modulerequests control instructions for a batch of predicted metadataover a fixed time interval (implemented as 2 seconds), andgenerates an event of new control instructions only when thepredicted metadata causes different sets of control instruction.Media Processor receives new control instructions through acallback function.

Failure detection procedure: We detect failure by implement-ing finite state machine (FSM) (as shown in Fig. 8). In FSMimplementation an operation must start from an initial stateand transit to another accepted state. To achieve this, we createFSM transition table that transits from a given state to a newstate when either the response is generated for a request (i.e.no failure case) or its guard timer has expired (i.e. failurehappens). By doing so, the proposed FSM only executes onnecessary functional module.Fail-over procedure: To successfully execute fail-over proce-dure, we are required to immediately resume IMS operationby (1) isolating faulty module, and (2) performing failoverby reconfiguring interfaces. To achieve these goals, we keeptrack of on-going device session before fault using a hashtable to store/retrieve user’s session information. When failureoccurs, the FSM transitions to Failover state. In Failover state,in-service module (that detected failure) retrieves last storeddevice session information from hash table. Then in-servicemodule updates the network configurations at incoming andoutgoing interfaces and contacts neighboring modules of failedmodule using on-going request identities. Thereafter, the in-service module takes charge of failed module’s operations andresumes the IMS service.

VIII. EVALUATION

We evaluate how our proposed design reduce latenciesand improve vIMS fault tolerance. The state-of-the art vIMSdescribed in Section VII-A serves as the baseline of ourexperiment with which we compare our design. We run ourtests on a local network of servers with Intel Xeon(R) ES-2420 V2 processor at 2.20GHZ x 12, 16M Cache size, and16GB memory. For each VM, we use Ubuntu Server 14.04.3LTS with the Open IMS Core. Each IMS NF is implementedover a separate server to make cluser of VMs in OpenStack.

We now present experimental results showing how ourproposed vIMS reduce latencies and improve fault tolerance.

A. Reducing latencies

To evaluate how our proposed design reduces media-planelatencies compared to state-of-the-art vIMS design, we launcha number of voice calls and inject voice packets over estab-lished media connection between caller and callee. These callsare launched in systematic way, where our script increases thecall rate by adding one call every second. In the first second,2 calls are dialed (2 calls per second), then 3 calls per secondand so on, upto 300 calls per second. This experiment is in-accordance to the experiment we show in Empirical Studysection (Section V-A).In state-of-the-art design, as the call rate increases the media-plane latencies start increasing, as shown in Fig. 9. Only60% of the traffic remains below 100 msec (upper boundof voice QoS requirement), where rest of 40% traffic incursas high latency as 600 msec (6 times the upper bound ofvoice QoS requirement). The main reason behind this wasthe frequent interaction between Media Processor and MediaControl modules. Media control module exercises fine-grainedcontrol on how voice packets are processed. It also interacts



10

with Media Policy module to fetch call execution XML, usingwhich Media Processor processes the packets.Fig. 9 shows that proposed design significantly reduces media-plane latencies. These latencies remain under 50 msec evenwhen MRF is processing around 40,000 simultaneous calls(when all active calls add-up, starting from 2 calls/sec to 300calls/sec). This significant improvement was achieved becauseMedia Processor module does not fetch control instructionsand simply processes the packets as soon as voice packetsarrive. The control instructions are sent to Media Processorover a separate interface, without meddling media packetsexecution. Media Processor module keeps only one copyof control instructions and updates it as soon as it receivesnew control instructions (calculated based on packet metadataprediction).

00.20.40.60.81

0 100 200 300 400 500 600

CDF

Delay in msec.

State-of-the-art

Proposed

Fig. 9: Comparing state-of-the-art vIMS media plane latencies withproposed vIMS design under increasing call rate. Proposed designreduces latencies by pipelining media plane execution with controlinstructions. Loops between control and media plane incurs latenciesin current state-of-the-art vIMS implementation

0

2

4

6

0 5 10 15 20 25

Tim

e (m

sec)


(a) Triggering of failure detection procedure

2.5

40

26

0

20

40

60

Propposed Behind NATvIMS

vIMS onOpenStack

Tim

e (s

)

(b) Failure recovery

Fig. 10: (a) Increasing call rate means there are always more numberof SIP messages flowing through IMS NFs modules. This helps toquickly start failure detection procedure in case next hop does notrespond to SIP message request; (b) Proposed design takes a coupleof seconds to recover from the failure, i.e. time taken in detectingfailure and reconfiguring the modules by isolating failed module.Behind NAT vIMS NFs and OpenStack use keep-alive mechanismto detect the failure. IMS without OpenStack implementatio requireshuman intervention to hangup media plane

B. Improving fault tolerance

Our design improves fault tolerance by (1) quickly detectingthe failure, and (2) running failover procedure after isolatingfaulty module. In our experiment, we trigger S-Incoming LegControl module failure during control-plane operation (i.ewhen call is being established) and let our system detect thefailure and perform failover procedure.How quickly failure detection mechanism triggers: In ourdesign, we are not using keep-alive mechanism to detect thefailure. Therefore, we are interested to observe how quicklythe failure detection mechanism starts after the failure has

triggered. We argue that fault tolerance is only importantfor running system, i.e. when control-plane and data-planemessages are flowing through the system. Faults in idle sys-tems do not have any impact on user applications. Therefore,we are interested to know how quickly failure is detectedunder control-plane operation. Fig. 10a shows that failuredetection mechanism triggering time sharply drops from 6msec to 1 msec when the call rate increases from 2 calls perseconds to just 7 calls per second. This is mainly becausea call request triggers at least 4 SIP messages (i.e. INVITE,PROGRESSING, RINGING, and 200OK) that arrive withinthe call establishment time (on average 26 msec). This meansroughly every 6 msec, one SIP message is processed at IMS– that is also the gap between failure occurrence time andstart of failure detection procedure time. However, this gapsignificantly reduces to 1 msec as soon as fewer than 10 callsare added to the system. In other words, we can say thatour design is efficient in triggering failure detection procedurewhich is almost real time.Failure detection: Failure detection procedure starts as soonas failure detection mechanism triggers, i.e. SIP message isreceived at IMS NF module that transitions to SIP Receivedstate (refer Fig. 8). Failure detection in our proposed IMS,Behind NAT IMS (that uses Session Traversal Utilities forNAT (STUN) protocol [44]), and OpenStack implementa-tions is based on timers. In our proposed design, failure isdetected when both timers, Timer A and Timer B, expireafter 2 seconds. Behind NAT IMS implementation declaresfailure after default value of 30 seconds, whereas OpenStackimplementation takes 16 seconds at minimum to detect thefailure [45]. Note that, OpenStack can only detect the failureof those modules that connect two NFs and cannot detectinternal modules failure. This is because OpenStack monitorsthe working of different NFs through heart-beat mechanismand does not monitor every individual module.Failover: Once the failure is detected, the failover procedurestarts. Fig. 10b shows how much time different systemstake to get back to service. Our proposed design simplyreconfigures modules and loads the faild module’s executable,and takes only 500 msec to recover from failure after failuredetection (that takes 2 seconds). However, both Behind NATand OpenStack take roughly 10 more seconds after failuredetection (8 seconds to prepare backup NF and restores theservice, and 2 seconds to register device and establish call).System load: We also tested failure detection and recoveryduring IMS SIP signalling load in our design, as shown inFig. 11. Increasing call rate generates increasing number ofSIP messages, loading IMS towards its capacity. However,the failure detection and recovery delays upto few secondson peak call rate which is acceptable because the systemis going to reach its capacity anyway. This delay mainlycomes from thread contention for system resources wheretriggering of failure detection timer (Timer B) is slightlydelayed, and overloaded nighboring modules also respond lateduring failover procedure.



11

1000150020002500300035004000

50 100 150 200 250 300

Tim

e (m

sec)


Failure detection Failover

Fig. 11: Failure detection and failover time during system load.As the call rate increases, more number of SIP messages traversethrough IMS NFs modules. This reflects a scenario of SIP signallingprocessing load on different modules. However, failure detection andfailover time slightly increases during SIP signalling load

C. Discussion

In operational network, both IMS NFs and LTE NFs reporttotal number of bytes they have processed to Policy andCharging (PCRF) NF. Then PCRF applies operator specific al-gorithm that takes total number of bytes processed at LTE andIMS as inputs in order to calculate subscriber’s billing record.We discover that discussed failure triggers discrepancies undertwo different scenarios at charging system (i.e. PCRF). In firstscenario, originating device can successfully transmit its ULmedia packets towards terminating device’s MRF (from steps1 to 5 in Fig. 12a). For these UL data packets the subscriberis charged by the PCRF when both PGW (step 2) and MRF(step 4) correctly report charging records to PCRF. However,originating device never receives any DL media packets andis billed even during speech mute issue.

In the second scenario, MRF at terminating device networkprocesses media packets and reports charging record to itsPCRF (step 6). It forwards media packets to PGW thatdrops the packets because the user has de-registered from thenetwork after failure. Therefore the PGW does not initiate anycharging report to PCRF NF. This issue can potentially resultsinto user overcharging because PCRF has processed chargingrecord for those media packets that terminating device havenever received. We show this in Fig. 12b where MRF in-state-of-the-art IMS implementation (i.e. in baseline IMS)charges all DL packets but PGW does not charge any packet.This results into potential charging discrepancies at PCRF.In contrast, our design does not cause any discrepancy as itquickly recovers from failure.

IX. RELATED WORK

We compare our work against recent efforts on NFV, vIMSand middle boxes’ fault tolerance space.

NFV: [46] provides general purpose NFV platform. [47]and [48] make use of software and hardware choices tomeet specific service demands. [49] and [50] discuss NFVintegration in mobile network. But these works do not discusshow NFV can provide same level of latency and fault toleranceas that of original carrier grade solutions. [51] [52] [53] dis-cuss algorithmic approaches to solve network slicing problem.Authors propose solutions for choosing optimal number ofvirtual instances to meet the requirements of mobile traffic.

Charging Discrepancy

PGWorig PGWterm

MRForig MRFterm

PCRForig PCRFterm

Originating device Terminating device

1

2 3

4

5

6

78

9

(a) NFs traversal

MRF (baseline IMS)PGW MRF (with proposed design)PGW100 0.0001 0.001 0.0001

0

20

40

60

80

100

MRF (baseline IMS) PGW MRF (withproposed design)

PGW

% T

raff

ic C

harg

ed

Chargingdiscrepencies

NO chargingdiscrepency

(b) Charging result discrepancies

Fig. 12: (a) Step by step procedure describing NFs traversal whenoriginating device initiates call to terminating device. In case ofdiscussed failure, originating device will be charged even if it doesnot receive any packet from terminating device (speech mute issue).(b) IMS failure causes charging report discrepancies at PCRF NF.IMS reports number of bytes it processed, whereas PGW does notreport any charging info because it has dropped all the packets. Ourdesign eliminates such charging discrepancies

[54] [55] solve NF placement problem for 5G networks. Incontrast to NF slicing and NF placement works, this paperaddresses the reliability issues in NFV.

vIMS: Our work [56] [57] discuss fault tolerance issuesin vIMS. [56] shows that highly modular 3GPP standardizedvIMS network functions incur latencies and are susceptibleto failures. To address these issues, we let IMS modules toreconfigure in real time. [57] proposes modular redundancydesign to address fail stop failure. In contrast, this paperaddresses fault tolerance and low latency issues in NFV byproviding vIMS as a use case. [58] provides dynamic resourceallocation algorithm for vIMS, [58] discusses merits of de-ployment strategies of vIMS. [59] enhances vIMS featuresfor M2M. But these efforts do not discuss latency and faulttolerance aspects in vIMS.

Fault Tolerance: [17] and [60] propose logging NF statesduring normal operations and reconstructing them after afailure. Their approaches cannot address real-time and tran-sitory NF sessions recovery. [61] [62] and [46] discuss faulttolerance in non-IMS (SIP based) voice over IP applications.[63] discusses general load balancing strategies in vIMS anddoes not discuss vIMS working during faults.

Latency: [64] and [65] imply dynamically schedulingschemes to meet changing traffic demands in NFV. [66]and [67] propose mobile edge computing designs to reducelatencies. [68] proposes trading off latency with other per-formance metrics. All of above approaches fail to reducesystem latencies which come from virtualizing system and itscomponents interactions.

X. CONCLUSION AND FUTURE WORK

Through IMS case study, we show that highly modularvirtualized system design incurs latencies when different NFsmodules interact with each other over a chain of operations.Also, such design, relying on cloud platform’s failure detectionmechanisms, does not tolerate module’s faults. This resultsdownlink data-plane operations to carry-on, even though de-vice control-plane has been terminated. Our design, refactorsNFs modules and reduces latencies by pipelining control andprocessing operations. It also reconfigures modules to tolerate



12

faults by first isolating faulty module and then resuming thefailed operation.From this work, we gain deep system insights and identifyother research issues – to be discussed in future work. Few ofthem include (1) multiple modules failure, as well as completeNF failure cases, (2) improving prediction with high varianceof media metadata (i.e. device mobility and its presence in lowradio signal strength areas), and (3) studying security issueswhen some modules may skip important security modulesduring failure.

REFERENCES

[1] What is NFV.http://www.etsi.org/images/files/ETSITechnologyLeaflets/NetworkFunctionsVirtualization.pdf.

[2] F. Networks. Virtual solutions for your nfv environment. In TechnicalReport.

[3] Survey: NFV expected to deliver opex, capex benefits.http://www.fiercewireless.com/tech/story/survey-nfv-expected-deliver-opex-capex-benefits/2013-12-07/.

[4] NFV, Initiative to Reduce Network Expenses (CAPEX & OPEX both).https://www.linkedin.com/pulse/nfv-initiative-reduce-expenses-capex-opex-both-apoorv-gupta/.

[5] Ericsson Blade System (EBS) for IMS.http://archive.ericsson.net/service/internet/picov/get?DocNo=266/03819-FAP130506&Lang=AE&HighestFree=Y.

[6] HP Blade System.https://www.hpe.com/h20195/v2/getpdf.aspx/4aa4-8125enw.pdf.

[7] Build massively scalable soft real-time systems.http://www.erlang.org/.

[8] N-Version Programming.http://www.inf.pucrs.br/∼zorzo/cs/n-versionprogramming.pdf.

[9] VoLTE Subscribers to Total 310 Million Come Year-End.http://www.telecompetitor.com/report-volte-subscribers-to-total-310-million-come-year-end/.

[10] 3GPP. TS 23.203: Policy and Charging Control Architecture, 2013.[11] 3GPP Specification series.

http://www.3gpp.org/dynareport/24-series.htm.[12] OpenIMS – The Open Source IMS Core Project.

http://www.openimscore.org/.[13] OpenStack Open Source Cloud Computing Software.

https://www.openstack.org/software/.[14] Quality of Service Design Overview.

http://www.ciscopress.com/articles/article.asp?p=357102.[15] Forecasting with Single Exponential Smoothing.

http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc432.htm.[16] NFV and the fear of failure.

http://virtualizedworld.com/2016/09/nfv-and-the-fear-of-failure/.[17] J. Sherry and et al. Rollback-recovery for middleboxes. In ACM

SIGCOMM, 2015.[18] M. Ghaznavi, A. Khan, N. Shahriar, K. Alsubhi, R. Ahmed, and

R. Boutaba. Elastic virtual network function placement. In CloudNetworking (CloudNet), 2015 IEEE 4th International Conference on,pages 255–260. IEEE, 2015.

[19] M.-A. Kourtis, G. Xilouris, V. Riccobene, M. J. McGrath, G. Petralia,H. Koumaras, G. Gardikis, and F. Liberal. Enhancing vnf performanceby exploiting sr-iov and dpdk packet processing acceleration. In NetworkFunction Virtualization and Software Defined Network (NFV-SDN), 2015IEEE Conference on, pages 74–78. IEEE, 2015.

[20] AT&T raises SDN network transformation goal to 55% for 2017.http://www.fiercetelecom.com/telecom/at-t-raises-sdn-network-transformation-goal-to-55-for-2017.

[21] DoCoMo to Offer NFV-Based LTE in 2016.http://www.lightreading.com/carrier-sdn/nfv-(network-functions-virtualization)/docomo-to-offer-nfv-based-lte-in-2016-/d/d-id/709223/.

[22] High Availability is more than five nines.http://www.ericsson.com/real-performance/wp-content/uploads/sites/3/2014/07/high-avaialbility.pdf.

[23] IMS Release 10 Tutorial.http://disi.unitn.it/locigno/didattica/AdNet/10-11/IMS Tutorial Scalisi.pdf,

[24] List of IMS-related Specifications.http://onlinelibrary.wiley.com/doi/10.1002/9780470695135.app1/pdf.

[25] VoLTE global status – 46 VoLTE systems launched.http://gsacom.com/paper/volte-global-status-30-volte-systems-launched-2/.

[26] Call volume statistics.http://www.arcep.fr/index.php?id=10519&L=1.

[27] LTE QoS baiscs.http://4g-lte-world.blogspot.com/2013/01/quality-of-service-qos-in-lte.html.

[28] E. Bauer and R. Adams. Reliability and availability of cloud computing.John Wiley & Sons, 2012.

[29] 3GPP. TS24.880: Media server control using the IP Multimedia (IM);Core Network (CN) subsystem;.

[30] Call Control eXtensible Markup Language.https://www.w3.org/TR/ccxml/.

[31] Voice Extensible Markup Language.https://www.w3.org/TR/voicexml20/.

[32] 3GPP. TS124.229: Internet Protocol (IP) multimedia call control proto-col based on Session Initiation Protocol (SIP) and Session DescriptionProtocol (SDP).

[33] C.-Y. Li and et al. Insecurity of voice solution VoLTE in LTE mobilenetworks. In ACM CCS, 2015.

[34] Virtual Router Redundancy Protocol (VRRP) using keepalived toprovide quick failover of layer-3 services.http://docs.openstack.org/mitaka/networking-guide/scenario-l3ha-ovs.html.

[35] Alcatel-Lucent End-to-End IMS Solution.https://www.alcatel-lucent.com/solutions/communications-collaboration.

[36] High reliability using ATCA platform for IMS.http://www1.huawei.com/en/products/core-network/singlecore/ims-core/index.htm.

[37] Fusion Programming.www.huawei.com/ilink/en/download/HW 327323.

[38] Default openstack timers.http://docs.openstack.org/kilo/config-reference/content/cinder-conf-changes-kilo.html.

[39] VMware Virtualization for Desktops & Servers.http://www.vmware.com/.

[40] A. Bernstein. Availability For Network Function Virtualization. Cisco:Technical Report, 2015.

[41] W. River:. Virtualization: Ensuring Carrier Grade Availability. WhitePaper, 2014.

[42] 3GPP. TS186.008–2: IIMS Network Testing: IMS Configurations andBenchmarks.

[43] Cisco ASR 5000 Series Command Line Interface Reference.http://www.cisco.com/c/dam/en/us/td/docs/wireless/asr 5000/10 0/OL-22948-01 CLI Reference.pdf/.

[44] P. M. J. Rosenberg, R. Mahy and D. Wing. Session Traversal Utilitiesfor NAT (STUN), 2008. RFC 5389.

[45] Apache ZooKeeper.https://zookeeper.apache.org/.

[46] S. Palkar and et al. E2: a framework for NFV applications. In ACMSOSP, 2015.

[47] R. Mahindra and et al. A practical traffic management system forintegrated LTE-WiFi networks. ACM Mobicom, 2014.

[48] M. T. Raza and et al. Finding the Right Virtualized LTE Instance forthe Job. CoNEXT Student Workshop, 2017.

[49] M. T. Raza and et al. Rethinking LTE Network Function Virtualization.In IEEE ICNP, 2017.

[50] Akyildiz and et al. Wireless software-defined networks (W-SDNs)and network function virtualization (NFV) for 5G cellular systems: anoverview and qualitative evaluation. Computer Networks, 93:66–79,2015.

[51] M. Bagaa and T. Taleb and A. Laghrissi and A. Ksentini and H. Flinck.Coalitional Game for the Creation of Efficient Virtual Core NetworkSlices in 5G Mobile Systems. IEEE Journal on Selected Areas inCommunications, 36(3):469–484, 2018.

[52] I. Benkacem and T. Taleb and M. Bagaa and H. Flinck. OptimalVNFs Placement in CDN Slicing Over Multi-Cloud Environment. IEEEJournal on Selected Areas in Communications, 36(3):616–627, 2018.

[53] A. Laghrissi and T. Taleb and M. Bagaa. Conformal Mapping forOptimal Network Slice Planning Based on Canonical Domains. IEEEJournal on Selected Areas in Communications, 36(3):519–528, 2018.

[54] Ksentini, Adlen and Bagaa, Miloud and Taleb, Tarik. On using SDN in5G: the controller placement problem. In IEEE Globecomm, 2016.



13

[55] Taleb, Tarik and Bagaa, Miloud and Ksentini, Adlen. User mobility-aware virtual network function placement for virtual 5G network infras-tructure. In IEEE ICC.

[56] M. T. Raza and et al. Enabling Low Latency and High Reliability forIMS – NFV. In ACM/IEEE/IFIP CNSM, 2017.

[57] M. T. Raza and et al. Modular Redundancy for Cloud based IMSRobustness. In ACM Mobiwac, 2017.

[58] Carella and et al. Cloudified IP Multimedia Subsystem (IMS) forNetwork Function Virtualization (NFV)-based architectures. In IEEEISCC, 2014.

[59] M. Abu-Lebdeh and et al. Cloudifying the 3GPP IP multimediasubsystem for 4G and beyond: A survey. IEEE CommunicationsMagazine, 54(1):91–97, 2016.

[60] Rajagopalan and et al. Pico Replication: A high availability frameworkfor middleboxes. In ACM Symposium on Cloud Computing, 2013.

[61] Bozinovski and et al. Fault-tolerant SIP-based call control system.Electronics Letters, 39(2):254–256, 2003.

[62] Pant and et al. Optimal availability and security for IMS-based VoIPnetworks. Bell Labs Technical Journal, 11(3):211–223, 2006.

[63] F. Lu and et al. A virtualization-based cloud infrastructure for IMS corenetwork. In IEEE CloudCom, 2013.

[64] P. Duan and et al. Toward Latency-Aware Dynamic Middlebox Schedul-ing. In IEEE ICCCN, 2015.

[65] R. Mijumbi and et al. Design and evaluation of algorithms for mappingand scheduling of virtual network functions. In IEEE NetSoft, 2015.

[66] B. Yang and et al. Seamless Support of Low Latency Mobile Applica-tions with NFV-Enabled Mobile Edge-Cloud. In IEEE Cloudnet, 2016.

[67] D. Lake. Enhancing Mobile Services with Edge Computing Capabilities.Newsletter, 2015.

[68] R. A. Hamada and et al. An IMS-Based LTE-WiMAX-WLAN archi-tecture with efficient mobility management. In IEEE MELECON, 2016.

Muhammad Taqi Raza received the Master degreein computer science from the University of Califor-nia, Los Angeles, CA, USA, in 2017. He is a 5th-year Ph.D. student in computer science departmentat UCLA.

His research interests include network security,network function virtualization, and networked sys-tems and their applications with an emphasis oncellular security.

Songwu Lu is a Professor in the Computer ScienceDepartment at the University of California, Los An-geles, CA, USA. His research interests cover wire-less networking, mobile systems, sensor networks,and data center networking.

Prof. Lu was on the editorial board of IEEE/ACMTransactions on Networking, and was on the boardsof IEEE Transactions on Mobile Computing, ACMWireless Networks, and IEEE Wireless Communica-tions Magazine in the past.

Mario Gerla received the engineering degree fromPolitecnico di Milano, Milano, Italy, in 1966 andthe M.S. and Ph.D. degrees in engineering from theUniversity of California, Los Angeles (UCLA), in1970 and 1973, respectively.

From 1973 to 1976, he was with the NetworkAnalysis Corporation, New York, NY. He is cur-rently with the Faculty of the Department of Com-puter Science, UCLA, where he is a Professor.He has designed and implemented various networkprotocols (channel access, clustering, routing, and

transport) under Defense Advanced Research Projects Agency and NationalScience Foundation grants. His research interests include distributed computercommunication systems and wireless networks. His work is cited more than21,000 with h-index of 118.

Xi Li is currently an associate professor and vicedean in the School of Software Engineering, Uni-versity of Science and Technology of China. He hasbeen directing research programs in the EmbeddedSystem Lab, examining various aspects of embeddedsystem with the focus on performance, availability,flexibility, and energy efficiency.

He has led several national key projects, severalnational 863 projects and NSFC projects. He is amember of ACM and a senior member of CCF.

Refactoring Network Functions Modules to Reduce Latencies ...

Documents

Transcript of Refactoring Network Functions Modules to Reduce Latencies ...