See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/323718924

SoftBox: A Customizable, Low-Latency, and Scalable 5G Core Network

Architecture

Article  in  IEEE Journal on Selected Areas in Communications · March 2018

DOI: 10.1109/JSAC.2018.2815429

CITATIONS

8READS

238

5 authors, including:

Some of the authors of this publication are also working on these related projects:

Software-defined Traffic Engineering View project

IoT security View project

Mehrdad Moradi

University of Michigan

11 PUBLICATIONS   87 CITATIONS   

SEE PROFILE

Yikai Lin

University of Michigan

9 PUBLICATIONS   65 CITATIONS   

SEE PROFILE

Zhuoqing Morley Mao

University of Michigan

173 PUBLICATIONS   8,030 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Mehrdad Moradi on 09 June 2018.

The user has requested enhancement of the downloaded file.

438 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

SoftBox: A Customizable, Low-Latency, andScalable 5G Core Network Architecture

Mehrdad Moradi , Yikai Lin , Z. Morley Mao , Subhabrata Sen, Fellow, IEEE, and Oliver Spatscheck

Abstract— We propose a novel cellular core network archi-tecture, SoftBox, combining software-defined networking andnetwork function virtualization to achieve greater flexibility,efficiency, and scalability compared to today’s cellular core.Aligned with 5G use cases, SoftBox enables the creation ofcustomized, low latency, and signaling-efficient services on a peruser equipment (UE) basis. SoftBox consolidates network policiesneeded for processing each UE’s data and signaling traffic into alight-weight, in-network, and per-UE agent. We design a numberof mobility-aware techniques to further optimize: 1) resourceusage of agents; 2) forwarding rules and updates needed forsteering a UE’s traffic through its agent; 3) migration costsof agents needed to ensure their proximity to mobile UEs;and 4) complexity of distributing the LTE mobility functionon agents. Extensive evaluations demonstrate the scalability,performance, and flexibility of the SoftBox design. For example,basic Softbox has 86%, 51%, and 87% lower signaling overheads,data plane delay, and CPU core usage, respectively, than twoopen source EPC systems. Moreover, our optimizations efficientlycut different types of data and control plane loads in the basicSoftBox by 51%–98%.

Index Terms— 5G, cellular core, MEC, SDN, NFV, container.

I. INTRODUCTION

CELLULAR networks play a very important role inour lives. Today’s 4G networks (Fig. 2) are composed

of the radio access networks (RANs) and evolved packetcore network (EPC). The EPC architecture suffers fromthree critical issues: (i) lacks fine-grained customizabilityand programmability in both its control and data planes [1],(ii) exhibits large control and data plane delays because ofrouting UEs’ signaling and data traffic through long paths [2],and (iii) consists of complex nodes and protocols generatinghuge control plane overheads or signaling storms [3]. EPC’sinefficient network policy management lies at the root ofthese issues; EPC partitions and distributes policies for amobile UE on different nodes. These nodes often must beplaced in geo-distributed data centers to maximize EPC’sefficiency in mobility support [4]. Thus, large forwarding

Manuscript received October 30, 2017; revised February 5, 2018;accepted February 27, 2018. Date of publication March 12, 2018; date ofcurrent version May 21, 2018. This work was supported by NSF underGrant CNS-1629894 and Grant CNS-1345226. (Corresponding author:Mehrdad Moradi.)

M. Moradi, Y. Lin, and Z. M. Mao are with the University of Michigan,Ann Arbor, MI 48109 USA (e-mail: moradi@umich.edu; yklin@umich.edu;zmao@umich.edu).

S. Sen and O. Spatscheck are with AT&T Labs–Research, Bedminster,NJ 07090 USA (e-mail: sen@research.att.com; spatsch@research.att.com).

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/JSAC.2018.2815429

Fig. 1. SoftBox consolidates the policies associated with each UE into a UEcontainer in its proximity.

delays, high signaling overheads, and bottlenecks in massiveservice customization are inherent in EPC (§II.B).

Motivated by these issues, operators are exploring 5G corenetwork designs [5], [6]. Although the requirements and usecases are not yet finalized, these networks are expected tohave three properties [7]–[10]: (i) build optimized and cus-tomized services on a per-UE basis to support the proliferationof heterogeneous devices (e.g., domestic robots), (ii) realizeultra-low latency (e.g., sub-5ms) with gigabit experience forUEs that run real-time applications (e.g., AR devices, self-driving cars), and (iii) have minimal signaling overheads thatcause severe performance degradation. Today, there is growingconsensus among operators that SDN and NFV are amongthe key technologies for achieving these properties [11].However, existing SDN/NFV solutions cannot easily realizethese properties because they often build on the EPC architec-ture and its inefficient policy management scheme, and thusinherit main weaknesses of today’s EPC networks. On the onehand, virtual EPC designs (e.g., SCALE [12], KLEIN [13],PEPC [14]) are focused on network automation and make nomajor enhancement to the EPC architecture. On the other hand,SDN EPC designs (e.g., SoftCell [1], SoftMoW [15], [16],MCord [17]), which decouple the EPC control and dataplanes to independently scale each, make matters worse. Theirdecoupling further distributes policies associated with mobileUEs on more nodes.

To address the EPC issues and realize the 5G core prop-erties, we explore a different point in the SDN/NFV designspace. We design SoftBox (Fig. 1), a radical rethink of the EPCarchitecture, that replaces the network policies scattered overthe EPC nodes far from users with a scalable, flat, and modular

0733-8716 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 439

TABLE I

SOFTBOX IN THE SDN/NFV DESIGN SPACE

architecture where a per-UE agent close to RANs enforcesmany complex policies (e.g., mobility management). UsingNFV, SoftBox creates a light-weight, programmable, logicalbox for each mobile UE in its proximity. The box consolidatesalmost all control and data network functions (NFs) neededfor processing the UE’s signaling and data traffic. LeveragingSDN, SoftBox programs the data plane to steer each UE’straffic through the corresponding logical box. We argue thatSoftBox is a flexible, scalable, and novel architecture:

First, the SoftBox architecture meets the requirementsof 5G core networks by enabling operators to buildcustomized, low latency, and signaling-efficient serviceson a per-UE basis. No existing solution supports all thesesimultaneously (see Table I). SoftBox can flexibly select andoptimize a different set of control functions (e.g., securitymanagement) and data functions (e.g., DPI) for each UEbox based on the UE’s needs and capabilities (e.g., batterylife). In addition, SoftBox supports each mobile UE witha consistent low latency experience; it migrates UE boxesindependently and ensures each of them always is in theUE’s proximity. Finally, our consolidation of policies intoUE boxes eradicates EPC’s distributed protocols that generateeast-west signaling (control plane) overheads.

Second, the SoftBox architecture is scalable. We are notsimply proposing a per-UE EPC-in-a-box design [18]. In fact,we rearchitect and optimize the EPC functions for the UEbox environment. In addition, SoftBox realizes each UE boxusing a container that is a lightweight, isolated Linux process.Containers have near zero virtualization overheads comparedwith virtual machines (VMs) [19]. Finally, our UE containersare compact as they only carry the binaries of optimized EPCfunctions. The combined effect is that SoftBox systems supportsubstantially more UEs on the same number of CPU cores thanEPC systems (6.2-8.3× more).

Third, SoftBox is a novel solution that redesigns thecore from ground up. It goes beyond being a UE containercluster management system (e.g., Google Kubernetes [20])and carefully addresses five network design and optimizationquestions: Which and how cellular core functions should berearchitected for UE containers? How does a SoftBox coreinteract with LTE RANs? Can we leverage UEs’ mobilitypatterns to more efficiently place and migrate UE containers in

the core? Can we leverage UEs’ radio state to reduce resourceusage of UE containers? And how should we steer each mobileUE’s traffic through its container?

A. Contributions and Roadmap

In summary, we make the following three contributions inthis paper:

• We propose SoftBox, a scalable and novel architecturefor the cellular core that fixes EPC’s policy managementissue and meets the customization, latency, and signalingrequirements of 5G core networks (§II). We explore theidea of slicing the core into UE containers, flesh out dif-ferent components of SoftBox, refactor the core function-ality into them, and design the lifecycle of UE containersand define their interactions with LTE RANs (§III).

• We develop novel solutions to further optimize SoftBoxby identifying challenges of slicing the core into manyUE containers (§III.F). We design efficient mobility-aware mechanisms to optimize resource usage of UEcontainers (§IV), SDN forwarding rules and updatesneeded to steer UEs’ traffic through UE containers (§V),control and data plane costs of UE container migra-tions (§VI), and performance of control plane (signaling)communication between UE containers and RANs (§VII).

• We build a detailed proof-of-concept prototype ofSoftBox using open-source software (e.g., Dockercontainer [21], RYU SDN controller [22], OAIEPC [23]). We evaluate SoftBox by combining real LTEtraces collected from 200 PhoneLab testbed UEs [24],synthesized LTE traces for 20M UEs, prototypeexperiments, and LTE/EPC testbed experiments onPhantomNet [25] (§VIII).

B. Summary of Results

We show that basic SoftBox has by 86%, 51%, and83%-87% lower signaling overheads, data plane delay, andCPU core usage, respectively, than two EPC systems (i.e., OAIEPC, OpenEPC). The improvements are independent of thenumber of UEs and packet processing technology. Moreover,our optimizations efficiently cut the data and control planeloads in the basic SoftBox by 51%-98% (§VIII). These resultspoint to the feasibility and potential of the SoftBox concepts.

II. MOTIVATION AND CONTEXT

Cellular core networks are stateful systems that sit betweenRANs and the Internet, connect user equipments (UEs) to theInternet, handle their mobility in the connected and idle modes,and enforce network policies on their signaling and data traffic.In addition to providing the basic functionality, SoftBox hasthree goals that will guide our design decisions. These goalswill be derived based on EPC architecture challenges (§II.B)and emerging 5G use cases [7], [8] in this section.

A. Design Goals for SoftBox

Goal 1: Per-UE customization of control and datafunctions. With the advent of IoT, UEs connecting to 5G

440 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

Fig. 2. EPC network architecture.

networks will be highly heterogeneous. Without creating per-formance bottlenecks and network management complexities,SoftBox must enable operators to compose a unique servicefor each UE or a class of UEs based on their needs and capa-bilities (e.g., hardware, plan, mobility). The per-UE servicecustomization and optimization must go beyond data planefunctions (e.g., DPI) that manipulate UEs’ data traffic in thecore. It should also encompass control plane functions (e.g.,paging and handover management) that process UEs’ signalingtraffic.

Goal 2: Ultra-low control and data plane latency.Today’s well-designed EPC deployments typically have10-30ms data plane delay and 10-60ms control planedelay [4]. To improve user experience and support a widerange of 5G use-cases (e.g., high-speed mobility, device-to-device communication), SoftBox must be able to createservices with ultra-low control and data plane latency (e.g.,sub-5ms [5], [26]) for UEs. Achieving this goal is challengingparticularly due to diversity of latency requirements acrossdifferent UEs and their unplanned mobility.

Goal 3: Minimal signaling overhead. The problem ofcellular network congestion is not much about UEs’ datatraffic, but in the EPC control plane generating tremendousoverheads in response to UEs’ signaling traffic [27]. Theincreasing number of UEs is further escalating the signalingoverhead and pushing EPC networks to their limit. Previousstudies report the global signaling overhead in EPC networkshas increased from 30M to 200M messages per sec in the pastthree years [3]. Optimizing sources of the signaling overheadsis of crucial importance for SoftBox.

B. EPC Architecture: Background & Challenges

Basing the design of SoftBox on the EPC architecturefundamentally limits us in efficiently realizing these designgoals. We highlight key issues in the EPC architecture toproperly derive SoftBox.

EPC architecture background. EPC has a hierarchicalstructure partitioning the core functions among a group ofdedicated nodes (Fig. 2). At the Internet edge, the packet datanetwork gateway (PGW) connects the core to Internet/contentproviders and enforces most of the data plane policies(e.g., NAT, DPI). At the RAN edge, enhanced node Bs(eNodeBs) are grouped into logical serving areas and connectto serving gateways (SGWs). Each SGW acts as a mobility

anchor point for its eNodeBs. It also forwards each UE’s datatraffic between the eNodeB and PGW using a separate GTP-U(GPRS tunneling protocol) tunnel. To connect to the network,UEs must register with the mobility management entity(MME) through eNodeBs. MME continuously exchangessignaling traffic with UEs and eNodeBs to perform securityand mobility functions (e.g., authentication, handover).To handle these tasks, MME accesses home subscriber server(HSS) that is a centralized database containing UE-relatedinformation (e.g., SIM card key). For connected UEs, policyand charging rule function (PCRF) authorizes the treatmentthat UEs’ data flows receive by supplying QoS rules toPGW/SGW in real time.

EPC Challenges. EPC partitions network policies or ser-vice associated with a UE and scatters the partial policieson its different nodes (e.g., PGW, MME). To maximizethe EPC efficiency, these nodes must be deployed in geo-distributed DCs often far from users [4]. This distributednetwork policy management has three known consequences.First, large control and data plane delays are unavoidabledue to increased propagation delay between distant EPCnodes, extra I/O delays at each node, and sub-optimal routingprotocol among EPC nodes [2]. Second, this design requiresto frequently synchronize the partial network policies/statesscattered on the EPC nodes (e.g., QoS policies on PCRFand PGW, UEs’ locations on MME and SGW) after eachUE’s control/data packet or flow [27]. The synchronizations,which often happen using complex distributed protocols (e.g.,GTP-C, Diameter), incur significant east-west signaling over-heads [3]. As the third consequence, per-UE customization ofcontrol functions (e.g., paging at MME) and data functions(e.g., DPI at PGW) do not scale. As such a customizationenlarges the EPC internal states, together with the delay andsignaling overhead issues, performance bottlenecks quicklyappear in EPC systems [12], [28]. Existing NFV/SDN EPCsystems virtualizing EPC on commodity servers or decouplingits control and data planes often build on the EPC architecture(nodes, protocol, interfaces) and thus, as documented, theyhave most of its weaknesses (detailed in §IX and summarizedin Table I).

III. SOFTBOX CORE ARCHITECTURE

This section describes our basic version of the SoftBoxarchitecture that is designed for next-generation core networks.SoftBox overcomes the EPC architecture challenges and real-izes our design goals motivated by evolving 5G use cases.Our key design decisions in the basic SoftBox are summarizedin Table II.

A. Need for the SoftBox Architecture

SoftBox is founded on a simple change to the EPCarchitecture. SoftBox consolidates each mobile UE’s networkpolicies, which are partitioned and placed on geo-distributednodes in EPC, into a UE container in its proximity. Containersuse lightweight OS-level virtualization technologies (e.g.,Linux cgroups and namespaces) that allow us to flexibly

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 441

TABLE II

SUMMARY OF DESIGN DECISIONS IN THE BASIC VERSION OF SOFTBOX

Fig. 3. Three conceptual benefits of SoftBox core networks.

package a logical service with its entire runtime environmentinto a single Linux process. A container image can beinstantly executed on different servers while retaining its fullfunctionality. Qualitative performance benefits of containersover VMs already presented in §I. Our approach of verticalslicing of the core functionality into UE containers at theradio edge equips SoftBox with four validated properties(§VIII) that no EPC supports simultaneously (see Table I).Before going into detail, we first present these properties tomotivate the need for adoption of SoftBox.

Property 1: SoftBox scalably supports per-UE cus-tomization of the core control and data planes. As shownin Fig 3-a, the isolation of UE services from each otherenables operators to select and optimize a different set ofcontrol and data NFs for each UE, without making otherUE services complex or degrading their performance. Anotheraspect of per-UE customization support lacking in EPC is thatSoftBox can flexibly allocate RAM/CPU resources on serversto UE containers based on service requirements. Conceptually,as long as an NF meets the following two criteria, one canplace it into UE containers with no modification to its logic:(a) always creates a separate packet processing pipeline foreach UE and maintains no shared state for different UEs andtheir traffic and (b) always performs local computation anddoes not need global network state to make efficient deci-sions. Through our careful analysis of the 3GPP-defined EPC

architecture [29] and systems [23], [25], we have found allthe EPC NFs already possess these properties and thus areamenable to the SoftBox approach of refactoring into UEcontainers (more detail in §III.C). Note that to improve theperformance of EPC, sometimes operators attach generic mid-dleboxes to PGW (e.g., video optimizer, traffic compression).Most of these NFs meet the above features as well so anoperator can flexibly place them in UE containers.

Property 2: SoftBox realizes ultra-low control and dataplane delays for mobile UEs by minimizing different factorscontributing to large delays in today’s EPC networks. By plac-ing the container for each UE in its proximity, SoftBox mini-mizes propagation delay in the core. Also, SoftBox cuts extraI/O and processing delays, which EPC’s redirection of a UE’straffic through multiple nodes incurs. Moreover, by decouplingUE services from each other, SoftBox independently migrateseach mobile UE’s service to its proximity to ensure a con-sistent delay experience (see Fig 3-b). Finally, by co-locatingcontrol and data NFs into UE containers, SoftBox minimizesEPC’s large synchronization delay between its control and dataplane nodes.

Property 3: SoftBox can minimize signaling overheadsin the core. EPC runs complex protocols (e.g., Diameter,GTP-C) to synchronize mobile UE states on its different nodes,causing east-west signaling overheads. SoftBox eradicates theneed for these protocols by centralizing and isolating thecore NFs for each UE in a UE container. Because differentNFs inside our UE containers synchronize UE states throughlocal message exchanges often on top of inexpensive publish-subscribe mechanisms (Fig 3-c), SoftBox significantly reducesthe east-west signaling overheads in the core.

Property 4: SoftBox can be deployed at large scale.Later, we show that SoftBox supports substantially moreUEs on the same number of CPU cores than today’s EPC(6.2-8.3× more) for three reasons. First, we go beyondcontainerizing EPC and instantiating EPC on a per-UE basis,and optimize the EPC NFs for the UE container environment.Second, containers are lightweight Linux processes with near

442 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

Fig. 4. SoftBox redesigns the cellular core to build customized, signaling-efficient, and low latency services.

zero overheads (e.g., virtualization, startup delay) comparedto VMs [19]. Third, the binaries of NFs are very small insize and per-UE instantiation of them in UE containers hasinsignificant overheads.

B. Overview: Transforming EPC Into SoftBox

We first provide an overview of five high-level steps that wetake to systematically transform EPC into the basic SoftBox.Each step is associated with proper forward references to ourdetailed technical discussion.

Step 1: Rearchitecting the EPC functionality for UEcontainers. While each basic UE container is expected tohave the EPC functionality at the very least, a naive per-UEcontainerized EPC is far from our vision for SoftBox. EPCis originally designed for distributed deployment scenariosand thus is too complex and inefficient for single-boxdeployments in UE containers. Moreover, EPC is notprogrammable to support fine-grained per-UE customizationas described in §II.B. To develop SoftBox, we will refactorand optimize the EPC functionality for the UE containerenvironment (§III.B).

Step 2: Deriving the SDN/NFV components of SoftBox.The most naive and incremental realization of SoftBox is toreplace EPC with UE containers in today’s EPC infrastructure(Fig. 2). One can put UE containers behind a logical loadbalancer inside existing EPC DCs and connect LTE RANsto the load balancer instead of EPC. As it turns out, whilethis naive SoftBox design still makes the core network morescalable and flexible, it substantially limits us in meetingour three design goals (e.g., ultra-low latency for mobileUEs in §II.A), largely because today’s infrastructure is notdesigned for SoftBox-like solutions. Therefore, we derivea minimal set of necessary software and infrastructurecomponents for SoftBox (Fig. 4) to ensure it can satisfy ourexpectations (§III.C).

Step 3: Connecting SoftBox to existing RANs/UEs.A practical core network design must be incrementallydeployable. SoftBox must not force operators to

access or modify the network stack of their subscribers,eNodeBs, or routers in peering ISPs. Hence we expand thedesign of SoftBox to ensure it can efficiently and seamlesslyinteract with these players (§III.D). In particular, the same asEPC, SoftBox continue to exchange IP traffic with ISPs andsignaling traffic over NAS (Non-Access Stratum) protocolwith UEs. For different performance and scalability reasons,we will develop new protocols between the SoftBox core andLTE RANs but we deploy them in a minimally disruptiveway without touching the source code of eNodeBs.

Step 4: Orchestrating UE containers. Automaticallymanaging the lifecycle of UE containers (i.e., their creation,placement, migration, and termination) is an essentialfunction in SoftBox. It is impossible for SoftBox toproperly perform these tasks without a direct and continuousinteraction with RANs and UEs. Existing NFV orchestrationschemes (e.g., [30]) have serious functional limitations inthis regard as they are originally designed for virtualizingwired networks functions. Consequently, we will designorchestration mechanisms and protocols in SoftBox toeffectively manage UE containers based on LTE eventsgenerated by RANs/UEs. In §III.E, we largely focus on UEcontainer creation and termination and let our more advanceddesign for other operations to evolve in later sections.

Step 5: Global SDN/NFV optimization of SoftBox net-works. The above steps result in a basic version of the SoftBoxarchitecture, which meets our three design goals in a scalableand efficient fashion (§VIII). Our basic SDN/NFV architectureprovides unique opportunities for global optimization of net-work resources and performance. We provide an overview ofthe process of developing an optimized SoftBox in §III.F andTable II, while presenting our technical solutions in the nextfour sections.

C. Software & Infrastructure Components of SoftBox

SoftBox replaces the EPC architecture with a fullySDN/NFV solution that globally controls UE containersin a programmable and elastic environment via openprotocols and interfaces. Existing network infrastructures

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 443

(deployed for EPC) make it either hard or impossible forSoftBox to fully achieve its properties (discussed in §III.A).Therefore, SoftBox, as a next-generation core networkarchitecture, makes a set of critical changes to the existinginfrastructures. A SoftBox network consists of four maincomponents: (1) mini data centers (DCs), (2) SDN switchingfabric, (3) UE containers, and (4) global SDN/NFV controlplane. We describe the rational behind these components andtheir functionality.

Component 1: Mini DCs and egress points close toRANs. Today’s EPC is distributed in a few DCs far fromRANs [4]. The EPC connects to ISPs at a single pointwhere the PGW is located (Fig 2). This DC infrastructureinherently contributes to a lot of the inefficiencies in EPC(e.g., large Internet access delay). To ensure UE containersare always in the proximity of mobile UEs, SoftBox envisionsa sufficient quantity of mini DCs close to RANs (e.g., aDC per state as shown in Fig. 4-b). Moreover, for quicklyprocessing UEs’ Internet traffic, a reasonable fraction of miniDCs are connected to the Internet. In general, the number,capacity, and location of mini DCs depend on many factors(e.g., latency requirement). Note that the industry pushtowards multi access edge computing (MEC) [31] alreadycreated the need for such DCs and more egress points soSoftBox can benefit from such trends.

Component 2: Programmable switches interconnectingDCs and RANs. To enforce network policies, SoftBoxsteers each mobile UE’s traffic through its correspondingUE container. A UE may simultaneously communicate withdifferent end points, e.g., other UEs connected to nearbyeNodeBs in the case of IoT Apps or external Internethosts. To quickly set up, modify, and tear down paths inthe data plane, SoftBox contains a flat switching fabric ofprogrammable SDN switches that seamlessly interconnectRANs and mini DCs (Fig.4-a). The inter-DC SDN network(Fig.4-b) ensures our latency-sensitive DC-to-DC traffic canbe routed over efficient paths inside the SoftBox core. Thistype of traffic is generated when UE containers migrateamong mini DCs to ensure their proximity to mobileUEs, or when two or more UEs (e.g., autonomous cars)with their UE container in different mini DCs directlycommunicate. Today’s EPC architecture and infrastructureare very inefficient in handling UE-to-UE traffic. First,EPC is designed to always route traffic of a UE to theInternet, even if the destination is a UE connected to anearby eNodeB. Second, EPC instances (e.g., in differentregions) cannot directly communicate with each other asEPC DCs are not connected to each other through direct links.

Component 3: Customizable and optimized UE contain-ers. In SoftBox, UE containers are the smallest unit of service,each embedding a customized bundle of data and controlNFs for processing a UE’s data and signaling traffic. Ourbasic yet extensible UE containers provide the EPC function-ality. However, they are extremely more efficient, lightweightand programmable compared to an EPC-in-box design. Our

basic UE containers are derived from EPC in two steps asfollows:

1) Optimizing the EPC functionality: We first extract thefunctionality distributed on the EPC nodes (i.e., S/PGW,MME, PCRF, HSS in Fig. 2) and refactor them intoa set of programmable NFs. Each container is a sin-gle process, where communication between its internalthreads occur through local message passing mecha-nisms (e.g., pub-sub). Since we consolidate the NFs intoUE containers, we naturally remove the EPC distributedprotocols (i.e., Diameter, GTP-C) from their implemen-tation. We also upgrade the protocols that run betweenEPC and RANs (e.g., GTP-U, S1AP/SCTP) with morescalable ones (will be explained in §III). Our approachof removing/upgrading complex EPC protocols makethe NFs and UE containers fast and lightweight at theend. In particular, through the refactoring process, ourbasic UE containers are equipped with the EPC datafunctions (DPI, NAT, firewall, buffers, accounting/QoS)and control functions (LTE mobility, security, and policymanagement NFs).

2) Developing a minimalist SDN architecture: Next,we design a programmable SDN architecture for UEcontainers by decoupling their control and data planes.As shown in Fig.4-c, SoftBox isolates each of the dataNFs into a Linux network namespace and interconnectsthem using a software switch. Then, it places the controlNFs on a platform called UE controller running atleast two applications: a) MobilityApp executing theLTE mobility management operations (e.g., handover,paging) and security management operations (e.g.,authentication, encryption) by exchanging signalingtraffic with the UE through RANs. b) PolicyAppenforcing data plane policies (QoS, monitoring,charging) by forwarding the UE’s data traffic throughdifferent in-container data NFs.

Component 4: Unified SDN/NFV control plane. Finally,SoftBox core networks are equipped with a global SDN/NFVcontroller (Fig. 4-a) that optimizes their performance andresources (e.g., minimizes migrations of UE containers andflow rules in the SDN fabric). For scalability and performancereasons, the global controller places an agent in each DC,SDN switch, and eNodeB to execute its commands. Ourglobal controller and SDN switch agents communicate overthe standard OpenFlow/P4 protocol. However, due to lackof protocols for configuring eNodeB/DC agents, we developcustom protocols between our global controller and them (willbe presented in §III.E, §V-§VII).

D. Connecting SoftBox to LTE RANs and UEs

SoftBox is an incrementally deployable solution as elabo-rated in §III. Here, we explain how SoftBox connects to exist-ing LTE RANs and UEs without any direct modification totheir network stack. In LTE/EPC networks, each LTE eNodeBexchanges two types of traffic with EPC (see Fig. 2). Withoutloss of generality, consider the traffic in the uplink direction.An eNodeB encapsulates each UE’s data traffic into a separate

444 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

TABLE III

SUMMARY OF DESIGN DECISIONS IN THE OPTIMIZED VERSION OF SOFTBOX

GTP-U tunnel and forwards it to EPC. GTP-U packets are fur-ther encapsulated into a UDP/IP header by the eNodeB sincecommodity devices (e.g., routers) between EPC and RANsdo not process this protocol. Moreover, an eNodeB maintainsa persistent SCTP (Stream Control Transmission Protocol)connection to MME. Over this connection, it acts as a proxyand multiplexes different UEs’ signaling or NAS (Non-AccessStratum) messages onto a single 4G S1AP (S1 ApplicationProtocol) session and transports them to MME.

Each UE’s NAS signaling and data traffic is processed intoa different UE container in SoftBox. To properly forwarddifferent UEs’ traffic to their corresponding UE containers,LTE RANs must communicate with the SoftBox controlplane (UE controllers and global controller) and exchangedifferent control messages and events. Since modifying thenetwork stack of eNodeBs for implementing new protocolsis impractical, we propose a minimally disruptive design tofacilitate communication of SoftBox with LTE RANs: Weinstruct our agents at eNodeBs (introduced in §III.C) toimplement a translation layer and map the EPC protocols tocustom SoftBox protocols and vice versa. At a high level,the translation layer performs two operations (Fig. 4). First,since some of the SDN switches (e.g., OpenFlow switches)do not support match+action rules on GTP-U packets andGTP-U is a complex protocol, it replaces each UE’s GTP-Utunnel with an MPLS tunnel. In §V, for minimizing the SDNrules and updates, we will propose a highly scalable SoftBoxprotocol over MPLS. Second, our translation layer reversesthe eNodeB’s multiplexing function, which puts different UE’ssignaling traffic onto a single S1AP/SCTP connection withMME. For each UE connected to the eNodeB, it establishesa separate transport connection (e.g., TCP, reliable UDP(RUDP)) with the corresponding UE controller, and properlyforwards the UE’s signaling traffic to it over our customapplication layer protocol (will be elaborated in §V).

E. Putting All Together: Orchestration of UE Containers

SoftBox dynamically and quickly provisions UE containersover the network by continuously and effectively interactingwith LTE RANs. When a switched-on UE sends an attachrequest, we instruct the eNodeB agent to send a UE container

creation request to the global controller that performs threeoperations: (1) fetches the UE’s profile (e.g., plan, type) fromthe subscribers (Fig. 4-a), (2) instantiates a customized UEcontainer for the UE in a DC through its DC agent, and(3) programs the eNodeB and SDN switches through itsagents to direct the UE’s traffic to the UE container. Next,the eNodeB establishes a session with the in-container UEcontroller and starts forwarding the UE’s signaling trafficto it. In the meantime, the UE controller installs rules intothe in-container software switch to enforce different NFs onthe UE’s data traffic. When the UE moves, other procedureshappen in SoftBox (e.g., updating data plane tunnels, con-tainer migration) that are discussed in the next four sections.For scalability, the global controller communicates with UEcontainers through its DC-level agents based on a publish-subscribe model. Through agents, it subscribes to certainevents in containers (e.g., low QoE) or reconfigures them withnew policies in runtime. When a UE turns off, the eNodeBagent notifies the global controller who fetches the UE statesfrom the container, backs them up in the subscribers database,and destroys the container.

F. Optimized SoftBox: Design & Optimization Challenges

The above basic version of SoftBox meets our designgoals (§VIII). To make SoftBox more effective and efficientfor deployments at scale, we further optimize it along fourdimensions. Our optimization challenges and solutions to themare unique to and novel in the context of SoftBox. We providean overview on the optimized SoftBox in the below andTable III. For brevity, we delay detailed discussion of relatedwork to later sections.

Challenge 1: Scalable optimization of idle UEs’ contain-ers. In cellular networks, UEs spend most of their time in theidle mode, so a large fraction of UE containers in SoftBoxcan be underutilized at any point in time. Each of themincurs small CPU/RAM overheads without receiving/sendingany traffic from/to the UE. Adopting the traditional NFVapproach [32] suggests that SoftBox’s global controller mustmonitor, stop and resume UE containers in response to UEs’transitions between the idle and active modes. Unfortunately,this approach does not meet our unique design requirements.

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 445

First, the strategy of stopping idle UEs’ container does notwork as some in-container NFs (e.g., paging) must always runin the core. Second, the global optimization is not scalable withmany UEs continuously changing their state in the network.

Challenge 2: Minimizing the SDN fabric rules andupdates. SoftBox steers traffic of UEs through their respectivecontainers for processing. Given there are millions of UE con-tainers in the network, scalable traffic steering is challenging.On the one hand, encapsulating each UE’s flows into a separatetunnel, similar to EPC, leads to huge forwarding states inSDN switches equipped with very small flow tables. Also,the mobility of UEs necessitates frequently updating the per-UE tunnels, degrading the controller throughput and trafficperformance. On the other hand, end-to-end traffic aggregation(e.g., between each eNodeB and gateways) does not work aseach UE’s traffic must be processed by a separate middlebox(container).

Challenge 3: Minimizing migration costs of UE con-tainers. SoftBox initially places the container of each UEin its proximity to realize ultra-low latency. Since UEs aremobile, their latency to their container can increase so SoftBoxneeds to migrate each of such UE containers to a DC that iscloser to the corresponding UE. Cloud operators can often planVM migration in advance, and only a small fraction of their

VMs need migration [33]. In contrast, UE container migrationsoccur in real time and at a large scale as UE mobility is thenorm and unplanned in cellular networks. Such migrationspose two issues: incur a control load on SoftBox’s globalcontroller, thus needing us to scale it up, and result in animbalance of load distribution on mini DCs, thus forcing usto over-provision their capacity.

Challenge 4: Scalable signaling sessions between SoftBoxand LTE RANs. Today’s protocols between LTE RANs andEPC are designed with an assumption that a single fixednode in EPC (i.e., MME) exchanges signaling traffic withUEs through eNodeBs (Fig. 2). Thus, SoftBox’s approachof processing each UE’s signaling traffic on a UE controller(Fig. 4) poses two challenges. First, it rapidly increases thenumber of connections from the core to RANs, which becomesunmanageable as the network grows. Second, it needs LTERANs to be able to determine the network location of UEcontainers corresponding to arbitrary UEs on the fly as they aredynamically created and can migrate between different DCs.This is challenging because this functionality is not availablein existing LTE RANs and deploying a central off-path registryand discovery service [34] is not scalable or efficient.

446 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

Roadmap. Next, we go into the details of each challengeto realize the optimized version of SoftBox.

IV. SCALABLE AND FLEXIBLE OPTIMIZATION

OF IDLE UE CONTAINERS

Optimizing resource usage of UE containers on serversis crucial to minimize power consumption of mini DCs inSoftBox. Each UE container runs some NFs, each spawningmultiple threads to process the UE’s traffic while buildingsome UE-specific states in memory. Thus, a UE container usesboth RAM and CPU resources. Storing the states typicallyrequires an insignificant amount of RAM (§VIII.C). Therefore,we incorporate unique characteristics of cellular networks tominimize CPU usage of UE containers. In LTE networks,UEs switch between the idle and active modes to save batterypower, while being idle most of the time. Although an idleUE does not send or receive any traffic through LTE RANs,its corresponding UE container still has a small CPU overheadon its server in the SoftBox core. The overhead comes fromthe fact that in-container NFs actively access their ports andqueues via polling and manipulate their threads and internalstates, even if they process no traffic. Such tiny CPU overheadsquickly add up since there are many UE containers in thenetwork.

A. Naive Approach: Centralized Optimization

We believe there is a good opportunity to optimize such UEcontainers. One might think we should borrow the global opti-mization approach adopted in many existing NFV platforms(e.g., Picocenter [32]): have our global controller (Fig. 4)continuously track each UE container, gracefully terminateUE containers corresponding to idle UEs, and instantiate newUE containers for those UEs once returning to the activemode. Although these procedures can be easily implementedin SoftBox by sending asynchronous notifications on topthe POSIX API [35] from the global controller to UE con-tainers, such global optimization approach does not meetour design requirements. First, stopping idle UEs’ containerceases all in-container NFs while some of them must alwaysrun, even though they are in the idle mode. For example,MobilityApp refactoring MME in UE containers (Fig. 4) needsto locate UEs particularly when they are idle through thepaging procedure. In theory, the set of “always-on” NFs canbe different for each UE. Second, it is not scalable to haveour global controller monitor millions of UE containers andstop/resume them in large-scale networks.

B. Our Approach: Self-Optimizing UE Container Design

To overcome the above shortcomings, we propose to care-fully design UE containers in SoftBox to optimize themselvesautonomously when their UE becomes idle. In other words,we distribute the logic and mechanisms of resource usageoptimization into UE containers. Our design of self-optimizingUE containers has two properties: (i) scalable and fast as itdoes not involve our global controller in the optimization loop,and (ii) flexible as it allows us to customize the implemen-tation of optimization in each UE container depending on

Fig. 5. Our scalable and flexible optimization of idle UEs’ container

its potentially unique set of always-on NFs. In more detail,we enhance our basic UE containers by developing a processmanagement module based on Supervisor [36] in them, whichuses advanced capabilities in Linux to do the optimizationtask. Our module listens to the UE state changes betweenidle and active (see Fig. 5). When the UE requests the UEcontroller to switch to the idle mode, it quickly freezes allnoncritical NFs (e.g., DPI) in the UE container and keeps onlycritical ones (e.g., MobilityApp) running (Steps 1-4). Whenthe UE returns to the active mode and connects to the network,it instantly resumes the frozen NFs (Steps 5-9). To seamlesslyfreeze/unfreeze the selected NFs, we leverage cgroupfreezerAPI [37] rather than the POSIX API inside UE containers.The cgroupfreezer library uses the Linux kernel freezer codeto prevent the freeze/unfreeze cycle from becoming visible tothe NFs being frozen so they can keep their memory states.The POSIX signals are observable within the NFs so theirthreads may select how to respond to them (e.g., block) thatcan cause them to break.

While our optimization method is not complex, it is noveland effective in the context of SoftBox networks. On theone hand, it meets our unique design requirements. On theother hand, it significantly cuts the peak CPU usage of UEcontainers as a large fraction of UEs are always idle (VIII).

V. TRAFFIC STEERING WITH MINIMAL

AND STABLE FORWARDING RULES

In SoftBox, the global controller must program tunnels inthe SDN fabric to properly steer each mobile UE’s trafficthrough the corresponding UE container that can migrateamong mini DCs (Fig. 4). Existing SDN switches have limitedflow table entries and updating them frequently (when UE andUE containers move) can cause traffic forwarding disruptions(e.g., packet drops). To cope with these limitations, our keyinsight is to pre-compute and pre-establish a minimal set ofpermanent tunnel segments into the data plane and reuse themfor steering different UEs’ traffic as much as possible. Thisapproach naturally reduces the numbers and update rates offlow rules in our SDN switches. We already discussed short-comings of other design options (e.g., establishing separatetunnel for each UE) in §III.F, and thus focus on realizingour novel traffic steering scheme in four steps in this section.Our scheme can be implemented on top of existing SR

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 447

Fig. 6. Our “recursive middleboxes” abstraction to scalably steer UEs’ trafficthrough containers

(segment routing) mechanisms [38]. At a high level, we haveSoftBox’s global controller: (Steps 1-2:) recursively breakdown the problem of finding a minimal and reusable set ofpermanent tunnel segments (endpoints and paths) into smallersubproblems by forming our novel abstraction, called “recur-sive middleboxes”, over SoftBox’ SDN switches. (Step 3:)then leverage source routing mechanism at RAN to relieve ourSDN switches of the task of switching UEs’ traffic betweendifferent tunnels (Step 4:) finally offload a part of trafficsteering task onto UE containers for scalability.

A. Step 1: Forming the Recursive Middleboxes Abstraction

We first explain how the global controller forms the “recur-sive middleboxes” abstraction to break down our large-scaletraffic steering problem. The controller views each DC, rack,host, and container in SoftBox’s data plane as an abstractmiddlebox. These middleboxes are recursively nested into eachother, where mini DCs and UE containers are the outermostand innermost abstract middleboxes respectively. Similar tohardware middleboxes, each abstract middlebox has two logi-cal ports to separate its ingress and egress traffic. Conceptually,each abstract middlebox embeds a group of SDN switchesin the network. Fig. 6 depicts child middleboxes recursivelyembedded into a DC middlebox.

B. Step 2: Recursive Tunnel Segment Computation

Over our abstraction, the global controller recursively pre-computes and pre-establishes a minimal set of permanenttunnel segments in the data plane and later reuses them for dif-ferent UEs. We now describe our tunnel segment computationfor UEs’ uplink traffic to the Internet and shortly discuss othertraffic types. Our tunnel segment computation procedure is asfollows: (i) the controller first addresses a small problem ofsteering aggregate uplink traffic of each eNodeB through DCmiddleboxes: it sets up a label-based tunnel segment betweeneach eNodeB and its nearby DC middleboxes (e.g., tunnel Ain Fig. 6), and a segment between each DC middlebox andclose Internet gateways (e.g., tunnel H). The Internet gatewaysand DCs can be determined based on operators’ requirementsfor closeness of UE containers to UEs (§VI), (ii) movinginside each DC middlebox, the controller solves the problemof steering the DC’s aggregate ingress traffic through differentrack middleboxes and exiting it from the DC. In this case,

it sets up a segment between the DC ingress point and eachrack middlebox (e.g., tunnel B) and a segment between eachrack middlebox and the DC egress point (e.g., tunnel G), and(iii) recursively, the same procedure continues in each rackand then in each server. Conceptually, the controller sets up asegment between the ingress port of each child middlebox andthat of its parent (e.g., tunnels B,C,D), and one between theegress port of each child middlebox and that of its parent (e.g.,tunnels E,F,G). On servers, the controller provisions tunnelsegments for the maximum expected UE containers. Whenit dynamically creates UE containers on servers, it reusespre-established tunnel segments (e.g., tunnels D,E) and onlyconnects UE containers to them. Except UE specific tunnels onservers (e.g., tunnels D,E), the rest of tunnels in the networkwill carry aggregate traffic of different UEs as follows.

C. Step 3: Source Routing on Abstract Middleboxes

Next, we explain how the global controller steers eachUE’s uplink traffic through its UE container. To minimizethe number of rules needed in SDN switches for tunnelswitching, it instructs agents in eNodeBs (see Fig. 4) toperform SR’s source routing. Assume a UE has sent a packetto an eNodeB (e.g., eNB-1 in Fig. 6). To redirect the packetthrough the nested middleboxes containing the correspondingUE container (e.g., Container-1), our eNodeB agent encodesa stack of labels into the packet (the stack can be compressedusing P4 [39], [40]). The top half of the stack consists ofthe labels of segments from the eNodeB down to the UEcontainer (e.g., [A,B,C,D]). The bottom half of them arelabels from the UE container up to the Internet egress point(e.g., [E,F,G,H]). After encoding the stack, the eNodeBagent sends out the packet. Our data plane switches alwaysforward the packet based on the outermost label sitting on thetop of the stack (TOS). When the packet arrives at the ingressport of each abstract middlebox (i.e., DC ingress switch, TORswitch, server NIC/soft switch), it pops the TOS label fromthe packet. Then, the packet is directed to the proper innermiddlebox based on the new TOS label. This continues untilthe packet reaches an ingress port of the UE container. Afterthe UE container completes its processing, the remaining lowerhalf of the stack is used to forward the packet from thecontainer to the Internet gateway.

D. Step 4: Label Stack Distribution and HandlingOther Traffic Types

First, our source routing requires proper distribution of labelstacks to eNodeB agents in the network. When a UE sendsLTE attach request through an eNodeB, the global controllercreates the UE container and instructs its eNodeB agent withthe proper label stack (§III.C). For scalability, the globalcontroller preloads each UE container with the set of labelstacks in the network and instructs it to proactively informdifferent eNodeB agents with proper label stacks when itsUE moves or it migrates among mini DCs. Second, ourtraffic steering scheme handles downlink and UE-to-UE traffic.For downlink traffic from the Internet to UEs, the controllerinstructs Internet ingress switches to do source routing similar

448 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

Fig. 7. Our container migration scheme with the distributed planning & mobility-aware heuristics.

to eNodeBs. For scalable classification of UEs’ downlinktraffic needed pushing proper label stacks onto each UE’sflows at the ingress switches, the controller uses the approachdescribed in SoftCell [1]. For UE-to-UE traffic, it pre-installsdirect tunnel segments among immediate child middleboxesof a middlebox (e.g., racks inside the DC in Fig. 6) to steersuch traffic through multiple UE containers.

We will show the above traffic steering scheme sig-nificantly reduces SDN rules and updates in large-scaleSoftBox (§VIII.C).

VI. SCALABLE & MOBILITY-AWARE UECONTAINER MIGRATION SCHEME

In SoftBox, the UE container placement is crucial to buildultra-low latency services on a per-UE basis. Since UEs aremobile, the latency between them and their container increasesso SoftBox’s global controller needs to transfer the UE con-tainers between DCs. There are diverse tools for seamless con-tainer migrations [32], [41]. Thus, we focus on two challengesthat are unique to SoftBox. Container migrations can (i) incura large control load to the global controller, and (ii) lead toan imbalanced distribution of containers among mini DCs.Traditional VM/container migration schemes do not meet ourperformance and scalability requirements (§III.F, §IX) sinceour environment deals with unplanned UE mobility, strictdeadlines, and many UE containers. Thus, we develop a con-tainer migration scheme that is novel in two aspects: it consistsof distributed and scalable components to schedule containersneeding migration, and leverages UE mobility patterns tochoose migration destinations efficiently.

A. Distributed Planning of UE Container Migrations

SoftBox allows operators to flexibly determine the maxi-mum latency between each UE and its container (e.g., 5ms).Assuming UE containers are already placed inside DCs,a migration event occurs when the UE-UE container latencybecomes more than the migration threshold. Since there aremany containers in the network, our design principle is toreduce the role of the global controller in migrations. It isnot scalable to have the controller (i) continuously track thelatency between millions of UEs and containers to determinewhich containers must be migrated, and (ii) collect and processfine-grained mobility patterns of many UEs (e.g., handoverand connection history) to be used in its migration decisions.

To address the issues, we distribute the load of schedulingmigrations and collecting mobility history of UEs on con-tainers in five steps. We design the UE controller’s Mobil-ityApp in each container to: (Step 1.) measure its latencyto the UE autonomously and continuously (see Fig 7-a),(Step 2.) record and prune the mobility information of the UE,and (Step 3.) make a decision locally regarding its migrationtiming and issue a migration event to the global controllerwhen it is necessary. Upon being notified, the global controlleronly: (Step 4.) runs our migration algorithms (in §VI.B) toselect a mini DC for the container based on its global viewof DC capacities and the mobility pattern sent by the UEcontroller, and (Step 5.) coordinates the old and new serversto handle the migration (§VIII.B).

B. Mobility-Aware Heuristics for UE Container Migrations

In response to each migration event, the global controllerselects a mini DC from the pool of eligible DCs satisfyingthe migration threshold; it also tries to achieve two goals inits real-time decision-making process (migration events are notknown a priori thus offline optimal algorithms cannot be used).First, it minimizes the total number of needed migrations overtime to reduce the load on itself. Second, it balances theload of containers among DCs (or minimizes the max load ofDCs in terms of the number of containers). In each migrationevent, our heuristic is to have the controller migrate the UEcontainer to a DC from the eligible DCs pool, that max-imizes the normalized term ContainerDurability(DC) +AvailableCapacity(DC). The durability function computeshow long a DC can host the UE container without forcing itto issue a migration event. The capacity function returns theremaining capacity of the input DC in terms of the numberof containers. Intuitively, continuous sum of the product ofthese two normalized functions in migration events reducesthe needed container migrations and DC load imbalances.The capacity function definition is easy but the durability onecan be realized in multiple ways. In this paper, we suggesttwo algorithms that both use the same capacity function buteach having a different durability function. To simplify thedescription of the algorithms, we use Fig. 7-b where theattachment of a UE to eNodeB C causes its container toissue a migration event. The problem is the controller needsto migrate the container either to DC1 or DC2 to meet thelatency requirement.

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 449

Fig. 8. Connectionless per-UE mobility management equipped with mobility-aware service discovery protocol.

• Our Least-loaded-proximity (LLP) algorithm assumesthe chance of next migrations reduces if the UE containeris moved closer to the UE. Thus, it has the durabil-ity function return reciprocal of the normalized latencybetween an input DC and the UE. In the example, LLPpicks DC1 because it is closer to the UE and the twoDCs are equally loaded.

• Our Least-loaded-mobility (LLM) algorithm assumesthe mobility/connection patterns of the UE in past timewindows (e.g., weeks) can determine the DC which canrun the container for the longest time. Thus, it computesthe durability function output for each eligible DC as fol-lows: it (i) first identifies eNodeBs that their latency to theDC is below the migration threshold, and (ii) normalizesand returns the total connection time of the UE to thoseeNodeBs in the past time window as the function output.In the example, LLM chooses DC2 as the UE has spentmost of its time in the past with the eNodeBs in the rangeof DC2 and the DCs are equally loaded.

Our distributed online container migration scheme is notcomplex, but it effectively cuts peak loads on our globalcontroller and DCs, has a high performance in diverse settingscompared to the optimal offline solutions, and meets ourdesign constraints (§VIII).

VII. SCALABLE INTERACTION OF

SOFTBOX CORE AND LTE RAN

For performance reasons, we distributed the MME on UEcontrollers inside UE containers. The protocols carrying UEs’signaling traffic between EPC MME and LTE eNodeBs are notscalable and sufficient for the SoftBox core containing millionsof UE controllers. They were originally designed for anenvironment where a centralized fixed MME handles all UEs’signaling traffic. Our UE containers are in a far larger quantityand migrate between different DCs so their network locationcontinuously changes. To handle the signaling traffic betweenSoftBox (UE controllers) and LTE RANs, we (1) design a fastand mobility-aware service discovery protocol that enables theRANs to locate the UE containers and (2) carry the signalingtraffic over a more scalable transport protocol.

A. Fast and Mobility-Aware UE Container Discovery

In a 4G LTE/EPC network, the MME node often is deployedat a fixed and static network location (IP address and port).

MME acts as a server and eNodeBs as clients (see Fig.8-a).Each eNodeB reads the network location of MME from aconfiguration file and then permanently connects to it. In Soft-Box, UE controllers in UE containers replace MME. UnlikeMME, both their numbers and network locations dynamicallychanges as a result of UE mobility events causing UE con-tainer migration, creation, and termination procedures. Clearly,the approach of storing configuration files in eNodeBs nolonger is effective or efficient as it would require tracking thelocation changes and frequently updating the configuration filein every eNodeB in the network. Therefore, we propose thatthe we enable eNodeBs to dynamically discover the latest loca-tion of UE controllers/containers corresponding to UEs. Thisapproach makes the network management easier comparedto the the traditional config file approach. Using the latestlocations, the eNodeBs can establish connections with UEcontrollers to properly forward UEs’ signaling traffic to them.One generic model to realize this discovery is to place a globalservice registry in SoftBox networks (e.g., [34]), register andupdate the location of each UE controller with it and havedifferent eNodeBs query the registry service. The clear draw-backs of this centralized model is (1) scalability as the globalregistry service can become a hotspot and (2) increased delayin accessing the LTE network as the global service can be farfrom eNodeBs and mobile UEs must wait for the discoveryprocess to complete. Rather than relying on such genericmechanisms with their associated shortcomings, we believea better approach to the discovery process is to incorporatecharacteristics of mobility protocols in cellular networks.

We design a fast and mobility-aware UE container registryand discovery service. Rather than having a global registryservice, we place a local container registry service on eacheNodeB (inside our agent/shim described in §III.C-E). EacheNodeB uses its local registry to quickly resolve its queries inzero-round trip time (0-RTT). We design the UE controllersto dynamically and proactively register/deregister themselveswith the registry instances on nearby eNodeBs before thoseeNodeBs need to discover them. To realize this, we build ona key aspect of mobility support in cellular networks. In LTEnetworks, MME registers each UE to a tracking area (TA).Each TA is a logical group of eNodeBs. At any time, MMElimits the mobility of a UE to its registered TA to ensure thebroadcast paging procedure and its associated radio wake-upswill not affect all UEs. This limitation has two consequences.

450 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

First, the UE runs a timer whose periodic expiration causes theUE to report back and confirm its current TA to MME. Second,when a UE wants to exit its current TA, it must explicitlyrequest MME to assign it to a different TA. Based on thisconcept, our UE container registry protocol works as follows(see Fig.8-b): (Step 1.) When a UE controller receives a TAupdate request from the UE intending to exit its current TA,it computes a new TA for the UE (e.g., TA2 in the example).(Step 2.) Then, it registers its location (e.g., [192.168.4.82,2153]) with the registries on eNodeBs in the new TA (e.g.,eNB1, eNB2). (Step 3.) Next, it sends a TA update to the UEconsisting of the new eNodeBs information. (Step 4.) Finally,the UE controller deregisters itself from the registry instanceson the eNodeBs in the old TA (e.g., eNB3, eNB4).

B. Connectionless RAN-Core Signaling Traffic

In 4G/LTE networks, each eNodeB establishes an S1AP/SCTP connection with MME Then, MME and eNodeBscreate logical NAS signaling sessions over these S1AP/SCTPconnections for different UEs (detailed in III-C). As a UEmoves around, MME quickly migrates the UE’s signalingsession on the SCTP connections with different eNodeBs inthe handover procedure. In SoftBox, because of distributingMME on UE controllers, multiplexing signaling sessions ofdifferent UEs into the same set of fixed S1AP/SCTP connec-tions is no longer an option as each UE must communicatewith a different endpoint (UE controller) in the SoftBox core.On the other hand, establishing dedicated connections foreach UE controller with different eNodeBs rapidly escalatesthe core-RAN connections by a factor equal to the numberof UEs. Managing these connections is very costly bothin terms of the network performance and the overhead onRANs. As UE containers migrate between different DCs andUEs move, RANs needs to continuously set up and teardown many connections with the SoftBox core. To addressthese problems, we transport the signaling traffic betweenSoftBox (UE controllers) and RAN (eNodeBs) through ourcustom application layer protocol over a reliable connection-less transport protocol (see Fig. 4 and Fig. 8-c). This approacheliminates the handshake cost/complexity while allowing anarbitrary number of UE controllers and eNodeBs to have low-cost communication with the guaranteed-order packet delivery.Our current prototype employs Reliable UDP (RUDP) [42] asa lightweight connectionless protocol. It is worth mentioningthat we continue using TCP between the global controller andother entities (e.g., switches, eNodeBs) as these sessions arestatic and permanent.

VIII. EVALUATION

A. Methodology

Using prototype and large-scale trace-driven evaluation,we first show that the basic (unoptimized) SoftBox architectureis more scalable and efficient than EPC. Our metrics are:(1) the number of CPU cores needed for a large-scale network,(2) the signaling overhead in the core, (3) the performancefor the device-to-device (D2D) communication. For compar-ison, we use widely-used EPC systems, OpenEPC [25] and

OAI EPC [23]. We carefully ensure that our comparisons withEPC are fair as our prototyped SoftBox has similar or betterfunctionality compared to the EPC systems, uses the samepacket processing technology. In addition, we study themunder similar deployment conditions. We then demonstratethat the enhanced version of SoftBox equipped with our fourschemes optimizing UE container migrations (§VI), idle UEs’containers (§IV), traffic steering through UE containers (§V),and UE container discovery (§VII) is even more effective andefficient for large-scale deployments due to having lower dataand control loads and higher performance. In the optimizedSoftBox evaluation, we validate that each of our optimizationtechniques is efficient in diverse settings with hundreds ofmini DCs and tiny UE container migration thresholds. Next,we describe the details of the SoftBox prototype and ourLTE dataset followed by our evaluation results. In evaluatingSoftBox, we used tens of servers with 16 CPU cores, 64GBRAM, and four 10GbE NICs on the Emulab-PhantomNetEPC/LTE testbed [43].

B. Prototype and LTE Dataset

Prototype. We developed a detailed prototype of SoftBoxthat can work with LTE eNodeBs equipped with our pro-tocols. We emulate each mini DC using 10 OVS switchesorganized into a two-level leaf-spine topology and a variablenumber of UE containers. In our large-scale experiments,we interconnect mini DCs and simulated RANs based on arealistic flat topology containing 1K OVS switches [44]. Ourglobal controller is implemented on top of RYU [22] with itstwo apps (Fig. 4-a). The first one instantiates and migratescontainers using Docker [21] and Flocker [41] respectively.The second one runs our recursive traffic steering schemeby configuring switches using OpenFlow. Inside DCs, ourprototyped UE containers run a minimal Linux and fullyrealize our design (Fig. 4-b). Inside individual UE contain-ers, we (i) refactor the EPC data plane by building high-performance DPI, firewall, NAT, buffers and charging/QoSNFs using the Netfilter [45] and nDPI [46] libraries to processUEs’ data traffic and (ii) refactor the EPC control planethrough developing the UE controller with its three apps ontop of RYU to process UEs’ signaling traffic (MobilityApp isa modified MME [23]).

Large-scale LTE traces. We also collected LTE traces from200 real UEs by deploying a mobile app on the PhoneLabtestbed [24]. Over 2 months, we captured the UEs’ radioand data messages with LTE networks (accessed their Qual-comm Diagnostic Interface) and their GPS locations. We thendeveloped simple heuristics (e.g., time-shifting, mixing) tosynthesize traces for 20M UEs based on the real ones. We builta RAN simulator to feed our prototype and EPC systemswith the data. For 20M UEs, our final dataset has 30 billionidle-active state changes, 10B handovers, 40K cells, and UEs’real-time location. Since our dataset does not have the cells’location needed for our container migration study and publicdatabases have little coverages for them, we used multipleclustering algorithms [47] to estimate the location of cellsbased on different UEs’ GPS samples. For brevity, we do notexplain them here.

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 451

TABLE IV

EPC & SOFTBOX SIGNALING OVERHEADS–**:COMMON

C. Evaluation of Basic SoftBox Architecture

We show that even the basic SoftBox is more scalable,flexible, and efficient than EPC. Additional optimizations forSoftBox as detailed in §IV, §VI, §V, and §VII are disabled inthe following experiments.

1) Scalability w.r.t. CPU Cores: We measure the CPU coresneeded by the EPC systems and our prototyped SoftBox (theglobal controller and UE containers) to handle a 20M UEnetwork.

Finding 1: Provisioning and maintenance costs of UEcontainers are minimal. We characterize the number ofCPU cores that SoftBox need to run the UE controller for20M UEs. measure the throughput of our global controllerrunning on one server. We generate synthetic attach requestsfrom turned-off UEs and UE container migration eventsfor registered mobile UEs. Our multi-threaded controllercan simultaneously process at least 400K attach and 300Kmigration requests per second on the server. In attach events,it instantiates UE containers and similar to EPC, programsone tunnel per-UE in its data plane (as we do not show ourtraffic steering optimization discussed in §V). In migrationevents, it randomly picks an eligible sink mini DC and server(as we do not show our migration optimization in §VI)and coordinates the old servers and new servers to do thecontainer migration. As per prior large-scale studies [1],the peak numbers of simultaneous attach and handoverevents for 20M UEs are significantly below 400K and 300K(our dataset shows a similar pattern). Thus, we were ableto provide low latency services for 20M UEs (with anymigration latency threshold) by using 16 CPU cores for theglobal controller, which is a minimal cost.

Finding 2: Our UE containers are lightweight andSoftBox is more scalable than EPC systems. We nowcharacterize the number of CPU cores that SoftBox needsfor 20M UE containers. Each of our container images islight-weight and 40MB on disk. Each running container isassigned to a single core in a server. Each UE container hasa high performance and can forward packets at 9-10Gbpswhen it is highly customized. By generating synthetic trafficbased on rates captured from real UEs in our LTE dataset,we observe each container consumes at most 1.2% of theCPU core’s processing capacity. Also, it occupies less than

0.5% of the total RAM on the server. As a result, we wereable to simultaneously execute 80 UE containers on eachcore and around 1250 of them on each of the servers.We have observed that OpenEPC and OAI EPC support atmost 200 and 150 simultaneous UEs respectively. Comparedto them, SoftBox requires 6.2-8.3× fewer CPU cores tosupport any given number of UEs. For 20M UEs, we require1.35M-1.88M fewer cores than these EPC systems. Verticalslicing of the core, eliminating most the EPC complexprotocols, engineering lightweight UE containers are the mainreasons for this efficiency.

2) Scalability w.r.t. Signaling Overhead: Using the samesetup, we now show SoftBox is more scalable than EPCsystems in terms of signaling overhead. We connect a simu-lated eNodeB and a UE to each of SoftBox, OpenEPC andOAI EPC. We measure the number of signaling messagesexchanged in them for the UE when it attaches, downloads a10MB file, and gracefully detaches. In this specific experiment,we do not study the handover operation due to the limitationsof OpenEPC and OAI EPC but this only underestimates thecapability of SoftBox in this regard.

Finding 3: SoftBox generates 86% fewer signalingmessages in the core compared to EPC. Table IV showsthe breakdown of signaling messages exchanged in OpenEPC(OAI EPC is similar) and SoftBox. OpenEPC creates 134 sig-naling messages and issues 385 database queries to handle theattach/detach procedure. This high overhead is because theEPC architecture distributes the policies associated with theUE among different nodes, continuously synchronizing themusing complex protocols. SoftBox cuts these overheads byconsolidating the control and data functions for the UE intoa container and eliminating EPC’s complex protocols (e.g.,GTP-C/U, Diameter, SCTP). For backward compatibility pur-poses, SoftBox does not change the EPC’s signaling ses-sions (NAS) with the UE so the NAS message type is commonamong SoftBox and OpenEPC. As shown in Table IV, SoftBoxgenerates 86% and 92% fewer control and DB messages,respectively, for the UE than EPC. This is significant inlarge networks, e.g., SoftBox produces 23.2M fewer signalingmessages for 20M UEs.

3) Performance w.r.t. Data Plane Delay: The D2Dtraffic that is generated through communication between

452 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

Fig. 9. Setup in the D2D experiment.

TABLE V

AVERAGE RTTS FOR THE D2D TRAFFIC

nearby devices (e.g., autonomous cars) is expected to surgein 5G. Most of D2D applications need small end-to-enddelays. We show that SoftBox is more efficient than EPCin supporting D2D low-latency communication due toconsolidating network policies close to RANs. In a simpleexperiment, we have two UEs attached to different eNodeBsand ping each other through a SoftBox and OpenEPC (Fig. 9).We measure the data plane delay experienced by the UEsin terms of the propagation and tunnel management delaysinside the two core networks. For a meaningful comparison,we assume an operator has two physical mobile switchingoffices (MOs) and the components of both networks aresimilarly distributed among them: we place OpenEPC’sPCRF-PGW in office MO2 and its MME-SGW-HSS in officeMO1 similar to real deployments [4]. SoftBox has an SDNswitch, a mini DC, and a global controller in MO1, andanother switch in MO2.

Finding 4: SoftBox has lower RTTs for D2D trafficthan similarly deployed EPC. First, SoftBox incurs 50%less propagation delay for the D2D traffic than EPC in theabove deployment since it creates two UE containers in theDC close to them and selects a more direct path between theUEs through them (Table V). In contrast, EPC inefficientlyroutes the D2D traffic to the PGW for the policy enforcement.Second, EPC’s distributed tunneling protocol (GTP-C/Uin Fig. 2) is very complex as explained in §III.B. Thus,we observe EPC has 2.9× higher processing delay comparedto SoftBox.

D. Evaluation of Optimized SoftBox Architecture

In this part, we evaluate the optimized version of SoftBoxand show that our four optimization techniques presentedin §IV, §V, §VI, and §VII lead to substantial improvementsof SoftBox’s performance.

1) Benefits of Idle-Mode optimization: We first show thatour optimization of idle UEs’ container (in §IV) further

Fig. 10. Effects of optimizing idle UEs’ container.

improves SoftBox’s scalability. Recall that we placed a moduleinside UE containers, which listens to UE state changes.When the UE becomes idle, it freezes all the container NFsusing cgroupfreezer (except MobilityApp that alwaysrun for the paging operation). When the UE becomes active,it unfreezes them quickly. Our optimized SoftBox is com-pared to the basic SoftBox as the baseline, which keeps UEcontainers and their internal NFs active regardless of the UEstate. Note that stopping idle UEs’ containers did not meetour requirements. Based on the methodology described earlier,our RAN simulator feeds our prototype with LTE traces (UEs’signaling/data traffic).

Finding 5: Our idle-mode optimization cuts the peakof overall CPU usage of UE containers by 51-73%.Fig. 10-Right shows that instantaneous peak of CPU usage ofUE containers for a variable number of UEs in the optimizedand basic SoftBox networks. We observe that the optimizationlowers the global peak of usage computed over 3 weeks byalmost 51-73% regardless of the number of attached UEs. Thereason is that a small fraction of UEs is simultaneously in theactive mode at any point in time, which could be true for anycellular network of any size. Fig. 10-Left provides a dailyview of the CPU usage optimization of 20M UE containersover 3 weeks, showing that our optimization reduces local(daily) peaks of CPU usage by 65-82%. The dramatic savingssuggest the potential for dynamically packing more containerson underutilized CPU cores, and reducing the overall neededCPU cores. We leave this to future work.

2) Benefits of Migration Optimization: Using the same LTEtraces, we show that our migration optimization of UE con-tainers simultaneously and efficiently (i) reduces the numberof migrations and (ii) balances the load of UE containers onDCs. In a broader sense, it enables us to provision fewerresources for the global controller and UE containers for agiven number of UEs. Our design of distributed migrationscheme, online migration algorithms (least-loaded-mobility(LLM), least-loaded-proximity (LLP)), and our implementa-tion of them discussed in §VI and earlier in this section.To show the potential for migration optimization, we comparethe LLM/LLP-equipped SoftBox with the unoptimized Soft-Box (baseline) that chooses an eligible DC in migration eventsrandomly. To evaluate the performance of our algorithms,we conduct Empirical Competitive Analysis: we measure theperformance of the LLM/LLP algorithm over optimal offlinealgorithms. In our experiments, we change different parame-ters (e.g., # DCs, # UEs, migration threshold).

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 453

Fig. 11. Effects of optimizing UE container migrations.

Fig. 12. Efficiency of our migration algorithms.

Finding 6: Our optimization significantly cuts thecontainer migration costs in low latency SoftBoxnetworks. We study an extreme case where the goal isto provide ultra-low latency services for all UEs in thenetwork. Thus, we set the migration threshold (max UE-UEcontainer latency) to 0.02ms. This extremely low value putsSoftBox under significant pressure and thus better illustratesthe implications of our optimization. Moreover, we evenlydistribute 100 DCs among cells and vary the number of UEs.Fig. 11 shows the peak number of migrations per secondand the peak DC load for different systems over 2 weekswith respect to the UE count. Compared to the baseline,we observe the online LLP and LLM algorithms on averagereduce the peak number of migrations by up to 66% and77%. Also, LLP and LLM on average cause by about 78%and 71% better DC load balancing respectively. Fig. 12 showsthe daily (local) peaks of UE container migrations and DCloads. We observe our LLM/LLP algorithm is close to theoptimal offline algorithm for each goal. LLM is more efficientthan LLP in reducing migrations due to considering UEs’mobility pattern in the past time windows (set to 7 days) inits decisions (Fig. 12-Left). LLP shows a higher performancein the DC load balancing than LLM (Fig. 12-Right) as itcauses more migrations and thus balances the DCs load morefrequently.

Finding 7: Our online algorithms for the migrationoptimization are efficient and reasonably close to optimaloffline algorithms. We show the efficiency of our algorithmsis not limited to one case by conducting empirical competitiveanalysis (defined earlier). We repeat the above 2-week longexperiment for every possible point in the space where themigration threshold varies from 0.005 to 10ms, and thenumber of DCs varies from 100 to 1K. In this large setof experiments, (i) LLM and LLP are at most 2.99× and8.53× worse than the optimal offline migration optimizer,and (ii) LLM and LLP are at most 3.83× and 2.91× worse

Fig. 13. Effects of our traffic steering optimization.

than the optimal offline DC load balancer. Overall, LLMis more efficient than LLP. Note that LLM and LLP aremulti-objective and online while the optimal algorithms aresingle-objective and offline, so we believe these are promisingresults.

3) Benefit of Traffic Steering Optimization: We nowvalidate if our enhanced segment routing scheme steeringUEs traffic through UE containers substantially minimizes (i)flow rules in SoftBox’s SDN data plane and (2) flow ruleupdates when UEs and UE containers move in the network.Our baseline is to establish a separate tunnel for each UEsimilar to EPC. Recall that our large-scale steering problemis unique to SoftBox and there is no other prior solution for itto the best of our knowledge. We continue using the previoussetup with the container migration threshold of 0.02 ms and100 DCs.

Finding 8: Our enhanced segment routing significantlyreduces tunnels and flow rules in SoftBox. As the numberof mobile UEs increases from 5M to 20M, Fig.13-Left depictsthe maximum number of flow rules across SDN switchesin the network (When both downlink and uplink traffic aresteered through UE containers). The max value stays below1K for the optimized SoftBox since it aggregates differentUEs’ traffic into small set of pre-established tunnel segments(derived through our recursive middlebox abstraction) andemploys source routing at RANs and Internet gateways tominimize rules needed for switching tunnels in the core.Meanwhile, the max value rapidly grows to 447K for thebaseline due to its installation of two end-to-end tunnels perUE into the data plane. It is worthwhile that the optimizedSoftBox demonstrates on average 0.5% and 0.3% smallerpacket drops and forwarding delays compared to the baselineas it precomputes and reuses minimal tunnels in the data plane.

Finding 9: Our enhanced segment routing minimizesthe data plane reconfiguration rate in peak mobility.Because UE containers and UEs move, established tunnelsfor UEs must be modified to ensure correct traffic steering.For 20M mobile UEs, Fig.13-Right depicts the flow rulemodification rates in the optimized SoftBox and baseline. Asmaller update rate means a lower risk of transient packetforwarding delays. The baseline issues by up to 4.7M ruleupdates per sec to the SDN switches in peak mobility hours.Due to establishing a separate tunnel for each UE, it ismore sensitive to mobility events and modifies rules in many

454 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

TABLE VI

EFFECTS OF OUR CONTAINER DISCOVERY OPTIMIZATION

switches in response to them. A large fraction of the updatesby the baseline maps to aggregation layer switches in theWAN and mini DCs. Updating this type of switches is riskierthan edge layer software switches on servers. In contrast,because the optimized SoftBox reuses pre-established tunnelsegments in the data plane, it does not need to update flowrules in its SDN switches. It only issues at most 700K per seccommands (not flow rules) to access layer switches on serversto connect UE containers to existing tunnel segments (see §V).

4) Benefits of UE container Discovery Optimization: InSoftBox, a UE container discovery request is triggered byan eNodeB when it starts serving a UE but does not knowthe network location of its corresponding UE controller.We designed a distributed mobility-aware protocol betweeneNodeBs and UE containers to optimize the performance andscalability of SoftBox in handling such discovery requests(§VII.A). Our key idea was to locally resolve them by placinga minimal container registry service at eNodeBs and havingUE controllers proactively and dynamically register with themas UEs move in the network. Here, we compare our protocolagainst the centralized approach as the baseline, where a globalregistry service co-located with our global controller handlesdiscovery requests from RANs.

Finding 10: Our optimized mobility-aware containerdiscovery protocol has near-zero resolution time. For avariable number of mobile UEs from 5M to 20M in the net-work, Table VI shows the average of UE container discoverytimes aggregated across different eNodeBs. We observe thatour protocol offers near-zero UE container discovery timesbecause it enables eNodeBs to resolve their queries locally.In contrast, the centralized approach has large discovery timesof 2.2-8.72ms, increasing the control plane delay in SoftBoxand degrading QoE experienced by mobile UEs. There aretwo reasons for this trend. First, the queuing delay at thecentral registry increases with respect to the number of UEs.Second, the propagation delay caused by round trips betweenRANs and the central registry significantly contributes to thediscovery time.

IX. RELATED WORK

Software EPC systems. Many virtual EPC systems(e.g., [12], [13]) port each of the EPC nodes to a VM to pro-vision and scale the nodes based on time-varying traffic load.They offer almost no changes to the EPC architecture, build onEPC’s inefficient policy management scheme, and thus inheritmost of its weaknesses. Existing SDN EPC systems in theliterature (e.g., [1], [15], [17]) are targeted on providing inde-pendent scaling of control and data planes and global routingof traffic in EPC. These systems decouple the EPC control and

data planes, centralize its control plane (i.e., MME, PCRF)on a logical SDN controller and disaggregate its data plane(e.g., in S/PGW-Data) into single-function middleboxes(e.g., DPI). The decoupling/disaggregation further scatters theEPC policies around the network and reproduces the knownEPC issues in the SDN environment [48]. In a parallel work,PEPC [14] suggests a per-UE EPC-in-a-box design to increasethe EPC packet forwarding rate. While there are some simi-larities between the SoftBox and PEPC proposals, differencesbetween them in terms of architecture and functionality aremajor. SoftBox is a clean-slate architecture that completelyredesigns and optimizes the core while PEPC is a differentform of vEPC deployment. Unlike SoftBox, PEPC (i) doesnot provide mechanisms for realizing ultra-low delays in thecore, (ii) its signaling overhead is similar to EPC due torunning EPC’s complex nodes (e.g., HSS) and protocols (e.g.,Diameter, GTP-C, SCTP), and (iii) more importantly, lacksglobal SDN/NFV control over the core that SoftBox providesto identify and address four critical optimization problems.

Multi-access edge computing (MEC). Reference [31] isa conceptual proposal for deploying general cloud servicesclose to users in cellular networks. ACACIA [49] is an MECrealization for the AR application. Unlike SoftBox, MEC isnot a cellular core architecture and almost does not touch theEPC stack. CloudLet [50] and MobiScud [51] create per-userboxes similar to SoftBox but for a different problem. Theyview the core as a “blackbox” and accelerate UE apps byoffloading their execution to clouds. In contrast, SoftBox isan architecture redesigning the core from ground up.

NFV research. Simple [30] uses OpenFlow to steer traf-fic through distributed middlebox chains. Such systems aredesigned for the traditional NFV. SoftBox consolidates eachUE’s NFs into a box, thus eliminating such complexities.Moreover, a wide range of packet processing platforms (e.g.,DPDK), high-performance NFV libraries (e.g., ClickOS [52])improve the efficiency of software packet processing. SoftBoxcan benefit from them. Our prototype uses Netfilter whileachieving a high throughput.

Container networking/migration. In the ISP context,CORD [53] leverages containers and virtualizes the single-point functionality on a “residential gateway” into a singlecontainer in the central office. SoftBox is conceptually dif-ferent from CORD as it consolidates the distributed EPCinto UE containers close to RANs and solves challengesspecific to cellular networks (§I, §III.B). There are works inVM placement algorithms both for intra and inter DCs [33],[54]. These algorithms are not well-suited for our containermigration problem dealing with unplanned UE mobility andorders of magnitude more boxes.

X. CONCLUSION

We proposed SoftBox, a novel architecture for the cellularcore, which addresses key limitations of EPC and SDN/NFVsolutions and enables customized, low latency, and signaling-efficient services on a per-UE basis. SoftBox consolidatesthe policies associated with each UE into a container in itsproximity. SoftBox has been designed to be incrementally

MORADI et al.: SOFTBOX: CUSTOMIZABLE, LOW-LATENCY, AND SCALABLE 5G CORE NETWORK ARCHITECTURE 455

deployable and scale to a large number of UEs, with spe-cial attention to efficient schemes for further minimizing theresource usage of UE containers, the migration costs of UEcontainers, data plane forwarding states, and costs of signalingcommunications between the SoftBox core and LTE RANs.In particular, our optimized version of SoftBox is equippedwith self-optimizing UE containers, enhanced segment routingprotocol, mobility-aware container migration schemes, andfast protocols for communication with RANs. Our results arepromising and point to the practical feasibility and potentialof the SoftBox concept.

ACKNOWLEDGMENT

The authors would like to thank the anonymous reviewersand Dr. Hamed Yousefi, Jeremy Erickson, Ofir Weisse, andAshkan Nikravesh at the University of Michigan for providingvaluable feedback on our work.

REFERENCES

[1] X. Jin, J. Rexford, L. Vanbever, and L. E. Li, “SoftCell: Scalable andflexible cellular core network architecture,” in Proc. ACM CoNEXT,2013, pp. 163–174.

[2] K. Zarifis et al., “Diagnosing path inflation of mobile client traffic,” inProc. PAM, 2014, pp. 23–33.

[3] Oracle Communications LTE Diameter Signaling Index. Accessed:Mar. 1, 2018. [Online]. Available: https://goo.gl/2k0niu

[4] LTE Design and Deployment Strategies. Accessed: Mar. 1, 2018.[Online]. Available: https://goo.gl/wZXhov

[5] 5G Vision: 100 Billion Connections, 1 ms Latency, and 10Gbps Throughput. Accessed: Mar. 1, 2018. [Online]. Available:https://goo.gl/dpYvW5

[6] 5G Systems: Enabling the Transformation of Industry and Society.Accessed: Mar. 1, 2018. [Online]. Available: https://goo.gl/hrQec7

[7] 5G PPP Architecture Working Group: View on 5G Architecture.Accessed: Mar. 1, 2018. [Online]. Available:https://goo.gl/EqAeao

[8] 5G Network Architecture: Network Architecture A High-Level Perspec-tive. Accessed: Mar. 1, 2018. [Online]. Available: https://goo.gl/frAZxN

[9] T. Taleb, B. Mada, M.-I. Corici, A. Nakao, and H. Flinck, “PERMIT:Network slicing for personalized 5G mobile telecommunications,” IEEECommun. Mag., vol. 55, no. 5, pp. 88–93, May 2017.

[10] T. Taleb, A. Ksentini, and A. Kobbane, “Lightweight mobile corenetworks for machine type communications,” IEEE Access, vol. 2,pp. 1128–1137, 2014.

[11] AT&T Unveils 5G Roadmap Including Trials in 2016. Accessed:Mar. 2018. [Online]. Available: http://goo.gl/eA0wSu

[12] A. Banerjee, S. Rangarajan, K. Van der Merwe, S. Kaseram, K. Sundare-san, and R. Mahindra, “Scaling the LTE Scaling the LTE control-planefor future mobile access,” in Proc. ACM CoNEXT, 2015, Art. no. 19.

[13] Z. A. Qazi, S. R. Das, K. Joshi, V. Gopalakrishnan, V. Sekar, andP. K. Penumarthi, “KLEIN: A minimally disruptive design for an elasticcellular core,” in Proc. ACM SOSR, 2016, Art. no. 2.

[14] Z. Qazi et al., “A high performance packet core for next generationcellular networks,” in Proc. ACM SIGCOMM, 2017, pp. 348–361.

[15] M. Moradi, Z. M. Mao, L. E. Li, and W. Wu, “SoftMoW: Recursiveand reconfigurable cellular WAN architecture,” in Proc. ACM CoNEXT,2014, pp. 377–390.

[16] M. Moradi, Z. M. Mao, and L. E. Li, “SoftMoW: A dynamic andscalable software defined architecture for cellular WANs,” in HotSDN,2014.

[17] M-Cord: Mobile CORD. Accessed: Mar. 1, 2018. [Online]. Available:https://goo.gl/9irf8H

[18] EPC/LTE-in-a-Box. Accessed: Mar. 1, 2018. [Online]. Available:https://goo.gl/AdpKBU

[19] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio, “An updated perfor-mance comparison of virtual machines and Linux containers,” in Proc.IEEE ISPASS, Mar. 2015, pp. 171–172.

[20] Google Kubernetes. Accessed: Mar. 1, 2018. [Online]. Available:https://kubernetes.io/

[21] Docker Containers. Accessed: Mar. 1, 2018. [Online]. Available:https://goo.gl/hxsJZc

[22] RYU Controller. Accessed: Mar. 1, 2018. [Online]. Available:https://goo.gl/TK7TSS

[23] OpenAirInterface EPC. Accessed: Mar. 1, 2018. [Online]. Available:http://openairinterface.org

[24] PhoneLab: A Smartphone Platform Testbed. Accessed: Mar. 1, 2018.[Online]. Available: https://phone-lab.org/

[25] OpenEPC. Accessed: Mar. 1, 2018. [Online]. Available: http://www.openepc.com/

[26] Y. Lin et al., “Pausing and resuming network flows using program-mable buffers,” in Proc. ACM SOSR, 2018. [Online]. Available: https://conferences.sigcomm.org/sosr/2018/sosr18-finals/sosr18-final27.pdf

[27] Handling of Signaling Storms in Mobile Networks. Accessed:Mar. 1, 2018. [Onlie]. Available: https://goo.gl/XWyFWF

[28] A. S. Rajan et al., “Understanding the bottlenecks in virtualizing cellularcore network functions,” in Proc. LANMAN, 2015, pp. 1–6.

[29] 3GPP Specifications. Accessed: Mar. 1, 2018. [Online]. Available:http://www.3gpp.org/specifications

[30] Z. A. Qazi, C. C. Tu, L. Chiang, R. Miao, V. Sekar, and M. Yu,“SIMPLE-fying middlebox policy enforcement using SDN,” CCR, 2013.

[31] M. Patel et al., “Mobile-edge computing introductory technical whitepaper,” Eur. Telecommun. Standards Inst., Sophia Antipolis, France,White Paper, 2014.

[32] L. Zhang et al., “Picocenter: Supporting long-lived, mostly-idle applica-tions in cloud environments,” in Proc. ACM EuroSys, 2016, Art. no. 37.

[33] X. Meng, L. Zhang, and V. Pappas, “Improving the scalability of datacenter networks with traffic-aware virtual machine placement,” in Proc.IEEE INFOCOM, Mar. 2010, pp. 1154–1162.

[34] A. Sharma, A. Yadav, D. Westbrook, A. Venkataramani, H. Uppal, andX. Tie, “A global name service for a highly mobile internetwork,” inProc. ACM SIGCOMM, 2014, pp. 247–258.

[35] U. Drepper and I. Molnar, “The native POSIX thread library for Linux,”Red Hat Inc, Raleigh, NC, USA, White Paper, 2003.

[36] Supervisor: A Process Control System. Accessed: Mar. 1, 2018. [Online].Available: http://supervisord.org/

[37] The Cgroup Freezer. Accessed: Mar. 1, 2018. [Online]. Available:https://goo.gl/lHhaeD

[38] Segment Routing. Accessed: Mar. 1, 2018. [Online]. Available:http://www.segment-routing.net/

[39] P. Bosshart et al., “P4: Programming protocol-independent packetprocessors,” ACM SIGCOMM Comput. Commun. Rev., vol. 44, no. 3,pp. 87–95, 2014.

[40] M. Moradi et al., “Caesar: High-speed and memory-efficient forwardingengine for future internet architecture,” in Proc. ACM/IEEE ANCS,May 2015, pp. 171–182.

[41] Migrating Stateful Containers. Accessed: Mar. 1, 2018. [Online]. Avail-able: https://goo.gl/KJZgcR

[42] Reliable UDP (RDUP) Protocol. Accessed: Mar. 1, 2018. [Online].Available: https://goo.gl/F5Z3ls

[43] PhantomNet: LTE/EPC Testbed. Accessed: Mar. 1, 2018. [Online].Available: https://www.phantomnet.org/

[44] N. Spring, R. Mahajan, D. Wetherall, and T. Anderson, “Measuring ISPtopologies with rocketfuel,” ACM SIGCOMM Comput. Commun. Rev.,vol. 32, no. 4, pp. 133–145, 2002.

[45] Netfilter. Accessed: Mar. 1, 2018. [Online]. Available:http://www.netfilter.org/

[46] nDPI: Open-source DPI Function. Accessed: Mar. 1, 2018. [Online].Available: http://goo.gl/1Nc3QC

[47] M. Ester, X. Xu, J. Sander, and H.-P. Kriegel, “A density-based algo-rithm for discovering clusters a density-based algorithm for discoveringclusters in large spatial databases with noise,” in Proc. KDD, 1996,pp. 226–231.

[48] B. Han, V. Gopalakrishnan, L. Ji, and S. Lee, “Network function virtual-ization: Challenges and opportunities for innovations,” IEEE Commun.Mag., vol. 53, no. 2, pp. 90–97, Feb. 2015.

[49] J. Cho et al., “ACACIA: Context-aware edge computing for continuousinteractive applications,” in Proc. ACM CoNEXT, 2016, pp. 375–389.

[50] M. Satyanarayanan et al., “The case for VM-based cloudlets inmobile computing,” IEEE Pervasive Comput., vol. 8, no. 4, pp. 14–23,Oct. 2009.

[51] K. Wang, K. Webb, J. Van der Merwe, A. Banerjee, J. Cho, and M. Shen,“MobiScud: A fast moving personal cloud in the mobile network,” inProc. Workshop All Things Cellular, 2015, pp. 19–24.

456 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 36, NO. 3, MARCH 2018

[52] J. Martins et al., “ClickOS and the art of network function virtualiza-tion,” in Proc. USENIX NSDI, 2014, pp. 459–473.

[53] Virtual Subscriber Gateway (vSG). Accessed: Mar. 1, 2018. [Online].Available: http://goo.gl/ySflpc

[54] M. Alicherry and T. V. Lakshman, “Network aware resource allo-cation in distributed clouds,” in Proc. IEEE INFOCOM, Mar. 2012,pp. 963–971.

Mehrdad Moradi received the B.S. degree insoftware engineering from the Sharif University ofTechnology, Iran, in 2012. He is currently pursuingthe Ph.D. degree with the Department of Electri-cal Engineering and Computer Science, Universityof Michigan, Ann Arbor, MI, USA. His researchinterests are in the design and development ofhighly scalable, reliable, and secure network systemsand architectures. His recent research focus is insoftware-defined networking and network functionvirtualization.

Yikai Lin received the B.Eng. degree in commu-nication engineering from the Beijing Universityof Posts and Telecommunications, China, in 2015.He is currently pursuing the Ph.D. degree with theDepartment of Electrical Engineering and ComputerScience, University of Michigan, Ann Arbor, MI,USA. His research interests are in networking andcomputer systems, especially in software-definednetworking and network function virtualization. Hehas been actively studying the application of SDNto several areas, including cellular networks andmanufacturing.

Z. Morley Mao received the Ph.D. degree incomputer science from the University of Californiaat Berkeley, Berkeley, CA, USA. She is currentlya Professor with the Department of ElectricalEngineering and Computer Science, Universityof Michigan, Ann Arbor, MI, USA. She hasauthored or co-authored 148 peer-reviewed researcharticles, and holds eight awarded patents. Herresearch interests include networking, distributedsystems, and security. She was a recipient of theU.S. National Science Foundation Career Award,

the Sloan Fellowship, and the IBM Faculty Partnership Award.

Subhabrata Sen (M’01–SM’14–F’16) received thePh.D. degree in computer science from the Uni-versity of Massachusetts, Amherst, MA, USA, in2001. He is currently a Lead Scientist at AT&TLabs–Research, where he has been since 2001. Hisresearch interests include Internet technologies andapplications, IP network management, applicationand network performance, video streaming, cross-layer interactions and optimizations for cellular net-works, network measurements, and traffic analysis.He has authored or co-authored 96 peer-reviewed

research articles, and holds 62 awarded patents. He was a recipient of theAT&T Science and Technology Medal, the AT&T Labs President ExcellenceAward, and the AT&T CTO Innovation Award. He is a co-inventor of thewidely used open-source Application Resource Optimizer tool for analyzingcellular friendliness of mobile app designs that earned top industry honors,including the American Business Awards’ Tech Innovation of the Year GoldStevie Award, in 2013. He served in the past as an Editor of the IEEE/ACMTRANSACTIONS ON NETWORKING.

Oliver Spatscheck received the Ph.D. degree fromthe University of Arizona, Tucson, AZ, USA,in 1999. He joined AT&T Labs in 1999. He wasnamed AT&T fellow in 2013 and is currentlythe Assistant Vice President for Cloud Strategy,Architecture, and Innovation at AT&T. He holdsover 100 patents, co-authored a book on webcaching and content distribution, and published over75 articles. He was also the co-inventor of suchsystems as GS-Tool, IDNS, and ARO which areused within and outside of AT&T. He is currently

focused on AT&T’s edge cloud architecture, technology, and application.

View publication statsView publication stats