SCOR: Software-defined ConstrainedOptimal Routing Platform for SDN

Siamak Layeghy, Farzaneh Pakzad and Marius PortmannSchool of ITEE, The University of Queensland

Brisbane, AustraliaEmail: siamak.layeghy@uq.net.au, farzaneh.pakzad@uq.net.au, marius@ieee.org

AbstractA Software-defined Constrained Optimal Routing(SCOR) platform is introduced as a Northbound interface in SDNarchitecture. It is based on constraint programming techniquesand is implemented in MiniZinc modelling language. Usingconstraint programming techniques in this Northbound interfacehas created an efficient tool for implementing complex Qualityof Service routing applications in a few lines of code. The codeincludes only the problem statement and the solution is found by ageneral solver program. A routing framework is introduced basedon SDNs architecture model which uses SCOR as its Northboundinterface and an upper layer of applications implemented inSCOR. Performance of a few implemented routing applicationsare evaluated in different network topologies, network sizes andvarious number of concurrent flows.

I. INTRODUCTION

Software Defined Networking (SDN) is introduced to im-prove network programmability, simplify network manage-ment and enhance network operation. It is a paradigm shift innetworking that suggests to decouple the control plane fromthe data plane and place the control of the system in a logicallycentralised node called SDN controller. Consequently, SDNcontroller has a view of the entire network resulting in therapid deployment of the network changes [1], [2], [3].

Fig. 1 shows the logical view of SDN architecture. Dataplane or forwarding elements, such as switches and routers,are located at the bottom layer. The next layer up is the controlplane or Network Operating System (NOS) which has a wholeview of the network and the ability to install, update and deleteforwarding rules on forwarding elements. At the top layer isthe application layer where the high level policy decisions suchas routing, traffic engineering, and security are implemented[4].

The control plane, SDN controller, communicates with thedata plane via a Southbound interface, as shown in Fig. 1.The most dominant Southbound interface is OpenFlow, whichallows the SDN controller to install, update and remove flowrules on the flow tables of the switches. It also make itpossible for the switches to send OpenFlow messages to thecontroller [1], [5]. The communication between the controllerand network applications is made possible using anotherinterface, the Northbound Application Programming Interface(API). It enables applications to program the network and /or request services from it. There are a range of NorthboundAPIs which provide various functionalities though a standardNorthbound interface is not introduced yet [6].

Infrastructure layer

Control layer

Application layer

NetworkOperatingSystem

Network ServiceNetwork ServiceNetwork Service

Southbound API(e.g. OpenFlow)

Application ApplicationApplication

Northbound API

Application

Fig. 1. SDN Architecture [4]

Several network applications are proposed in SDN whichaddress different Traffic Engineering (TE) requirements suchas load balancing and link utilisation optimisation. Todaysnetworking environment necessitates using various Quality ofService (QoS) routing and TE techniques to optimise traffic fordifferent media applications like voice and video. The imple-mentation of these QoS routing and TE applications might becomplex in some cases. Since there is no standard Northboundinterface these applications require various implementationcompatible with various SDN controllers. Despite the vitalimportance of QoS routing and TE in handling media trafficand network optimisation, none of the proposed Northboundinterfaces provides specific abstractions or functions for im-plementing them.

In order to address this concern, this work is introducing aSoftware-defined Constrained Optimal Routing (SCOR) plat-form. This includes a northbound interface which is basedon Constraint Programming (CP) techniques that have beensuccessfully applied in similar problems e.g. vehicle routing.The main idea behind using CP methods for this NorthboundAPI is to separate complexity from the user. In CP, users do notspecify step by step solution of the problem, as in the caseof procedural programming. They only state the constraintsthat the solution should have. In other words, in CP, userstates the problem and the solution of the problem is thenfound by another program, the solver. This remedies users

arX

iv:1

607.

0324

3v1

[cs

.NI]

12

Jul 2

016

from searching in a wide range of possible algorithms forimplementing every single QoS or TE problem. This facilitymakes it possible to efficiently develop applications usingexpressive and declarative programming.

The proposed Northbound API is controller independentand does not use specific features of any controller whichmakes it a more appropriate API for developing QoSrouting and TE applications. A set of eight primitives asa new layer of API is also created which constituents aninterface for developing QoS routing and TE applications.A few example QoS routing algorithms are implementedin the form of several applications. The performance ofthese implemented applications are evaluated in variousexperiments. A series of networks with various sizes andtopologies utilised in these experiments. The results of theseevaluations indicate a practical performance for even largescale networks. Explanation of the new Northbound API isstarted with a brief background in Section II, then relatedworks are presented in Section III. Section IV-A explains thearchitecture of a routing framework for SDN which includesthe proposed Northbound API and Section V describesits implementation. Section VII discusses the results andevaluation and Section VIII concludes the paper.

II. BACKGROUNDA. Routing Network Flows

Routing in general, SDN-based or not, involves two entities,network state information and routing algorithms. Networkstate dynamically changes due to transient load fluctuations,connections in and out and links up and down, so it constitutesa significant part of traffic load and uses resources of therouting devices in many networks. In legacy networks, networkstate information is gathered via distributed routing protocolswhich acquire and distribute it from and to routing devices.In SDN though, the network state information is centrallycollected and updated by the controller that directly commu-nicates with routing devices. Consequently, maintaining andupdating network state information through SDN controllerfrees the resources from the network and devices [7].

Routing algorithms are routines or formulas used to makedecisions concerning the paths for network flows. The purposeof a routing algorithm is in general determined by the flowscharacteristics such as source-destination pairs and the demandas well as services which a provider may want to provideto its customers e.g. guaranteed bandwidth. Since there arefinite resources in a network, e.g. capacity, an objective for anetwork in general is to provide an efficient and fair routing[8].

The network flow routing problems are mathematicallystated in terms of graph notions. Wherever graph is referred inthis work, it is assumed directed. A directed graph G(N ,A),is a set of nodes N and a set of arcs A which is representedby a set of ordered pairs of distinct nodes (A = {(u, v)|u N and v N}). The number of nodes is N and the numberof arcs is A. An arc is an ordered pair so, the arc (u, v)

1

2 3

5

4

C = 30L = 1

C = 100

L = 1C = 30

L = 1

C = 20L = 2

C= 20L = 2

C = 10, L = 2

(A)

Link array

StartNode

EndNode

Capacity (C)

Cost = Loss (L)

1 2 30 1

1 4 10 2

1 5 20 2

2 1 30 1

2 3 100 1

3 2 100 1

3 4 30 1

4 1 10 2

4 3 30 1

4 5 20 2

5 1 20 2

5 4 20 2

(B)

Fig. 2. A sample network graph and its link array

is distinguished from the arc (v, u) (arc and link are usedequivalently in this text). There are multiple concepts andnotions relating to graphs used in this work which are requiredto be defined before use.

An arc (u, v) is said outgoing from node u and incoming tothe node v. The node u is called the start node and the nodev is called the end node of the arc (u, v) and the arc is calledincident to its end nodes u and v. The degree of a node u isthe number of arcs incident to u. It is assumed there is onlyone arc between each pair of nodes in the same direction. Anundirected graph can be replaced with a directed graph whereevery undirected arc (u, v) is replaced with two directed arcs(u, v) and (v, u) [9].

For the purpose of programming, a graph (either directedor undirected) can be represented in two ways, an adjacencymatrix or an adjacency list. In the adjacency matrix, graphnodes are numbered from 1 to N and a N N matrix M =(muv) represents the graph where

muv =

{1 if (u, v) A0 otherwise

(1)

In the algorithms implemented in this work the adjacency-list representation is used for all the graphs. It is an array Lwhich includes the adjacency lists of all nodes. The adjacencylist of a node is the list of all the nodes for which there isan outgoing arc [10]. This concept is illustrated for a samplenetwork graph in Fig. 2-A with five nodes and six bidirectionallinks (bidirectional links are represented as two unidirectionallinks and so there are 12 unidirectional links). The adjacencylist of the graph is shown in the first two columns of the tablein Fig. 2-B and the two next columns are other link parametersthat will be explained and used in the later sections. Theadjacency list for the node number 1, for instance, containsall the nodes for which there is an outgoing arc from the nodenumber 1 that includes nodes 2, 4 and 5. Node number 1is considered in the adjacency lists of these nodes, for thepurpose of its incoming links, which are expressed in 4th, 8thand 11th rows.

A path P in a graph is defined as a sequence ofnodes (n1, n2, ..., nk) with its corresponding sequence of(a1, a2, ..., ak1) arcs, where k 2. In this arc sequence theith arc, ai, is either (ni, ni+1) which is called forward arc or(ni+1, ni) that is called backward arc. Nodes n1 and nk arecalled start node (or origin) and end node (or destination)of the path P . A path is called simple path if it includesno repeated arcs and nodes except the origin and destinationnodes. It is important to know that a sequence of nodes byitself (without its corresponding sequence of arcs) does notspecify a path unless the path is either forward (all arcsforward) or backward (all arcs backward). In all usage of thepath in this work, it is assumed forward (unless it is statedotherwise) and a sequence of nodes (or equivalently a sequenceof arcs) adequately identifies a path.

A flow is a variable to measure the quantity of what isflowing through an arc. The flow of an arc (u, v) can bestated mathematically by a real number fuv (or f(u, v)) thatis assumed non-negative without loss of generality. There isanother concept related to flows in the network which is calleddivergence. The divergence of a node u is the total flow exitingthe node less the total flow entering the node [9]

yu =

{v|(u,v)A}

f(u, v)

{v|(u,v)A}

f(v, u) u N

(2)

In order to explain the flow path in a network, the flowvector concept is used. A flow vector represents a set of flowsf = {f(u, v)|(u, v) A} in the graph G. A simple path flowis a flow vector , i.e. set of arc flows, that is equivalent to senda (positive) flow along that simple path. For the simple pathP it is mathematically stated by a flow vector f

f(u, v) =

a if (u, v) P+

a if (u, v) P

0 otherwise(3)

where a is a positive scalar and P+ and P are forward andbackward subsets of arcs of P respectively. A flow vector fcan have two distinguished nodes the source and sink whichhave positive and negative divergence respectively. A path Pis said conformed to a flow vector f if the flow for all forwardarcs is positive and it is negative for all backward arcs, andthe paths start and end nodes are the source and the sink ofthe flow vector f respectively, if they are not the same.

The flow vectors studied in this work are constrained bytwo constraints. The first constraint is called the capacityconstraint which limits the flows to a lower and upper boundsas [9]

b(u, v) f(u, v) c(u, v) (u, v) A (4)

in which b(u, v) and c(u, v) are called the flow bounds of thearc (u, v). The second constraint is called the conservationof flow that states [9]{v|(u,v)A}

f(u, v)

{v|(u,v)A}

f(v, u) = su u N

(5)

In the above equation if su > 0, it is called the supply of thenode u, and if su < 0, the scalar su is called the demandof the node u. The flow conservation simply states that thetotal flows exiting a node is equal to flows entering the nodeunless the node is either a flow source or sink. A flow vectorsatisfying capacity and flow conservation constraints is calledfeasible flow vector.

The capacity of an arc (u, v) is a non-negative real numbercuv (or c(u, v)) that is the upper flow bound of the arc.The capacity for the graph G can be defined as a functionc : A R+0 that maps the graphs arcs (ordered pairs) tothe non-negative real numbers. If there are two distinguishednodes in a graph G as the flow source s and sink t then thegraph G along with its capacity function c is a flow network(G, c, s, t) [11].

The problem in a routing algorithm usually consists offinding a path from a flow source to a flow sink that optimisesa parameter, e.g. minimising a linear cost function, subject tosome constraints. Some routing algorithms require to satisfya condition on a single or multiple parameters rather thanoptimising them. If the constraint satisfaction or optimisationin finding paths include parameters to satisfy the QoS, therouting algorithm is called the QoS routing algorithm [12].Examples of these algorithms along with their mathematicalstatements are presented in section V.

The concept of the QoS represents the quantitative and / orqualitative performance criteria in contract between serviceprovider and its customers. These requirements are givenas a set of constraints on a link, single or multiple pathin the network [13]. Hence, QoS routing is finding routesin the network based on these requirements which is thefirst step toward achieving end-to-end QoS guarantees [14].The QoS routing problem, i.e. identifying feasible paths thatcan satisfy the given QoS constraints, constitutes one of thekey issues in all QoS-based networking framework such asintegrated services (IntServ), diffrentiated services (DiffServ),and multiprotocol label switching (MPLS) [15]. The Intservframework requires a QoS-aware underlying IP routing tableto reserve its resources and guarantee some specific QoSconstraints. The Diffserv framework utilises the QoS-basedroutes to ensure a certain service level agreement. These routescan be requested by, for instance a network administrator, forTE objectives. In MPLS the route selection, possibly subjectto QoS constraints, and resource reservation along the path, isdone by the source node. The significance of the constraint-based routing algorithms is clearly indicated in these examplesas some of the major networking frameworks [7].

B. Constraint Programming

Constraint programming techniques were initially intro-duced in the 1960s and 1970s in artificial intelligence andcomputer graphics [16]. Then they have found applicationsin many fields such as operations research, programming lan-guages and databases [17]. They have been successfully usedto solve many practical problems in a number of fields suchas scheduling of air traffic, network security, vehicle routing,planning, chemistry, biology and bioinformatics [17], [18].Nonetheless, these techniques have potential to be applied ina further range of real-life applications and problems.

The main idea behind CP is to separate stating the problemfrom its solution. So, users are only required to state theproblem and the solution is found by the general purposeconstraint solvers which are designed for this purpose. Thisallows declarative descriptions to generate very flexible mod-elling that can be used to effectively solve large, particularlycombinatorial problems [19].

In order to have a constraint solver to solve a real worldproblem, it must be stated in the form of a CP model. A CPmodel includes at least three parts Decision variables that represent tasks, metrics or re-

sources of a real world problem. Variable domains that are a finite set of possible values

for each decision variable. Constraints that state the relations (conditions, limita-

tions, properties and bounds) between decision variables.The constraints in fact restrict the values that all decisionvariables can have at the same time [19].

As an example, consider the Shortest Path routing algo-rithm. Here it is assumed that arcs of the graph are associatedwith a scalar number a(u, v) which represents the cost andthe cost of a forward path is sum of the costs of its arcs. Thenfor a given pair of source-sink nodes the problem is to find aforward path with minimum cost that connects the source tosink. It can be stated mathematically as [9]

minimize

(u,v)A

a(u, v)f(u, v)

subject to

{v|(u,v)A}

f(u, v)

{v|(u,v)A}

f(v, u) =

1 if u = s,

1 if u = t,0 otherwise

(6)

Then the vector f of the form

f(u, v) =

{1 if (u, v) P0 otherwise

(u, v) A (7)

is a feasible flow vector for the problem (6) and the cost off is equal to the length of P . The problem can be stated as aCP model as follows:

Decision Variables:P{ni} : represents flow membership in which{ni| i in [1 N ]} represents the network nodes andDomain[P ] = {0, 1}.

Constraints:for i in [1 N ] :

P includes no repeated niSum[Flow in[ni]] Sum[Flow out[ni]] = = 1 if i is Source = -1 if i is Sink = 0 else

Objective:minimize Length [P ]

The solution of a CP model is the allocation of values forthe decision variables from their domains that simultaneouslysatisfy all the constraints. It might be possible to find

a single solution with no priority all solutions an optimal solution, given an objective function

If there exist such an objective function in terms of someor all decision variables that identifies an optimal solution(similar to the above model), the problem is called anOptimisation Problem (OP) rather than being called aConstraint Satisfaction Problem (CSP) [19]. The solversenumerate possible variable-value combinations intelligentlyand search the solution space either systematically orthrough some forms of complete or incomplete searches.The performance of these search methods depends on thestatement of the problem [17].

Link-optimization routing

Link-constrained routing

Path-optimization routing

Path-constrained routing

Link-constrained Path-optimization routing

Link-constrained Link-optimization routing

Multi-Path-constrained routing

Path-constrained Path-optimization routing

Path-constrained Link-optimization routing

Link-constrained Path-constrained routing

Multi-link-constrained routing

Basic routing problem Composite routing problem

Fig. 3. Basic and composite QoS routing problems [20]

III. RELATED WORKS

There are many works related to QoS routing problems inthe legacy networks. One of the best classifications of theQoS routing problems is presented in [20] which is shownin Fig. 3. It divides the unicast routing problems into twomajor categories, basic and composite routing problems. Thecriteria used for this classification is the number of QoS metricsapplied in the problem. A basic routing problem includes onlya single QoS metric such as delay, jitter or bandwidth. Therouting problem in this case might include the metric optimi-sation, such as minimum delay path, or it might include metricconstraint, such as bandwidth constrained path. In addition, themetric might be related to a link parameter, e.g. bandwidth, orit might be related to a path parameter such as delay. Hence,there are four types of the basic unicast routing problems asseen in Fig. 3. The composite routing problems can be usedto define multi-constraint QoS routing problems. They can beexpressed as the combination of a single-metric-optimisationproblem with one or more metric-constraint problems. Somemulti-constraint QoS routing problems can be expressed asthe combination of two or more metric-constraint problems.Depending on the metric type, is a link or path metric,several types of composite routing problems are created whichcorrespond to various proposed QoS routing problems.

In the works related to QoS routing in SDN, the North-bound API is part of SDN architecture that is expected toprovide related facilities. The Northbound and SouthboundAPIs are the two key elements of SDN framework whichprovide communication between their below and above layers.While there is a widely accepted proposal for the SouthboundAPI, i.e. OpenFlow, no common proposal is accepted forthe Northbound API. Despite the variety of the availableNorthbound APIs, little work is done on addressing the QoSrouting in these platforms. In fact, there is no NorthboundAPI to implement abstractions required for implementing allthe basic and composite QoS routing algorithms, similar tolegacy networks.

Some existing controllers such as Floodlight [21], NOX [22]and OpenDaylight [23] have proposed and defined their ownNorthbound APIs with the specific definitions. They tend todeal with the packet and port level manipulations rather thanrouting and QoS routing abstractions. Even the QoS moduleprovided by the Floodlight controller mostly deals with thehardware, packet level details and low level configurations.This makes the proposed Northbound interfaces unsuitablechoice for developing QoS routing applications that requirehigher levels of abstractions to implement QoS routing con-cepts (such as capacity constraint in the path selection).

Some programming languages such as Frenetic [24], Pyretic[25] and Hierarchical Flow Tables (HFTs) [26] also providean abstraction of the controller functions and data planebehavior for the upper layer. Pyretic, Frentic and HFTs areexamples of declarative Northbound programming languageswith facilities for expressing packet forwarding policies. Theyalso provide facilities for parallel and sequential composition

of various non-overlapping modules. Despite the abstractionsand facilities created for applying packet forwarding policiesand module composition, they still work at the packet level andno high level abstraction of QoS functions is provided yet. Fur-thermore, implementing routing and QoS routing algorithmsin these languages is done through the same algorithms thatis currently applied in legacy networks, e.g Dijkstra algorithmfor the shortest path routing in Pyretic [27]. This makes de-veloping QoS routing and TE applications a cumbersome job,that requires users be familiar with programming techniquesfor efficient implementation of algorithms in possibly hundredsof lines of code.

The PANE controller [28] provides an API that allows ap-plications such as VoIP and video to request network resourcesand reserve bandwidth for specific QoS requirements. It trulyrecognises the need for an API between the networks control-plane and its users, applications, and end-hosts. Though, thefunctionalities that PANE provide is limited to a few featuresprovided in the PANE controller and users are not able todevelop QoS routing and TE applications. As such, it can beconsidered an application rather than a platform for developingnetworking applications.

SFNet [29] is another Northbound API that emphasises onthe closer interaction between the applications and network. Itprovides some network services to the applications through ahigh-level API that hides the details of the implementation. Itsuggests a software friendly networking (SFNet) paradigm inwhich it is possible for the applications to reserve bandwidthor query network congestion state. The main idea of SFNet,i.e. to create abstractions of the network services, is one ofthe core ideas of SDN, the implementation of SFNet cannot be considered optimal. It has also a very limited scopethat focuses on services such as bandwidth reservation andqueries to request the networks congestion state. The merefunctionality of the QoS component of SFNet is to reserves thebandwidth along the shortest path between two network nodes.This can clearly be insufficient or useless in many situationsin which the shortest path lacks the enough bandwidth.

While the main purpose of SDN is to enhance innovationin the network, a platform for the efficient implementationof present and new networking ideas is a clear gap in thisfield. This is specially the case with QoS routing and TEapplications. The above Northbound APIs and programminglanguages, despite providing some features and facilities, can-not be considered the desired platform. Most of the availableNorthbound APIs are dealing with the lower level details suchas packet and hardware manipulation. Only a few NorthboundAPIs such as the QoS module of the Floodlight controllerprovide abstractions for QoS functionalities that is still dealingwith the packet and hardware level. Some of these workssuch as PANE are more considered application, rather thana Northbound API or a platform for developing applications.Some of them like SFNet have a limited scope which is farfrom a platform that can be used to address real QoS routingrequirements of todays networks.

Due to these issues, implementing QoS routing applications

in these Northbound APIs and programming languages isinconvenient, similar to developing desktop applications in As-sembly language. The users not only have to find an efficientalgorithm to solve each individual QoS routing problem, butthey also must implement it through programming languagessuch as Python, JAVA and C which might require hundredsof lines of code. As such, it necessitates both theoreticalknowledge of the competent algorithms for each QoS routingapplication and the ability for its efficient implementation. Fur-thermore, since these Northbound APIs are mostly compatiblewith their corresponding controller, applications are needed toreimplemented for each new controller.

DEFO is a very recently presented approach to controlforwarding paths in non-SDN carrier grade networks. It uses adeclarative and expressive language to efficiently address manyQoS routing requirements. It has a two-layer architecture thatseparates connectivity and optimization tasks. The high-levelgoals expressed in the declarative language is translated intocompliant network configurations by the centralized optimizer.The results of the evaluation of DEFO indicates it achievesbetter trade-offs for the classic QoS goals and supports alarger set of goals. DEFO has been able to optimise large ISPnetworks in a few seconds for the classic QoS goals whichrequired more than one night on a powerful server throughlinear programming. DEFO relies on CP and new heuristicsin order to achieve these advantages in the scalability andflexibility [30]. Although DEFO is not a Northbound API inthe SDN architecture, there are lessons to be learned fromits approach. In fact, it includes many features of a desiredNorthbound API for efficient addressing of QoS routing andTE requirements in the network.

The main advantages of DEFO are achieved by using CPtechniques for the network optimisation. This is not the firsttime CP is suggested for the network optimisation objectives.Previously, it has been suggested in [31] for efficient solving ofvarious problems in legacy networks such as TE and networkdesign. The same idea can be applied in SDN which providesmore facilities and freedom for the implementation of CP inthe network optimisation. Since the task of a QoS routingalgorithm is to find paths subject to a single or multipleconstraints, CP is an evident choice for solving these problems.

Using CP in a Northbound API makes it possible todevelop QoS routing applications without the requirement ofknowledge of various algorithms to solve each QoS routingproblem. Users state the problem in terms of CP variables andconstraints and the solver finds the solution without furtherintervention from the user. While the simplicity of statingthe problems make it a really useful tool for developingapplications, the presence of solvers with strong intelligentsearch techniques ensures the efficiency is not less than otheravailable solutions. Indeed, the DEFO example indicates it iseven much faster than similar methods for some QoS routingand TE applications.

A CP-based Northbound API, hence, can address thecurrent gap of an efficient application development platformin SDN. The QoS routing functionalities / constraints,

due to the nature of the problem, can easily be statedas some CP primitives. These primitives can model basicQoS routing problems and their combinations can be usedto express composite QoS routing problems. This is themethodology used in this work to design and implement thenew Northbound API, SCOR. A routing framework is alsoproposed in this work which includes other elements requiredby the QoS routing applications. The design and architectureof the proposed solution is discussed in the next section.

IV. DESIGNA. Architecture

Fig. 4 shows the structure of a routing framework basedon the proposed Northbound API. It has three layers as theNetwork Operating System (NOS), the Northbound API andthe Network Applications. The bottom layer of the frame-work, NOS, directly communicates with the Network Elementsthrough common protocols such as OpenFlow. It uses the threecomponents Network State Monitor, Network State Databaseand Route Calculator to acquire network state such as trafficloads and flow demands from Network Elements and provideit for the applications.

The Network State Monitor is the component of NOS thatcollects the network state information by communicating withthe Network Elements (layer). It includes four modules asHost Tracker, Topology Discovery, Traffic Load Calculatorand Flow Demand Estimator. The Host Tracker and Topol-ogy Discovery are two standard modules present in manycontrollers which provide network topology information. TheTraffic Load Calculator module periodically reads the flowstatistic of ports and keeps track of network links utilisationand available network resources [32]. The last module, FlowDemand Estimator, provides periodic estimations of the flowdemands in the network inputs.

The network state information provided by the above com-ponent is recorded in the Network State Database componentwhich is queried by the Network Applications through theRoute Calculator component. This information is needed byany network application that modifies routing or other parame-ters of the network. The Route calculator also receives networkroutes from the applications and translate them to flow rules.These flow rules are then installed in the network elementsthrough a sub-module, the Flow Installer.

The next layer up is the proposed constraint programming-based Northbound API, SCOR. It is explained in more detailsin the next sub-section as the main contribution of this work.The top layer of the framework is the Network Applications.This layer includes different applications that are created inthe SCOR. The range might include common QoS routingand TE applications such as maximum bandwidth path,maximum residual capacity path, link utilization optimization,and load balancing. It might also include applications thatimplement newer services such as service chaining. Theseapplications get the network topology and state informationfrom the NOS through route calculator component, calculate

Network Operating System

Network Applications

Network Operating System

Network Elements

Network State Monitor

Route Calculator- - - - - - - - - - - - - - - - - - - - - - - -

Flow Installer

Host Tracker

Topology Discovery

Traffic Load

Calculator

Flow Demand

Estimator

QoS Routing and TE Interface

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - CP-Based Programming Language

Maximum Residual Capacity

Widest Path Routing

Link Utilization

Optimisation

Load Balancing . . .

NetworkState

Database

Service Chaining

OpenFlow

. . .

Fig. 4. The routing framework based on SCOR

the route, and return it to route calculator.

B. QoS Routing and TE Interface

Purposing to use CP in SDN and designing the requiredprimitives is the main contribution of this work. This includesdesigning the Northbound API and an interface for developingQoS routing and TE applications. As it seen in Fig. 4, SCORconsists of two sub-layers which are equivalent to two levels ofprogramming. The bottom level is the CP-based ProgrammingLanguage and the higher level is the set of primitives whichforms the QoS Routing and TE Interface. The bottom sub-layer, CP-based programming language, can be considered asa general platform for developing routing and other types ofapplications. CP techniques have been shown efficiently fitto address many problems in network design, planning andanalysis [31]. This CP-based programming language enablesusers to state their problems in a wide range of networkingdomains by focusing on what should be done rather thanhow they should be done.

The second sub-layer, QoS routing and TE interface, isparticularly designed to address QoS routing and TE require-ments. It is consisted of two groups of predicates which totallyinclude eight predicates as listed in Table I. The first groupincludes predicates 1 to 5 which are designed to implementthe single and multi-metric QoS routing problems. Theyimplement various constraints such as the flow conservation,capacity constraint and cost, and they are fitted to realise bothbasic and composite routing problems.

The first and the most fundamental routing concept to realiseis the flow path. Theoretically, the flow path is defined as

TABLE ISET OF PREDICATES FORMING QOS ROUTING INTERFACE

Item Predicate Name Functionality

1 Network Path Finds all the paths from a flow source s toa flow sink t in the given network graph

2 Link CapacityConstraint

Places the flows on arcs that can accommo-date their demands

3 ResidualCapacity

Calculates the residual capacity of arcs,after placing all the flow demands

4 Path CapacityConstraint

Removes all arcs with a capacity less thana specified value, from the network graph

5 Path Cost Calculates the total cost of a path based onthe cost of arcs forming the path

6 Delay Calculates the queueing delay for the givennetwork graph and flows

7 Congestion Calculates the congestion for the given net-work graph and flows

8 Link Utilisation Calculates the link utilisation for the givennetwork graph and flows

a set of arc flows that complies with two constraints, flowconservation rule (Eq. 5) and the capacity constraint (Eq. 4).Though, in general it is possible to define a flow path that doesnot satisfy the capacity constraint. As such, the Network Pathpredicate applies only the first rule, the flow conservation, toidentify a flow path and the capacity constraint is designed as aseparate predicate, the Link Capacity Constraint. Combinationof these two predicates can identify a feasible flow path(Section II) in a flow network.

The next two predicates, the Residual Capacity and Path

Capacity Constraint create abstractions relating to capacitythat are frequently used in the QoS routing applications. Theformer finds the value of the residual capacity of all arcs inthe network graph, assuming all the known flow demands arerouted. The latter removes arcs with a capacity less than aspecified limit from the network graph. This is particularlyuseful when calculating routes with two or more metricsincluding the bandwidth or capacity limitation. while thispredicate applies the capacity constraint, it reduces the networkgraph and makes imposing the next constraints faster.

The path cost is the other concept required for the im-plementation of many QoS routing problems. The Path Costpredicate which realises this constraint assumes the cost of apath is the sum of its arcs (links) costs. The cost metric canbe different network parameters such as number of hops, linksdelay (transmission and propagation) or the link lease price.Current routing protocols use various network parameters asdistance or cost metric. In Open Shortest Path First (OSPF)protocol it is possible to use either of throughput of the links,link availability or Round-Trip Time (RTT) as the cost. TheRouting Information Protocol (RIP) uses the number of hops tothe destination and Interior Gateway Routing Protocol (IGRP)uses a combination of node delay and available bandwidth.

There are two cases of routing algorithms that use multiplecost metrics at the same time. In the first case, one costmetric is optimised and the rest are constrained to satisfy somelimits, which is an optimisation problem. In the second case,all cost metrics are constrained to satisfy their correspondinglimits, that is a constraint satisfaction problem. As an exampleconsider a routing algorithm that finds the path with minimumRTT and the link availability greater than specific value. Bothmetrics can be used as a cost metric and the problem is a multi-cost-metric routing problem. SCOR can easily implementthese algorithms by assigning the cost metrics to differentdecision variables and applying the Path Cost predicate foreach one separately.

Routing problems that are addressed by the group of fiveabove predicates include all problems in which one or morestatic network parameters are either optimised or constrainedas stated in Section III. Though, there are still other TE prob-lems in which the desired network parameters dynamicallychanges with the traffic. For instance, the queueing delay ineach network node, not only depends on the capacity of net-work paths, but it also depends on the amount of traffic flowinginto the network. It can be compared to the transmission delayof links which is a static parameter depending only on thecapacity of the links. In this way, another group of predicatesare needed to realise these concepts.

Accordingly, the second group of predicates which includesDelay, Congestion and Link utilisation realises constraintsrequired to implement advanced TE applications such aslink utilisation optimisation, maximum concurrent flow andnetwork load balancing. All these predicates are based onthe concept of the residual capacity which is implementedas a separate predicate. They use the Residual Capacitypredicate and illustrate the Nested Predicate feature as a

technique in efficient code re-usability. Though, it is notlimited to these predicates and it is also applied in thepredicates of the first group. These are explained thoroughlyin next section which demonstrates the implementation of theNorthbound API and also some applications created through it.

V. IMPLEMENTATIONA. Routing Framework

The routing framework is currently implemented based onthe POX controller. Nonetheless, it does not bound to anyspecific feature in POX and can be easily adopted for the othercontrollers. This is specially the case with the ONOS [33] andits intent subsystem. Intents are policy-based directives thatmakes it possible for the applications to state their networkrequirements in terms of network policies rather than themechanisms. An intent compiler translates intents, receivedfrom the applications, to actionable operations on the networkenvironment. This is very similar to the procedure happeningin SCOR. In a similar manner, users / applications statetheir requirements in SCOR in terms of high level / globalnetwork policies without knowing how to do it, and SCORcompile them to installable network flow rules. This makesONOS and intent framework a strong candidate for the futureimplementation of this routing framework.

SCOR is implemented in MiniZinc [34]. MiniZinc is asimple declarative CP modelling language. In addition to itspre-packaged solvers, e.g. Gecode [35], MiniZinc can easilyuse other available solvers such as Jacop [36] and ECLiPSe[37]. The ease of implementation, simplicity, expressivenessand compatibility with many solvers has made it a practicalchoice for the standard CP modelling language. It includes awealthy library of the global constraints which model high-level abstractions. [34].

A problem is usually stated in MiniZinc as two parts / files,the model and model data. The model uses variables (includingparameters and decision variables) and constraints to de-scribe the structure of a CP problem. The value of parametersare fixed and can be given when declared or in a separatestatement or in a separate file, the model data. The value ofdecision variables is unfixed and can only be fixed after solvingthe problem. Indeed, the whole aim of the constraint solvingis to fix the value of decision variables. Different model datafiles can be used with a single model to assign different valuesfor the parameters [38]. For instance, the Least Cost Pathmodel described in Algorithm 9 models the least cost pathalgorithm for an individual network topology which can bedefined through a model data file. Different model data filescan be used to apply a single least cost path model on variousnetwork topologies.

Figure 5 shows the flow of events in the implementedrouting framework. When a new packet arrives in a networkelement, if there is no rule installed to specify the requiredaction for that packet, a PacketIn message is sent to thecontroller. The routing framework, as a module of POX, islistening to the POX core events and receives the PacketIn

Network Elements

Controller

Network Elements

Controller

Route Calculator

MiniZinc

Route Calculator

PacketIn

Network States, Flow specification

MiniZinc Model,MiniZinc Data

A series of node-ports

Forwarding rules

Link Flow Membership

New Packet

Fig. 5. The flow of events in the implemented routing framework

events. It extracts the flow specifications (network protocol,source and destination addresses, and upper layer source anddestination ports) from the packet and passes them along withnetwork state information to the Route Calculator module.The network state collection starts from the beginning of thecontrollers operation through Topology Discovery and HostTracker modules of POX that are called in the routing frame-work. The Route Calculator module converts the receivedinformation, the flow and network state, to a MiniZinc datafile.

However, MiniZinc is not able to use this data file unlessthe model file is defined. These models express the users /applications objectives (similar to intents in ONOS) in theform of high level network policies or constraints. In thecurrent implementation, a selected MiniZinc model is definedin the Route Calculator module which is used for each of theexperiments explained in Section VII. In the next versions itwill be possible to apply a model from the set of models au-tomatically selected based on the traffic type or other networkrequirements. The Route Calculator module runs MiniZinc bycalling a chosen CP solver through command line and passingthe model and data files as arguments of the command.

Then the MiniZinc solver finds the solution of the receivedmodel-data problem and returns the output in the formof an array of 0s and 1s which is called Link FlowMembership. A 1 value in the Link-Flow-Membershiparray indicates its corresponding link is determined as part ofthe flow path by the solver, and a 0 means it is not on theflow path. The Route Calculator module reads the commandline output of MiniZinc, i.e. the Link-Flow-Membership array,and uses the network information to convert this array to asequence of node-ports which constitutes the flow path. Thissequence of node-ports is then passed to the controller whichconverts it to a series of forwarding rules to be installed ontheir corresponding network elements. Although the currentversion of the framework is implemented for calculating theconstrained network routes, it is not limited to this applicationand route calculator can be seen as a general network rule

calculator / generator.

B. MiniZinc predicates

The global constraints of MiniZinc, which state predefinedconstraints, can be added to models by using an includeexpression (e.g. include globals.mzn;). New constraints canalso be defined and included in the models through MiniZincpredicates in a similar manner. MiniZinc predicates that caninclude one or more constraints, are similar to functions ormethods in procedural programming languages. They addmore abstraction and modularisation to MiniZinc and makeprogramming the constraints more beautiful.

The predicates explained in this section use a fewparameters and decision variables that are listed in TableII along with a short description and their types. In orderto facilitate user to understand the predicates code, somecommon MiniZinc expressions with their descriptions arestated in Table III. Though, a comprehensive tutorial isavailable through [39] that provides syntaxes, descriptions,ample examples and a lot of other helpful information. Sincesome predicates use other predicates, the order in whichpredicates are explained is different from Table I.

1) Network Path Predicate: All routing applications inSCOR require to identify network paths and then apply otherconstraints which states their objectives. They rely on thispredicate for the network path definition. This predicate isbased on Eq. 5 that defines a network path as a set of arc flowswhich comply with the flow conservation rule constraint. Inall predicates explained in this section the flows are assumedunsplittable. The MiniZinc code for this predicate is shown inAlgorithm 1. Parameter and variable names are as explainedin Table II. The Links array not only includes the adjacencylist of the network (Section II), but it also includes otherparameters of the links such as the capacity and cost (seeFig. 2 for an example).

TABLE IIPARAMETER AND VARIABLE NAMES USED IN THE PREDICATES

Name Description Type

Nnodes Number of nodes Integer

Nlinks Number of links Integer

Nflows Number of flows Integer

Nodes List of nodes 1D Array of integers

Links List of start and end nodes of linksalong with their cost and capacity

2D Array of integers

LFM Link-Flow-Membership array 2D Array of [0,1]

Flows List of flow demands 1D Array of integers

s List of flow sources 1D Array of integers

d List of flow sinks (destinations) 1D Array of integers

Limits List of lower bound of capacities 1D Array of integers

TABLE IIISOME COMMON MINIZINC EXPRESSIONS

Expression Description

constraint expresses constraints to be satisfied

include accessing library

index set gives the index of a 1D array

index set1of2 gives the index of a 2D array for the 1st dimension

index set2of2 gives the index of a 2D array for the 2nd dimension

output expresses what to be printed out

solve defines constraint satisfaction or what to be optimized

The decision variable in this predicate is the LFM , Link-Flow-Membership array. The number of rows of LFM is equalto the numbers of rows of the Links array (i.e. Nlinks) andthe number of its columns is equal to the number of concurrentflows (i.e. Nflows). For instance, LFM [10, 1] = 0 indicatesthat link number 10 (i.e. the 10th row of the Links array, e.g.the link from node number 4 to node 5 in Fig. 2) does notbelong to the path of flow number 1. The objective is to finalisethe value of the decision variable. It means to determine linksand flows relationship i.e which links belong to the set of linksthat form a path for each flow.

The Network Path predicate enforces the unity flowconservation rule (Section II) for each flow in which a unitflow is generated in the flow source node, s[j], and sinkedin a flow destination node, d[j]. This necessitates sum ofthe flows entering each node i (line 3, Links[k, 2] = i meanswherever the node i is the end node of the arc) to be equalto flows exiting the node i (line 5, Links[k, 1] = i meanswherever the node i is the start node of the arc), unless thenode i is either the source of jth flow (line 6, i = s[j]), or itsdestination (line 4, i = d[j]). The predicate investigates thisrule at each node and generates an equation that states theabove summations of flows entering in and exiting from thenode. The decision variable, LFM , is finalised after solvingthe resulting system of equations from all nodes in the solver.

2) Path Capacity Constraint Predicate: The main purposeof this predicate is to reduce the network graph whichmight speed up the computation or enforcement of anotherconstraint. Whenever there is a lower limit on the link

Algorithm 1 The Network Path predicate, to calculate networkpaths1 : forall(i in 1..Nnodes)(2 : forall(j in 1..Nflows)(3 : sum(k in 1..Nlinks where Links[k, 2] = i)(LFM [k, j])+4 : (if i = d[j] then 1 else 0 endif) =5 : sum(k in 1..Nlinks where Links[k, 1] = i)(LFM [k, j])+6 : (if i = s[j] then 1 else 0 endif)

)

)

capacities, this predicate can be used as an initial stage beforeapplying a next time consuming procedures. This predicateremoves all the links with capacities less than a specifiedvalue from the network graph. Hence, the resulting graph issmaller than initial network and applying a new constraintis faster. The MiniZinc code for this predicate is shown inAlgorithm 2 where the decision variable is the LFM . Itinvestigates the capacity of all links (Links[i, 3]) regardingthe lower bounds on capacities associated with all flows(Limits[j]). Using separate values for each flow is preferredas it make it possible to apply it on multiple concurrent flows.When a link i has a capacity less than the associated limit ofa flow j, it removes the link from the graph of that flow byassigning zero for the corresponding Link-Flow-Membershipentry (i.e. LFM [i, j] = 0 line 3).

3) Residual Capacity Predicate: This predicate computesthe residual network, i.e the network in which capacity ofthe links is reduced by the value of flows passing throughthem. In other words, it computes the residual capacity ofall links when all flows are placed (paths for all flows aredetermined). The residual capacity is a common concept inmany congestion-related routing algorithms. When there aremultiple concurrent flows, the order of placing flows changesthe residual network and the condition for placing the nextflows. In this case where capacity constraints (Eq. 4) should besatisfied by a flow, each next flow can only be placed in a linkif there is enough residual capacity on the link. This createsa dynamic variable which not only depends on the networkresources but it also depends on other concurrent demands.For a multiple-flow network the residual capacity of a link iscalculated as:

r(u, v) = c(u, v) f(u, v) (8)

in which c(u, v) indicates the (initial) capacity of the linkbetween nodes u and v, and f(u, v) indicates the total flowon this link as:

f(u, v) =

k=1..K

fk(u, v)xk(u, v) (9)

where fk(u, v) is the flow number k, assuming K concurrentflows. The xk(u, v) {0, 1} is the entry of Link-Flow-

Algorithm 2 The Path Capacity Constraint predicate, toremove links with capacities less than specified limits1 : forall(i in 1..Nlinks)(2 : forall(j in 1..Nflows)(3 : if Links[i, 3] < Limits[j] then LFM [i, j] = 04 : else true endif

)

)

Membership array in the kth column and the row correspond-ing to link (u, v) that determines if the flow number k ispassing through link (u, v) or not.

The MiniZinc code for this predicate is shown in Algorithm3. Although the LFM is the main decision variable in theapplications implemented in SCOR for routing purposes,Residuals, is the decision variable in this predicate. Thepredicate investigates all links (line 1). When there is no flowon a link (line 2), the residual capacity of the link is set toMAX (line 3), a big constant number. This is to prevent someissues that might happen in using the predicate in non-uniformnetwork topologies (i.e. links have different capacities). Anexample is when an algorithm uses the minimum residualcapacity of all links. Suppose there is a link with no flowon it and its capacity is less than the minimum residualcapacity of other links (links with flows). In this case, suchan algorithm chooses the smaller value, i.e. the capacity ofthe link with no flow, as the minimum residual capacity. Itmeans that the output of such an algorithm includes a linkwhich is not on any path. Then, using a value (bigger thanall capacities), for the residual capacity of such links solvesthe problem. For the links with flows, the residual capacity iscalculated through Eq. 8 and 9 in lines 4 and 5.

4) Link Capacity Constraint Predicate: This predicate en-forces the flow bound constraint as stated below

f(u, v) c(u, v) (u, v) A (10)

on each link per each flow. It causes the flows to be placedonly on links that can accommodate their demands, i.e.r(u, v) 0 (compare Eq. 8 and Eq. 10). While the PathCapacity Constraint predicates serves as an initial stage forapplying other constraints, this predicate is used in line withother constraints to guarantee the final output satisfies the flowdemands. The MiniZinc code for this predicate is shown inAlgorithm 4. The decision variables in this predicate includethe LFM and Residuals. The first line is an example ofusing a predicate in a MiniZinc code. In fact, this is a nestedpredicate which uses another predicate, residual capacity. Inthe second line it calculates Residuals, the residual capacityof the links and the next lines enforce the flow placements tosatisfy the positive residual capacity (Residuals[i] > 0) for

Algorithm 3 The Residual Capacity predicate, to calculate theresidual capacity of all links of a network graph1 : forall(i in 1..Nlinks)(2 : if sum(j in 1..Nflows)(LFM [i, j]) = 03: then Residuals[i] = MAX4 : else Residuals[i] =5 : Links[i, 3] sum(j in 1..Nflows) (Flows[j] LFM [i, j])6: endif

)

all the links. It is possible to modify this predicate in orderto implement routing with capacity overbooking.

5) Path Cost Predicate: This is the last predicate of the firstgroup of predicates that are used to model basic and compositeQoS routing problems. It calculates the total cost of a path Pfor the kth flow given by:

ak(P ) =

(u,v)Nodes

ak(u, v)xk(u, v) (11)

xk(u, v) {0, 1}

in which ak(u, v) is the cost of the link (u, v) for the kthflow. The cost is assumed a known predefined parametersuch as loss ratio, link or bandwidth price, etc., that can bedifferent per flow or be the same. The MiniZinc code for thispredicate is shown in Algorithm 5. The first decision variablein this predicate is LFM which is used to locate flows andthe next decision variable is Total Cost which implementsEq. 11 to compute the total cost of a flow path (lines 2, 3and 4).

6) Delay Predicates: The Delay, Congestion and LinkUtilisation belong to the second group of predicates. Whilethe first group uses exact measures for the implementedparameters, the second group uses approximations to find thevalue of these parameters. This is due mainly to lack of thenecessary information for the calculation of accurate values.In all predicates of this group, it is assumed that the networkflow behaviour can be modelled by a M/M/1 process. Thisis an assumption made and utilised in highly cited referencessuch as [40] and [41]. In queueing theory, the M/M/1 systemsconsists of a single queueing station with a single server,requests arrive according to a Poisson process, and the serverpasses customers in the exponentially-distributed independenttimes. Adopting M/M/1 system for modelling network flowbehaviours require to accept the Poisson distribution for thenetwork flows which might not be the case in some occasions.Though, other assumptions about the network flow statisticsthat may result in different equations can be implemented inSCOR in a similar manner.

The end-to-end delay of a flow path is the sum of thedelays experienced by the flow at each node from the sourceto destination. The delay at each node consists of a fixedand a variable component. The fixed component includesthe transmission and propagation delays, and the variablecomponent includes the processing and queueing delays ateach node [42]. The average variable delay is closely related

Algorithm 4 The Link Capacity Constraint predicate, toenforce link constraint1 : include Predicate residual capacity.mzn;2 : residual capacity(LFM,F lows, Links,Residuals)

3 : forall(i in 1..Nlinks)4 : (Residuals[i] > 0)

Algorithm 5 Path Cost predicate, to calculate the the totalcost of a path1 : forall(j in 1..Nflows)(2 : Total Cost[j] =

3 : sum(i in (1..Nlinks)4 : (Links[i, 4] LFM [i, j])

)

)

to the congestion and link utilisation that depend on theflow demand. The fixed part of the end-to-end delay can becalculated by the Path Cost predicate provided the link costis defined as the link delay.

The Delay predicate calculates the total average variabledelay of a network topology using the delay formula forM/M/1 systems [43]:

Delay =

(u,v)Nodes

1

c(u, v) f(u, v)(12)

in which c(u, v) is the link capacity and f(u, v) is the totalflow passing through the link. The MiniZinc code for thispredicate is shown in Algorithm 6. This is another exampleof a nested predicate in which the Link Capacity constraintpredicate is utilised (line 1). Creating abstractions is animportant aspect of the Northbound APIs. While the LinkCapacity constraint predicate is a nested predicate that usesthe Residual Capacity predicate, the Delay predicate createsanother level of abstraction by using it. Although the decisionvariables in this predicate include the Delay, LFM andResiduals arrays, the main variable is Delay and the valueof other variables are assumed known. The second lineenforces the link capacity constraint which prevents from linksaturation and calculates the residual capacity of the links.Then all links are investigated (line 3) to see if there is anyflow on the links (line 4). The value of queueing delay forthe flow-less links (links without any flow) is set to zero (line5) and for the links which include flows it is the inverse ofthe remaining capacity of the link (line 6).

7) Congestion Predicates: The Congestion predicate calcu-lates the the total network congestion through [41]:

Algorithm 6 The Delay predicate, to calculate the totalqueueing delay of a flow path1 : include Predicate link capacity.mzn;2 : link capacity(LFM,F lows, Links,Residuals)

3 : forall(i in 1..Nlinks)(4 : if sum(j in 1..Nflows)(LFM [i, j]) = 05: then Delay[i] = 06 : else Delay[i] = 1 / Residuals[i]7 : endif

)

Congestion =

(u,v)Nodes

f(u, v)

c(u, v) f(u, v)(13)

The MiniZinc code for this predicate is shown in Algorithm7. The implementation is quite similar to the Delay predicatewith a difference that uses Eq. 13 instead of Eq. 12. Thedecision variables in this predicate include Congestion,LFM and Residuals arrays and the Congestion is themain variable.

8) Link Utilisation Predicates: This predicate uses thebelow equation [44]:

Utilisation =

(u,v)Nodes

f(u, v)

c(u, v)(14)

to calculate link utilisation. The MiniZinc code for thispredicate is shown in Algorithm 8. It investigates all links(line 1) and calculates the link utilisation (line 2,3 and 4)based on the Eq. 14. The decision variables in this predicateinclude the Link Utilisation and LFM arrays, withLink Utilisation as the main variable and the LFM whichis seen as a known value.

VI. USE CASES: IMPLEMENTED APPLICATIONS

The main propose of SCOR is to facilitate QoS routingapplication development in SDN which is not currentlyaddressed in the available northbound interfaces. By usinga few examples, it is shown in this section that, how easily,simple and complex QoS routing and TE applications areimplemented in SCOR. Each QoS routing application isimplemented by including the Network Path predicate tomodel the network paths, and some other predicates to applytheir objective constraints on the paths. The first applicationis the least cost path routing. The next application is leastcost path routing with the capacity constraint which is simplyimplemented by adding a new predicate to the previousapplication. This is an example of the abstractions createdin the SCOR and its accommodation for developing QoSrouting applications. The last application is a complex TE

Algorithm 7 The Congestion predicate, to calculate the totalcongestion of a flow path1 : include Predicate link capacity.mzn;2 : link capacity(LFM,F lows, Links,Residuals)

3 : forall(i in 1..Nlinks)(4 : Congestion[i] =

5 : sum(j in 1..Nflows)6 : (Flows[j] LFM [i, j]) / Residuals[i]

)

Algorithm 8 The Link Utilisation predicate, to calculate thelink utilisation of a flow path1 : forall(i in 1..Nlinks)(2 : Link Utilisation[i] =

3 : sum(j in 1..Nflows)(4 : Flows[j]LFM [i, j])/ Link[i, 3]) 100

)

)

application which is simply modelled in SCOR. This clearlyillustrates SCOR potential for this purpose. The routingalgorithms modelled in these applications are explainedbriefly in this section and might require more details anddiscussion, the interested reader can refer to the referencesfor further information.

A. Least cost Path Routing

The first use case models the least cost path routing al-gorithm. It is single model that can model multiple routingalgorithms based on the definition of cost. If the link delayis defined as the cost it will give the minimum delay path(fixed part of delay) and if the distance (hop count) is usedas the cost metric, it will give the shortest path. The shortestpath routing is the most common network routing algorithmin legacy networks which is used in many routing protocolssuch as OSPF, RIP and IGRP.

The formal statement of the problem is presented inSection II (Eq. 6). Several algorithms such as Dijkstra andBellman-Ford have been proposed to efficiently solve thisproblem in procedural programming. In SCOR though, theimplementation only includes stating the problem, and nothow to solve it, as it can be seen in Algorithm 9. Thefirst two lines (line 2 and 3, lines starting with a % arecomments) are include items which make it possible to usetwo predicates, Network Path and Path Cost. Lines 5-11declare the parameters which are required to model a leastcost path problem (same as Table II). Lines 13-14 alsodeclare the two decision variables Cost and LFM similarto what explained in the SCOR predicates. The main bodyof the program includes the Constraints item (lines 16-17)and the Solve item (line 19). The first constraint, NetworkPath predicate, lists all possible paths from the source todestination and the second constraint, Path Cost predicate,calculates the cost associated with each path. The solve itemindicates the objective, i.e. what the solver should find that isthe path has the minimum cost.

B. Least Cost Path Routing with Capacity Constraint

In order to illustrate the usefulness of SCOR, another imple-mented application is demonstrated here. This application usesall code of the previous application plus a single line of codewhich implements a new constraint on network parameters.This is a clear example of combining various pre-implemented

predicates in SCOR to create numerous applications withmultiple constraints.

The path selected by the least cost path routing algorithmdoes not necessarily provide sufficient bandwidth for a flowdemand. If the selected paths capacity or bandwidth is lessthan the flow demand it can cause delay or even packet loss.When the service level is guaranteed for a flow demand itis necessary to prevent this by applying QoS mechanisms i.e.considering the capacity constraint when finding the path. Theformal statement of the problem is similar to least cost pathrouting, except there is an extra constraint, Eq. 10 the capacityconstraint, which should be satisfied.

As seen in Algorithm 10 it is implemented in SCOR byadding the three lines of code to the previous application. Thefirst added line (line 2) indicates a new predicate, CapacityConstraint, is used in this application and the second addedline (line 4) declares a new parameter, Limits (explainedin Table II). Finally, the third added line (line 6) appliesthe capacity constraint. This constraint removes the linkswith capacities less than Limits from the network graph.Consequently the output of the algorithm not only is the pathwith least cost from s to d but it can also accommodatethe flow demands to the specified Limits.

C. Maximum Residual Capacity Routing

The previous application illustrates the efficacy of abstrac-tions and modularity in SCOR. It not only shows illustratesthe easiness of creating new applications by adding a newpredicates or combining them, but it also demonstrate thecompliance of stating the problem in SCCOR and its statementin natural language. Here, implementation of a complex QoSrouting application in SCOR is compared to its implementation

Algorithm 9 Least Cost Path in SCOR1 : % Include item2 : include Predicate network path.mzn;3 : include Predicate path cost.mzn;

4 : % Parameters5 : array[int, int] of int : Links;6 : int : Nlinks = max(index set 1of2(Links));7 : array[int] of int : Nodes;8 : array[int] of int : Flows;9 : int : Nflows = max(index set(Flows));10 : array[1..Nflows] of int : s;11 : array[1..Nflows] of int : d;

12 : % Decision V ariables13 : array[1..Nflows] of var int : Cost;14 : array[1..Nlinks, 1..Nflows] of var 0..1 : LFM ;

15 : % Constraints item16 : constraint network path(LFM,Links,Nodes, s, d);17 : constraint path cost(LFM,Links, Cost);

18 : %Solve item

19 : solve minimize Cost[1];

in procedural programming. This demonstrates how efficientlySCOR simplifies the QoS routing and TE application devel-opment.

The Maximum Residual Capacity belongs to a group ofrouting algorithms that aim to route the maximum numberof concurrent flows in a given network. In these algorithms,usually the given flows must be routed between single ormultiple source-destination pairs and then, algorithms try toeither minimise the congestion or maximise the utilisation.In Maximum Residual Capacity the objective is to route allflows in a way that the minimum residual capacity of the linksis maximised [41]. This is a very complex network routingproblem which can be numerically intractable even for smallnetworks of a few nodes [45].

Algorithm 10 Least Cost Path with Capacity Constraint inSCOR (only lines not in Least Cost Path predicate)

.1 : % Include item

2 : include Predicate capacity constraint.mzn;..3 : % Parameters

4 : array[1..Nflows] of int : Limits;..5 : % Constraints item

6 : constraint capacity constraint(LFM,F lows, Links,. Limits);

.

Algorithm 11 Maximum Residual Capacity in SCOR1 : % Include item2 : include Predicate network path.mzn;3 : include Predicate residual capacity.mzn;

4 : % Parameters5 : array[int, int] of int : Links;6 : int : Nlinks = max(index set 1of2(Links));7 : array[int] of int : Nodes;8 : array[int] of int : Flows;9 : int : Nflows = max(index set(Flows));10 : array[1..Nflows] of int : s;11 : array[1..Nflows] of int : d;

12 : % Decision V ariables13 : array[1..Nlinks, 1..Nflows] of var 0..1 : LFM ;14 : array[1..Nlinks] of var int : Residuals;

15 : % Constraints item16 : constraint network path(LFM,Links,Nodes, s, d);17 : constraint residual capacity(LFM,F lows, Links,

Residuals);18 : constraint forall(i in 1..Nlinks)(Residuals[i] >= 0);

19 : %Solve item

20 : solve maximize min(Residuals);

For a given graph G(N ,A) and a set of flows f , the problemcan be mathematically stated as [45]:

maximize Z

subject to

Z r(u, v) (u, v) A (15)

r(u, v) = c(u, v) f(u, v) (16)

{v|(u,v)A}

f(u, v)

{v|(u,v)A}

f(v, u) =

1 if u = s,

1 if u = t,0 otherwise

(17)

f(u, v) =

k=1..K

fk(u, v)xk(u, v) (18)

where Eq. 16, 17 and 18 are same as Eq. 8, 5 and 9 respec-tively. There is an implementation of the maximum concurrentflow routing in [46] that can be adopted to solve this problem.It is implemented in 400 lines of code in Python languagebased on the solution proposed in [47]. The implementationof the Maximum Residual Capacity algorithm in SCOR isshown in Algorithm 11. Lines 2-3 are include item for theNetwork Path and Residual Capacity predicates. Lines 5-11declare parameters and lines 13-14 declare the two decisionvariables, Residuals and LFM .

The main body of the program consists of the three con-straints (lines 16-18) and the solve item (line 20). The Net-work Path predicate lists paths for all the source-destinationpairs, the Residual Capacity predicate calculates the residualcapacity in each path, and the last constraint enforces the flowplacements to satisfy (Residuals[i] 0). The solve itemindicates that the solver should select a path for each flowthat maximises the minimum residual capacity of the wholenetwork.

VII. EVALUATION AND RESULTS

The previous section illustrates the usefulness of SCOR bydemonstrating the efficacy of its abstractions and explainingthe simplicity of developing complex applications in SCOR.In order to evaluate these applications, two other aspects mustbe also investigated, the completeness and the scalability.The first question is already answered in Section IV wherethe design of predicates is discussed. It is shown there thatthe set of eight implemented predicates in SCOR can modelmost of the basic and composite QoS routing algorithmsthat deal with capacity, delay, jitter etc. Furthermore, thelast three predicates make it possible to model advancedTE problems, such maximum concurrent flow problem.In order to answer the second question, it is needed toinvestigate SOCR in terms of applications implemented

Grid Order Nodes Links

2 4 4

3 9 12

4 16 24

5 25 40

6 36 60

7 49 84

8 64 112

9 81 144

10 100 180

11 121 220

12 144 264

13 169 312

14 196 364

15 225 420

16 256 480

17 289 544

18 324 612

19 361 684

S

d

(A) (B)

Fig. 6. A) Network sizes used for the grid topologies. B) A sample gridtopology of order 6

in it, i.e verify if they operate within a reasonable timeframe. In fact, the suitability of SCOR as a Northbound isvalidated through evaluating the run time of these applications.

A. Evaluation method

Numerical experiments are necessary in order to evaluatethese applications in real networks dimension. There areseveral parameters that might affect the run time of theseapplications on a real network. The three parameters, networktopology, network size and the number of concurrent flowsare considered in designing these experiments. Though, thelist of effective parameters may not be limited to this and onecan investigate other parameters, e.g. the effect of varioussolvers, which is an ongoing research topic in CP.

1) Network Topology: Network topology is the trivial effec-tive parameter in the evaluation of routing algorithms. Topolo-gies used in this study include grid and fat-tree topologies.The grid or mesh topology is common in computer networkssuch as sensor networks. Since it provides a variety of pathsbetween source-destination pairs, which might be challengingfor finding the appropriate path, it is selected for the run-time evaluation of the implemented applications. A samplegrid topology is shown in Fig. 6-B. In this topology each nodeis connected to all its neighbours through four links in the caseof internal nodes and two or three links for the border nodes.In all experiments in this section, the flow sources and sinks(destinations) are placed at the ends of the grids diagonal.

The fat-tree topology used in this evaluation is suggestedby [48] for interconnecting the SDN switches in data centreswhich is a special instance of the fat-tree topology. In themain fat-tree topology each layer is connected to the higherlayer through a single link with higher capacity (rather thanmultiple equal-capacity links). This necessitates to use highcapacity switching devices for the core and aggregation layers.Though, Since all the nodes are identical in the adopted fat-tree

topology, it is possible to use the cheap commodity switchesin the core and aggregation layers. A sample adopted fat treetopology is depicted in Fig. 7-A. The main idea is to make atopology of similar switches with equal number of ports (k).The fat tree order k includes k pods, containing k switchesin two layers, aggregation and edge layers. Below the podsis the place where the hosts are located and each k/2 ofthem are directly connected to one k-port edge switch. Theremaining k/2 ports of the edge switches are connected to theaggregation switches which in turn their remaining k/2 portsare connected to the core switches. Each of the (k/2)2 coreswitches is connected to all k pods.

2) Network Size: Basically, size of the network determinesits situation in real world applications. For instance, a realworld carrier network might include hundreds of nodes whilea campus network might only include tens of nodes. Indeed,size of the network is a significant scalability factor that canbe used as a parameter in the run-time evaluation of therouting algorithms. The network sizes used in this study forthe grid and fat tree topologies are shown in Fig. 6-A and7-B respectively .

3) Number of Concurrent Flows: If flow specifications areconsidered for the path calculations in a routing algorithm,the number of concurrent flows is another essential parameterin the run-time evaluation. One significant aspect of routingmultiple flows, instead of a single flow, is the effect ofselecting order. In routing a single flow, only constraintsrelating to the same flow, e.g. the bandwidth to accommodatethe flow demand, is considered. In routing multiple concurrentflows though, not only a flows own constraints is considered,but the various cases for the residual network (Section V), thatmight happen corresponding to each flow, must be consideredtoo.

Whenever there are multiple flows, the order of placingflows will result in a different residual network. Owingto the fact that there might be various paths for routing a

S d

Pod 0 Pod 1 Pod 2 Pod 3

Edge

Aggregation

Core

Fat Tree Order Nodes Links2 7 64 36 486 99 1628 208 384

10 375 750

(A)

(B)

Fig. 7. A) The adopted fat tree topology for SDN switches of order k=4 [48]B) Network sizes used for the fat tree topologies.

single flow in a network, there might be multiple differentresidual networks after routing the first flow. If another flowwas routed first, other set of residual networks would havebeen generated for routing the next flows. Consequently,the number of possibilities for choosing path and so thecomplexity of the problem is increasing exponentially thatsignificantly affects the run time of routing application.

B. Results

The set up used in these experiments includes a virtualmachine with 4 virtual CPUs, each with 100% executioncapacity, and 11 GB memory. The virtual machine runs ona host computer with a 64-bit quad core CPU at a clockspeed of 3.6 GHz and 16 GB of memory. The host machineruns a 64-bit Windows 7 Enterprise operating system andthe guest machine is a Linux with the kernel version of3.13.0-24-generic. The virtualisation platform is the OracleVirtualBox hypervisor version 5.0.2 with an extension packversion 5.0.2 106931. The SCOR is running on version 2.0.2of G12 MiniZinc package and the implemented applicationsare evaluated through an interface developed in Python 2.7.The two solvers used in these experiments include Gecodeversion 4.4.0 and mzn-g12mip (mixed integer programming)which come with MiniZinc version 2.0.2.

The three implemented applications are exactly evaluatedaccording to the above mentioned method. For the effectof topology and size, all the three applications, the LeastCost Path, the Least cost Path with Capacity Constraint andMaximum Residual Capacity routing applications are used inthe experiments. Though, for the effect of various numberof concurrent flows, only the Maximum Residual Capacityrouting application is used.

There are network emulation environments such as Mininet[49] that are commonly used in SDN to implement variousexperiments in a single PC. According to [49], Mininet can beused to implement large network topologies. However, whenthere are resource-consuming applications, the practical limitis much smaller due to limitations in system resources such asRAM and CPU. In this case, the resources might not suffice

0

50

100

150

200

250

300

350

400

450

0 5 0 1 0 0 1 5 0 2 0 0 2 5 0 3 0 0 3 5 0 4 0 0

SO

LV

E T

IME

(M

S)

NUMBER OF NODES

SOLVE TIME

Least Cost Path with Capacity Constraints

Least Cost Path

Fig. 8. The solution time for the Least Cost Path and Least Cost Path routingwith Capacity Constraint routing applications in a grid topology of differentnetwork sizes (number of nodes)

for the applications and the run-time evaluation might beinaccurate. A possible solution is to evaluate these applicationswithout the routing framework, by sending the network graphsand flow statistic through a separately developed interface inpython. The graphs for the grid and fat-tree topologies ofvarious sizes are created and their adjacency lists are sentto the applications. The graphs in all experiments includelinks with a unique capacity of 1000 Mbps in both networktopologies. The flow demands are also set equal to 10 Mbpsin all experiments. For the grid topologies the source and sinknodes are assumed at the end of grids diagonal (similar toFig. 6-B) for all flows. In the fat-tree topologies the sourceand sink nodes are the first and last hosts (similar to Fig. 7-A)for all flows.

Fig. 8 displays the solve time of the Least Cost Path andLeast Cost Path with Constraint Capacity routing applicationsfor grid networks of various sizes. The horizontal axis indi-cates the number of nodes which represents the network sizethe vertical axis is the solve time in millisecond. The solvetime comprises, in fact, three parts modelling, i.e. convertingthe model to a set of finite constraints, solving, and mappingthe solution to the original problem. While the term solvetime in references about CP is mostly used for the secondpart of these three parts, in this work it is used for the runtime. i.e the total time spent to call the routing applicationfrom the Python interface until receiving the results. As seen,the run-time / solve-time values for a grid network of the size350 nodes or more is about 400 ms. This is in the range of theshortest path algorithm using Dijkstra implemented in python.

Fig. 9 displays the solve time of these applications innetworks of fat-tree topology of different sizes. The horizontaland vertical axes are similar to Fig. 8. The figure indicates thatfor a fat-tree network with 375 nodes and 750 links (fat-treeorder 10, i.e. a network of 10 port switches), the time requiredto calculate the Least Cost Path route is about 400 ms. Thisis a practical value for the route calculation in such a bignetwork. The route calculation for the Least Cost Path withCapacity Constraint is less than 500 ms which is very closeto the solve time of the Least Cost Path.

Comparing the results of the two applications in gridand fat tree topologies of different sizes indicates that theirperformance is similar and in some cases the Least Cost Pathwith Capacity Constraint is a few millisecond faster. Sincethe unique capacities of all links are assigned big enough toaccommodate all flows, the capacity constraint does not resultin a shortened network.

Nonetheless the graph of the solve time for both topologiestend to be exponential, the results are still practical for the net-work sizes indicated in the figures. The illustrated results aregenerated by applying the routing application on homogeneousnetworks which is the worse case in terms of route calculation.In a more realistic scenario, networks are not homogeneousand the capacity of the links are most probably different. Inthis case, the the capacity constraint probably might removesome links from the network which results in smaller networktopology i.e. a smaller search scope for the least cost and

0

100

200

300

400

500

600

0 5 0 1 0 0 1 5 0 2 0 0 2 5 0 3 0 0 3 5 0 4 0 0

SO

LV

E T

IME

(M

S)

NUMBER OF NODES

CHART TITLE

Least Cost Path with Capacity Constraint

Least Cost Path

Fig. 9. The solution time for the Least Cost Path and Least Cost Path routingwith Capacity Constraint routing applications in a fat tree topology of differentnetwork sizes (number of nodes)

0

100

200

300

400

500

600

700

800

0 50 100 150 200 250 300 350 400

So

lve

Tim

e (

ms)

Number of Nodes

Chart Title

Grid Topology

Fat Tree Topology

Fig. 10. The solution time for the Maximum Residual Capacity routingapplication in the Grid and fat tree topology of different network sizes (numberof nodes)

shorter solve time. Although using heterogeneous networksmight improve the run time, it might bind the run time to theselected capacity values.

The results of Fig. 8 and 9 are generated using the mzn-g12mip solver. The other solver which comes with Minizincin the package, Gecode or in its command line format mzn-gecode, performs much (thousands of milliseconds) slowerthan mzn-g12mip solver for the above two applications.Though, this is not a general rule for all applications and it ismuch faster for the next application, the Maximum ResidualCapacity routing. This is a clear example of the fact that thereis no universally optimised solver for all CP problems, a factindicated in many references of CP studies [50].

Fig. 10 displays the solve time of the Maximum ResidualCapacity routing application on the grid and fat-tree networktopologies of various number of nodes using mzn-gecodesolver. Again, the horizontal axis indicates the size of thenetwork in terms of number of nodes and the vertical axis isthe solve time in milliseconds. It needs about 700 millisecondsto calculate the Maximum Residual Capacity route for a singleflow in a network of 375 nodes with the fat tree topology.A similar time (less than 700 milliseconds) is required forthe route calculation in a grid network of 361 nodes. Forsuch a complex QoS routing application, which efficiently isimplemented in a few lines of code (see Algorithm 11), the

solve times are quite practical in the identified network sizes.The effect of number of concurrent flows on the run time

is illustrated in Fig. 11 for the Maximum Residual Capacityrouting application. It shows the solve time for various numberof concurrent flows in a fat tree topology of various number ofnodes. The horizontal axis indicates the network size (numberof nodes) and the vertical axis indicates the solve time inmilliseconds. As seen, for the small network sizes such as 7and 36 nodes, adding new flows does not affect the solve timeseverely. For instance, for 10 concurrent flows the solve timeis about 3 times of a single flow. But for the larger networksizes the ratio of the solve time grows faster. For example, fora network size of 375 nodes the solve time of 10 concurrentflows is more than 13 times of a single flow.

Nonetheless, the solve time for 10 concurrent flows is stillquite practical. It is 622 ms for a network with 99 nodes and9750 ms for a network with 375 nodes. In [30], a similar TEobjective is implemented in linear programming and run on apowerful server which has taken more than few hours for anetwork of 100 nodes. The SCORs success is a result of thefact that it uses CP and its intelligent search techniques tosolve the problem and find a path among the huge number ofpossible paths in such network sizes. These results indicatethat SCOR not only simplifies developing QoS routing andTE applications in SDN, but it also provides scalable andfully practical applications.

VIII. CONCLUSION

The main contribution of this work is to introduce a newNorthbound API for SDN, based on constraint programmingtechniques. The fundamental goal in the design of this North-bound API is to simplify developing QoS routing and TEapplications in SDN. The proposed solution, SCOR, providesbuilding blocks, the predicates, that can be re-used, not onlyas nested predicates, but also for multiple objectives bycombining with other blocks. Applications created in SCORare quite scalable and practical in large network sizes and cancompute excellent solutions to complex TE problems in fewsecond.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

7 36 99 208 375

SO

LVE

TIM

EE

(M

S)

NUMBER OF NODES

M A X I M U M R ES I D U A L C A P A C I T Y - F A T T R EE T O P L O G Y - G EC O D E S O L V ER

1-flow

2-flows

3-flows

4-flows

5-flows

6-flows

7-flows

8-flows

9-flows

10-flows

Fig. 11. The solution time for the Maximum Residual Capacity routingapplication per various number of concurrent flows in the fat tree topologyof different network sizes

SCOR is implemented in MiniZinc CP modelling languagewith an extra level of abstraction which forms an interfacefor QoS routing and TE. The main advantage of using CPtechniques in this Northbound interface is to unlock QoSrouting application development from complex mathematical/ heuristic solution algorithm implementations. In this way,an efficient tool is created that allows to painlessly implementsimple and complex QoS routing and TE applications in a fewlines of code.

As another contribution, a routing framework is introducedthat uses SCOR for its Northbound API. It includes a NetworkOperating System (NOS) layer that collects network states,such as topologies, capacities, traffic loads and flow demandsfrom the network elements and provides it for upper layersthrough a network state database. The application layer ofthis routing framework includes various routing applicationscreated through SCOR. They use the abstractions and networkstates supplied by the NOS to efficiently implement assortedQoS routing and TE applications.

The scalability of the developed applications is evaluatedthrough a series of experiments. The results indicate for asingle flow routing problem, either simple or complex, thesolve time is quite practical even for real world network sizes.For the case of multiple concurrent flows and complex TEproblems such as maximum residual capacity routing, pro-cedural programming techniques such as linear programmingmight be quite impractical. Though, SCOR provides fullyscalable and practical applications that are faster by threeorders of magnitude.

Nonetheless, using CP has provided even more options toimprove the scalability. Most of the solvers make it possiblefor the user to limit the solve time. In this case, the solverwill give the best solution that is found in the designated timeframe. The results of experiments indicate that the solutionsachieved even in one tenth of a full solve-time are very similarto the final solution for practical purposes. This feature canbe utilised in the cases