Tiramisu: Fast and General Network VerificationAnubhavnidhi Abhashkumar∗, Aaron Gember-Jacobson†, Aditya Akella∗

University of Wisconsin - Madison∗, Colgate University†

Abstract: Today’s distributed network control planessupport multiple routing protocols, filtering mechanisms,and route selection policies. These protocols operateat different layers, e.g. BGP operates at the EGP layer,OSPF at the IGP layer, and VLANs at layer 2. The be-havior of a network’s control plane depends on how theseprotocols interact with each other. This makes networkconfigurations highly complex and error-prone. State-of-the-art control plane verifiers are either too slow, or donot model certain features of the network. In this paper,we propose a new multilayer hedge graph abstraction,Tiramisu, that supports fast verification of the controlplane. Tiramisu uses a combination of graph traversalalgorithms and ILPs (Integer Linear Programs) to checkdifferent network policies. We use Tiramisu to verifypolicies of various real-world and synthetic configura-tions. Our experiments show that Tiramisu can verifyany policy in < 0.08 s in small networks (∼35 devices)and < 0.12 s in large networks (∼160 devices), and itis 10-600X faster than state-of-the-art without losinggenerality.

1 Introduction and BackgroundMany networks, including university, enterprise, ISP anddata center networks, employ complex distributed controlplanes running atop rich underlying network structures.The control planes run a variety of routing protocols, suchas RIP, OSPF, and BGP, which are configured in intricateways to exchange routing information both within andacross protocols. In some cases, the protocols are assistedby other protocols (e.g., iBGP assisting in distributingBGP information throughout a network). Networks alsoemploy techniques to virtualize the control plane, suchas virtual routing and forwarding (VRF), as well as tech-niques to virtualize multiple network links into differentbroadcast domains (VLANs).

Bugs can easily creep into such networks, e.g., througherrors in the detailed configurations that many of theprotocols need. Often, bugs may not be immediatelyapparent, and the network may be running “just fine” un-til a failure causes a latent bug to be triggered. Whenbugs do manifest, a variety of catastrophic outcomes canarise: the network may suffer from blackholes; serviceswith restricted access may be rendered wide open; crit-ical applications can no longer be isolated from otherservices/applications, and so on.

Verification tools and their trade-offs. A variety of

tools attempt to verify if networks are violating, or couldpotentially violate, policies of the above kind. Data planeverifiers [13, 10, 11] analyze the current forwarding ta-bles and check for blackholes, loops, or broken path iso-lation. Unfortunately, these tools don’t have the means toanalyze if the network’s new data plane that materializedupon reacting to a failure, can satisfy relevant policies ornot.

To overcome this issue, a variety of control plane ana-lyzers were developed [5, 2, 16, 4, 6]. These proactivelyanalyze the network against various environments, e.g.,failures or external advertisements. While a significantstep forward in network verification, control plane toolstoday make trade-offs between performance and general-ity.

On the one hand are “graph-based” tools such asARC [6]. ARC encodes all paths that may manifest in anetwork under various failures into a series of weighteddigraphs. This abstraction enables analyzing the networkunder many potential environments at once by runningvery fast polynomial time graph algorithms; e.g., check-ing if two hosts are always blocked amounts to check-ing if they are in different graph connected components.Unfortunately, ARC ignores many network design con-structs, including modeling the intricacies of BGP andiBGP, and the existence of VLANs, and VRFs.

On the other hand are “SMT-based” tools such as Bag-pipe [16] and Minesweeper [2]. These tools create adetailed model of the control plane by symbolically en-coding routing information exchange, route selectionlogic, and the environment (e.g., failures) using logicalconstraints/variables. By asking for a satisfying assign-ment for a SMT formula that encodes the network andthe property of interest, they can identify a concrete en-vironment that leads to property violation. These toolsoffer much better coverage of control plane protocolsthan graph-based ones, but their verification performanceis very poor, especially when exploring failures (§6),despite many internal SMT-specific optimizations.

Decoupling encoding and properties. We ask if itis possible to design a verification tool that marries thespeed of graphs with the generality of an SMT-basedencoding.

We start by noting that today the trade-off between per-formance and generality in tools, arises from a couplingbetween the control plane encoding used in the tools andhow properties are verified. In graph-based tools, the

arX

iv:1

906.

0204

3v1

[cs

.NI]

5 J

un 2

019

weighted graph control plane model requires graph al-gorithms to verify properties. In SMT-based tools, thedetailed constraint-based control plane encoding requiresa general constraint solver to be used for all properties.

Our framework, Tiramisu, decouples the encodingfrom the property: it uses a sufficiently rich encoding forthe network that models various control plane featuresand network design constructs. But then, it permits theuse of custom algorithms that offer the best performanceto verify a property of interest.

Richer graphs. Our framework starts with graphs asthe basis for modeling networks, because graph-basedcontrol plane analysis has been shown to be fast [6].We then embellish the graph model with richer graphconstructs, such as hierarchical layering, the notion ofhedges [7] (edges that fail together), and rich, multi-attribute edge and node labels. Our resulting graph canmodel all aspects we observe used in real world networksand configurations.

Given the graph structure, we then develop a suite ofcustom techniques that help verify various properties ofinterest.

Avoiding path enumeration. First, we note that someproperties of interest do not require the computation ofthe actual path that the network would induce on a certainfailure; they care mostly about whether paths exist or not.An example is whether two hosts are always blockedfrom each other. For such properties, we develop twotechniques that avoid path enumeration altogether: amodified depth first search graph traversal algorithm, anda simple integer linear program (ILP) formulation thatcomputes graph cuts. Importantly, graph traversal runsin polynomial time. And, for a given property, the sim-plified property-specific ILP can be solved much fasterthan a general SMT-encoding in SMT-based tools. Thisis because the former explores symbolically only thevariables that are relevant to the property being veri-fied, whereas the SMT solver for the general encodingsearches through a much larger search space.

Domain-specific path computation. For the remain-ing properties that require computation of paths, we runa modified path vector protocol atop our richer graph ab-straction. Here, we leverage foundational work by Griffinet. al which showed that various routing protocols can bemodeled as instances of the stable paths problem [8] (thisinsight was used in Minesweeper), and that a “simplepath vector protocol” (SPVP) [8] emulates the compu-tation of a solution to this problem. In our version ofSPVP, each node consumes the multi-dimensional at-tributes of incoming edges and neighboring vertices, anduses simple arithmetic operations that encode existing

protocols’ logic to select among multiple paths availableat the moment. Importantly, in simple networks (e.g.,those that use a single routing protocol) our protocolnaturally devolves to being similar to distance vectorprotocol, which runs in polynomial time (being basedon the Bellman-Ford algorithm). For more general net-works, we empirically find that our protocol can quicklycompute the paths that are relevant to verifying specificproperties. The performance is faster than SMT-basedtools, because our approach essentially uses a highlydomain-specific approach to finding paths, compared toa SMT-based general search strategy; for instance, evenfor simple networks, a SMT-based strategy would invokethe solver to find a satisfying assignment.

Prototype and evaluation. We implementedTiramisu in Java (7K LOC) and evaluated it with manyreal data center and university networks, as well asnetworks from the topology zoo [12]. We find thatTiramisu’s rich multi-layered graphs can be computedfrom configurations in a few µs per traffic class. UsingTiramisu’s custom algorithms, various properties can bechecked for complex networks in 3-80ms per traffic class.Compared to Minesweeper that uses a general encod-ing with an SMT solver, Tiramisu offers speed of up to600X for reachability policy verification and 10-50X forbounded-length and path preference policies (both underfailures). Tiramisu’s algorithmic approach renders it sub-stantially faster even when verifying properties under nofailures. Finally, Tiramisu scales well, providing verifi-cation results in ∼100ms per traffic class for networkswith ∼160 routers.

2 MotivationGiven the significant performance benefits of graph ab-stractions [6], we use graphs as the basis to encode con-trol plane computation in Tiramisu. However, the ab-straction we use is significantly different from ARC [6].In what follows, we provide an overview of ARC’s graphbased approach for verification. We then identify its keydrawbacks which motivate Tiramisu’s graph design.

2.1 ARC

ARC (Abstract representation for control plane [6]) isa state-of-the-art control plane verifier. It models a net-work’s control plane using a collection of directed graphs.There is one directed graph per traffic class which modelsthe forwarding behavior of packets belonging to that traf-fic class. In ARC, nodes represent routing processes, anddirected edges represent possible flow of traffic enabledby exchange of route advertisements between routingprocesses. Using a single attribute edge-weight, ARCcan model OSPF costs and AS path length. Finally, ARC

2

Policy class Graph characteristicsP1: src and dst are alwaysblock

src and dst are in separate compo-nents

P2: All paths from src to dsttraverses a waypoint

after removing the waypoint, srcand dst are in separate components

P3: src can reach dst whenthere are < K link failures

min-cut of src−dst graph is >= K

Table 1: Policies as graph characteristics

B

E

CA

B

D

CA

OSPF OSPF

OSPF

OSPF

eBGP iBGP

iBGP

eBGP

eBGP

BGP adjacencies OSPF adjacencies

1

1 5

1

1

B

D

CA

E

Physical adjacencies

eBGP

eBGP

iBGP

Figure 1: Multilayer dependency

verifies policies by checking some simple graph charac-teristics. Table 1 lists some of these policies and relatedgraph properties. By leveraging graphs, ARC offersorder-of-magnitude better performance [6] compared tostate-of-the-art [5, 2].

However, ARC’s drawback is that it is limited in itsnetwork design coverage. It does not model layer 3protocols such as iBGP and VRF. It does not model anylayer 2 protocols including VLANs. It also does notmodel BGP protocol attributes such as local preference,communities etc. We use examples to show how theselimitations can affect ARC’s correctness, and what wedo in Tiramisu to overcome them.

Cross layer dependency. Consider the networkin Figure 1. The network consists of 5 routers (A−E).Here, routers A and B are eBGP peers of router E, and,router C is an iBGP peer of router B. All routers except Ebelong to the same OSPF process, with each link exceptC−D having a link cost of 1. Link C−D has cost 5.In this scenario, C learns a route to E through its iBGPneighbor B, with B as its next hop. C forwards all trafficdestined to E, to B. iBGP peers forward traffic at the IGPlayer. Hence C uses its OSPF process to find the path toreach B. The path computed varies for different failurescenarios, which affects reachability of traffic. Depend-ing on the path used under a given failure, the operator’spolicy for traffic between C and E may be violated:i) Under no failures, OSPF prefers path C→ B (OSPFcost 1), and then traffic flows from B→ E;ii) When the link B→C fails, OSPF prefers a differentpath C→ A→D→ B (cost 3). Crucially, traffic at A getsdirectly rerouted to E, because A is the eBGP peer of E;

A eBGP S1 CeBGP

D eBGP

E eBGP

B

eBGP

OSPF

vlan2 vlan1

block

comm

c1

remove

comm

c1

add comm

c1

ospf = 1

lp = 100

lp =

50

OSPF

T

S

U

Figure 2: Multiple network features

iii) When links A→ C and B→ C fail, OSPF preferspath C→D→ B (cost 6). Here traffic gets dropped at D,because D did not learn a route for E, and C is discon-nected from A. Note that, had router A or B redistributedit’s BGP routes to OSPF , then D would have learned aroute to E, avoiding the blackhole situation (iii) above.

Such dependencies between iBGP and OSPF cannotbe modeled in ARC. ARC cannot analyze these scenariosand compute the actual paths used, and because of it, itcannot be used to verify any policies in this network.

ARC’s lack of modeling of cross-layer dependenciesimpacts its applicability in other network scenarios too.Consider the network in Figure 2. This network has 5routers (A−E), a switch S1, and three hosts S−U . Allrouters run BGP, and routers B and E also run OSPF(with cost 1). Switch S1 connects to router C and A onVLAN 1 and 2, respectively. D adds community “c1”to its advertisements, A removes “c1”, and E blocks alladvertisements with community “c1”. Finally, to preferroutes learned from B over A, E assigns local preferencevalues 100 and 50 to B and A, respectively.

Although routers A and C are connected to the sameswitch S1, they belong to different VLANs. Hence, traf-fic cannot flow through switch S1. By default, ARCassumes layer 2 connectivity. Hence according to ARCrouters A and C are reachable and traffic can flow be-tween them.

The overall theme is that protocols “depend” on eachother. E.g. iBGP depends on OSPF, BGP and OSPFdepend on VLANs etc. These protocols also operate atdifferent “layers”. BGP operates at the EGP (ExteriorGateway Protocol) layer, OSPF at the IGP (Interior Gate-way protocol) layer, and VLANs at Layer 2. A graphabstraction needs to encode layers and cross-layer de-pendencies. Thus, Tiramisu introduces a new multilayergraph abstraction, where traffic flow in the higher lay-ers may depend on the traffic flow in the lower layers.Figure 3 and 4 show the multilayer graphs of the afore-mentioned networks, which we explain in detail in §3

Impact of BGP attributes. In Figure 2, the pathtaken from E to C depends on communities and lo-cal preference. There are two paths from E to C: i)E → B→ D→ C, and, ii) E → A→ D→ C. Becauseof local preference, path (i) is preferred over path (ii).However, E blocks all advertisements with community“c1”. Since router D adds this community, all advertise-ments through D will have community “c1”. Hence theadvertisement for path (i) is blocked. Although the ad-vertisement for path (ii) also comes through D, router Aremoves community “c1” and router E does not see thatcommunity. Therefore, there is only one path between

3

Abgp

EGP layer

IGP layer

ebgp

ospf

ibgp

fibphysical

Ebgp Bbgp Cbgp

Efib Bfib CfibAfib Dfib

Bospf CospfAospf Dospf

Figure 3: Correlated Edges

Dbgp

Cbgp

Bbgp

Bospf

Ebgp

Eospf

Abgp

S1v1S1v2

T

SEGP layer

IGP layer

Layer 2

<ad=20,

as = 1, lp = 50>

add c1

remove

c1

block c1<ad = 20,

as = 1>

<ad= 110, ospf= 1>

Efib Bfib Afib Dfib Cfib

Note: In this example, edge metrics

are same in both direction

Figure 4: Multilayer Hedge Graph for traffic class S−T

E and C: E → A→ D→ C. ARC, on the other hand,characterizes path (i) as valid.

Additionally, ARC cannot model other BGP attributeslike local preference, MED etc. To support all BGP at-tributes as well as metrics of other protocols (OSPF cost,AD), Tiramisu uses multiple edge and node attributes.We elaborate in §3.

Physical link dependency. Consider the networkin Figure 2 without link A− E. Here, according toARC, traffic from S to U can flow through 2 pathsS→ Eosp f → Bosp f → U and S→ Ebgp → Bbgp → U .To evaluate reachability under failure, ARC calculatesmin-cut of this graph as 2, and concludes that it can with-stand arbitrary 1 link failures. However, this is incorrectbecause edgeEosp f→Bosp f and edgeEbgp→Bbgp belong to thesame underlying physical link E−B whose failure causesdisconnection.

In graphs, the notion of hedges [7] can be used tomodel such dependency among edges. A hedge is a setof edges that fail together. A graph can have multiplehedges, each with more than one edge in it, and a singleedge can belong to multiple hedges. In Tiramisu, tosupport edges that fail together, we label sets of “related”edges as hedges. Overall, Tiramisu converts the controlplane into a series of multilayer multi-attribute hedgegraphs one per source-destination pair.

3 Tiramisu Graph AbstractionWe discuss the components of the Tiramisu graph ab-straction.

Nodes. In Tiramisu, nodes are created for bothswitches and routers. For switches, nodes are created perdevice, per VLAN. E.g, in Figure 4, S2 has two nodesS2v1 and S2v2 for VLANs 1 and 2. For routers, nodesare created per device, per routing process. E.g, in Fig-

ure 4, router B has two nodes Bbgp and Bosp f for it’s BGPand OSPF routing processes. If the router supports VRF(Virtual Routing and Forwarding), then the nodes arereplicated per device, per virtual routing process (sim-ilar to VLAN). Note that by default, each routing pro-cess has a default vrf, and nodes like Bbgp represent thedefault vrf of B’s BGP process. We identify routingprocesses, VLANs, and VRFs from device configura-tions. We also create, per router, a node representing therouter’s forwarding information base (FIB). Finally, wecreate two special nodes representing the src and dst ofa traffic class.

Edges. Tiramisu categorizes it’s edges into multiplelabels, depending on the edge’s end nodes. These labelsinclude: FIB (f), eBGP (b), IGP (o), static (s), redistribute(r), iBGP (i), and physical (p). Some of these labels arefor inter-device edges and some for intra-device edges.We show examples of these labels in Figure 4, and ex-plain them below.

Inter device edges. An inter-device edge exists be-tween different devices that have a physical link connect-ing them. There are three cases: (i) if both end nodes areswitches and belong to the same VLAN, connect themwith an edge of label “p”, (ii) if one node is a switchand other a router, connect them with edge of label “p”,e.g. edgeS1v1→Abgp . (iii) if both nodes are routers, theybelong to the same routing process (IGP or eBGP) andthe same VRF, then connect them with edge label “o” or“b”, respectively, e.g. edgeDbgp→Abgp .

Intra device edges. An intra-device edge exists be-tween nodes that belong to the same device. There arethree cases: (i) if the router redistributes routes fromprocess X to Y , add edge of label “r” from Y to X , e.g.edgeBbgp→Bosp f , (ii) if the router is an iBGP peer, addedge of label “i” from the router’s BGP process to theunderlying IGP process, e.g. edgeBbgp→Bosp f of Figure 3,or (iii) every node corresponding to a routing process ona given router is connected (both ways) to the router’sFIB node; e.g., Bosp f is connected to B f ib (Figure 3).

Each traffic class has a srcRouter and dstRouter thatoriginate the src and dst IP addresses. Tiramisu connectsthe src node to all routing processes of srcRouter. Fi-nally, it connects all routing processes of dstRouter withdst node.

Edge costs. Tiramisu supports multiple routing pro-tocols. Each protocol uses a different set of metrics toexpress link and path costs/preferences. E.g. OSPF useslink costs, and, BGP uses AS-path length, local prefer-ence (lp), Multi-Exit Descriptor (MED) etc. Addition-ally, an administrative distance (AD), allows operatorsto choose routes from different protocols. Hence a sin-

4

gle edge weight cannot model all protocols. Tiramisureplaces these edge weights with a vector of metrics. Thevalues of these metrics are inferred from device config-urations. Depending on the edge label, certain metricswill be set as null, e.g. OSPF cost is null for “b” edges.

Hedges. All edges that represent the same physi-cal link belong to the same hedge. E.g. in Figure 4,edgeBbgp→Ebgp and edgeBosp f→Eosp f belong to the samephysical link B→ E, and thus the same hedge.

Traffic class specific constructs.: Some aspects ofthe network are specific to traffic classes, e.g. ACLs,filters, static routes, etc. Thus, Tiramisu first creates asingle base graph representing the above features/pro-tocols for the entire network. From that, it then createstraffic class-specific graphs.

ACLs prevent specific traffic classes from enter-ing/leaving a router, and filters block advertising specificprefixes. Thus, for a given traffic class, Tiramisu logi-cally removes an edge of the traffic class graph if there is(i) an ACL that blocks this traffic class on the interfaceassociated with this edge, or, (ii) a filter that blocks thistraffic class’s destination prefix on the routing processassociated with the start node of this edge.

Static routes are also traffic class specific. Tiramisuadds an edges of label “s” from the node representing thesource of the static route process to all nodes associatedwith the next hop router, for the relevant traffic class.

Node communities. Communities are also a trafficclass specific construct. In Tiramisu, each node rep-resenting a BGP process has three sets of communi-ties: “ac” - communities added by the node, “rc” com-munities removed by the node, and “mc” communitiesmatched/acted upon by the node. The action associatedwith each “mc” is to either block traffic or change edgemetrics like local preference and MED.

3.1 Prohibited paths

All the algorithms we present to verify various proper-ties rely on examining properties of source-destinationpaths in the above graph abstraction. In designing thesealgorithms care must be taken to avoid certain paths thatcannot materialize in a real network under any failure.This mainly arises due to constraints on interactions be-tween routing protocols and route redistributions/staticroutes, and due to communities. We show how we reasonabout which routes cannot be taken due to the formerreason. We handle communities later.

For example, as shown in §2, traffic crossing from anEGP to the IGP layer gets dropped if the intermediatenodes at the IGP layer do not learn/have a route to dst.

Intuitively, a path can materialize in the network ifthere is relevant forwarding information available toward

the destination in the RIB of some routing process run-ning on each router on the path, and thus in the router’sFIB; this observation forms the basis for the seminalwork on static reachability analysis of IP networks [17].A path cannot materialize if it includes a router with noforwarding information for the destination. To keep trackof such disallowed paths for graph traversal or when com-puting paths taken, Tiramisu uses a “tainting” strategy.In a preprocessing step, taints track how forwarding in-formation may flow through a network’s RIBs and maypopulate FIBs. Taint on a node implies that a node mayknow of a route to a destination; lack of a taint impliesthe node will not know of any route.

For each traffic class, taints propagate in the corre-sponding Tiramisu graph starting at the destination dst;the routing process connected to the dst is tainted, as areall other peer processes of the same routing protocol in-stance on other routers. Taints flow across redistributionand static route edges, and spread throughout the verticescorresponding to a single IGP. Taints also propagate fromone iBGP peer to another (because iBGP peers learn ofroutes from each other).

Specifically, to identify nodes in the IGP layer thathave a route to dst, Tiramisu first finds the node at theEGP layer that redistributes its route to IGP, and taintsthis node.1 Next, Tiramisu marks all nodes in the IGPprocess’s layer as tainted, implying that they may knowof a route to dst in their RIB due to redistribution fromthe EGP (recall: IGPs flood learned routes, to routers inthe same IGP process). On the other hand, if there is noredistribution from EGP into IGP, the corresponding IGPnode is marked as untainted, because it will not not havea route to the destination in its RIB (but a route may existin the router’s FIB because the EGP process computeda route); all other nodes in the same IGP process arealso untainted. E.g. in Figure 3, all OSPF nodes areuntainted.

The above processes is applied to all routing layersand EGP/IGP crossings, of which there may be many ina network. In the end, we have a tainted graph, with asubset of vertices carrying taints and others without taint;the latter vertices will not have a forwarding entry to thedestination.

Given a tainted graph, we determine which paths aredisallowed. Intuitively, an untainted node must resideon the same router as a tainted node for a path to thedestination to be found in the router’s FIB. Thus, a po-tentially valid path has ≤ 2 consecutive untainted nodes

1Note, that edges in Tiramisu represent the flow of traffic and notof advertisements. Hence, redistribution of EGP routes into IGP isrepresented by an “r” edge from the IGP node to the EGP node.

5

(e.g., the two IGP nodes). Any path with > 2 consec-utive untainted nodes is an invalid or prohibited path.E.g. in Figure 3, Cosp f → Dosp f → Aosp f is a subpath ofprohibited paths.

Although our example considered iBGP, IGP and BGPinteractions, taints can also be used to understand howstatic routes can shape the flow of routing information,and consequently impact paths.

Theorem 1 The above approach correctly identifies allprohibited path.

We prove this by contradiction as shown in Ap-pendix B.1.

A potentially valid path may still not materialize inthe network due communities. We handle communitiesspecially in our algorithms in subsequent sections.

4 Avoiding Path EnumerationFor many properties of interest, verifying them does notactually require computation or enumeration of actualpaths that may arise in the network; rather, we simplyneed to know of the existence of paths (possibly of acertain kind). Knowing of the existence of paths is farsimpler than actually computing paths under a givenfailure.

Thus, we design two sets of algorithms, one that tra-verses a graph along all potential paths, and another thatuses ILPs to reason about the number of paths and high-level path properties (maximum path length). We presentthese algorithms in the context of the properties they aidin the verification of. All algorithms use taints to take in-valid paths out of consideration, and handle communitiesspecially as mentioned above.

4.1 Tiramisu Depth First Search

DFS (Depth First Search) is a graph traversal algorithmthat identifies all nodes that are reachable from a givensrc. A node remains unvisited after DFS iff there existsno path from src to that node. The DFS algorithm canbe naturally used to verify certain reachability policies,such as, P1 (“always blocked”): P1 is true iff underevery failure scenario, there exists no path between srcand dst node. However standard DFS does not avoidprohibited path constraints, nor does it account for howBGP communities may shape the existence of paths. E.g.in Figure 3, paths with subpath Cbgp→Cosp f → Bosp f →Dosp f are prohibited. But DFS identifies them as valid.

We first assume no communities are in use and proposea modified DFS algorithm (Algorithm 1) that avoids pro-hibited paths, T DFS (Tiramisu DFS). We then accountfor communities by invoking T DFS on carefully selectedsubgraphs.

Algorithm 1 Tiramisu Depth First SearchInput:

G is the graphsrc is the root node for depth first search

1: procedure initialize(G,src)2: Set all nodes of G as unvisited3: T DFS(G, src, 0)

Input:u is current node being traversednumUntaint tracks # consecutive untainted nodes

4: procedure T DFS(G, u, numUntaint)5: Set node u as visited6: for each e ∈ G.outgoingEdge[u] do7: v← end node of edge e8: Ignore v, if already visited9: if v is untainted then

10: increment numUntaint by 111: Ignore v, if numUntaint is 312: else13: numUntaint← 014: T DFS(G, v, numUntaint)

Like DFS, T DFS explores all unvisited outboundneighbors of a node (line 6 to 8). However, it usesan additional variable numUntaint to avoid traversingprohibited paths.

In this algorithm, numUntaint represents the numberof consecutive untainted nodes seen by T DFS; T DFSsimply avoids all paths with more than 2 consecutiveuntainted nodes (line 11). E.g., in Figure 4, when T DFSreaches node Dosp f , numUntaint becomes 3 and T DFSstops exploring this path. T DFS’s complexity is the sameas standard DFS.

T DFS algorithm does not handle communities. Nodesmay add communities to path advertisements that oth-ers filter on. When such filtering happens, the corre-sponding paths cannot be taken by network traffic. Tosupport communities, Tiramisu uses another simple algo-rithm, commT DFS (Algorithm 2). commT DFS makes aconstant number of calls to T DFS verify P1. Thus, itsoverall complexity is still polynomial-time.

As shown in §2, nodes can add, remove or block oncommunities. In presence of such nodes, the order inwhich these nodes are traversed in a path decides if dstis reachable.

In the commT DFS algorithm, Tiramisu first checksif src and dst are unreachable according to T DFS (line5, 6). If they are reachable, then commT DFS checksif all paths that connect src to dst has i) a node (sayX) that blocks on a community in an advertisement fordst (line 8 to 10), ii) followed by a node (say Y) thatadds that community to an advertisement for dst (line11 to 13), iii) and no node between them X and Y that

6

Algorithm 2 Always blocked with communitiesInput:

G is the graphs and d are source and destination nodesIt uses T DFS to answer multiple (un)reachability

queries1: procedure commTDFS(G,s,d)2: cA← nodes that add community3: cR← nodes that remove community4: cB← nodes that block on community5: if s and d are unreachable by T DFS then6: return true, since nodes are already unreachable7: else8: remove nodes ∈ cB9: if s and d are reachable by T DFS then

10: return false, since there is path between s andd that is not blocked by community

11: add back all nodes, remove nodes ∈ cA12: if nodes ∈ cB and d are reachable by T DFS then13: return false, since blocking nodes can receive

advertisement without community14: add back all nodes, remove nodes ∈ cR15: if nodes ∈ cB and their respective nodes ∈ cA are

unreachable by T DFS then16: return false, since communities are always re-

moved before reaching blocking nodes17: Return true, if all above conditions fail

removes that community (line 14 to 16). (Recall herethat advertisements flow in the opposite direction of datatraffic.) If all these conditions are satisfied, then srcand dst are unreachable. If any of these conditions areviolated, the nodes are reachable. In Figure 4, EBGP→ABGP→ DBGP violates condition (iii).

This algorithm can also verify P2 (“always waypoint-ing”). After removing the waypoint, if src can reach dst,then there is a path that can reach dst without traversingthe waypoint.

4.2 Tiramisu Hedge Min-cut

Another policy that does not require path enumerationis P3 (reachable < K failures): here, operators want toverify that the src can reach dst as long as there are < Klink failures. For this, ARC [6] computes the min-cutfor the src-dst graph, i.e., the minimum number of edgeswhose removal can disconnect the graph. If min-cut is≥ K, then P3 is satisfied.

Tiramisu uses a multilayer hedge graph abstraction.Unfortunately, finding min-cuts in hedge graphs is aknown NP-hard problem [7]. Thus, to verify P3, we pro-pose a new Integer Linear Program (ILP) to find hedgegraph min-cut, while also accounting for communitiesand prohibited paths.

For this property, Tiramisu uses integer variables, sim-

ilar to Minesweeper [2]’s SMT encoding. However,Tiramisu’s property-specific ILP encoding is simpler thanMinesweeper as it only adds integer and boolean vari-ables relevant to hedge min-cut and avoids variables andconstraints associated with route selection, which signifi-cantly reduces variables and constraints associated withroute advertisements. Minesweeper’s detailed encodingexercises all of these variables in an attempt to computethe actual path, which renders it slow even for a smallnumber of failures (K = 1).

Until now, the edges in our graphs represented the flowof traffic from src to dst. For ease of understanding, inspecifying the ILP, we reverse the edges to represent theflow of advertisement from dst to src. For brevity, weexplain the constraints at a high-level, leaving precisedefinitions to Appendix C. Equation numbers belowrefer to equations in Appendix C.

The objective of the ILP is to minimize the number ofphysical link failures required to disconnect the src fromdst. Note that a single physical edge’s failure can cutmultiple Tiramisu graph edges. All such edges belongingto hedge i will share the same Fi variable. Our objectivethen is:

Objective: minimize ∑i∈phyEdges

Fi (1)

Advertisement Constraints. We first discuss the con-straints added to represent reachability of advertisements.The base constraints state that the dst originates the ad-vertisement (Eqn 4). To disconnect dst from src, theadvertisement must not reach the src (Eqn 5). For othernodes, the constraint is that an advertisement reaches anode if it gets propogated on any of its incoming edge(Eqn 6).

Next, we discuss constraints on propogation of adver-tisements. An advertisment can propogate through anedge e, if it reaches the start node of the edge, the physi-cal edge does not fail, the advertisement does not carrya community that is blocked on this node, and if edge ecarries atleast one advertisement that does not create aprohibited path. This is represented as shown in Eqn 7.

Communities. The base constraints state that eachnode that adds a community forwards that communityand each node that removes that community does notforward it further (Eqn 8 and Eqn 9).

For other nodes, we add the constraint that node nforwards a community c iff any of its inbound neighborsforwards that community to node n (Eqn 10).

Finally, we add the constraint that an edge e carriesa blocked community iff the start node of e forwards a

7

community that is blocked by the end node of edge e(Eqn 11).

Prohibited Path Constraints. An edge e, that onlypropogates advertisements that create prohibited paths,is an edge that satisfies the condition that start and endnodes of e are untainted (uEdges) (Eqn 12). Also, thestart node receives advertisement only from untaintedneighbors and not from any other neighbor. Such edgesalways create subpaths with three consecutive untaintednodes (§3.1) (Eqn 12).

4.3 Tiramisu Longest path

Similar to cut-based properties, there are others that don’tneed path enumeration and can instead rely on custom-crafted ILPs. These properties compute high-level at-tributes of paths, such as bounds on length. An exampleis the always bounded length policy (P4). For a givenK, P4 is true if under every failure scenario, traffic fromsrc to dst never traverses a path longer than K hops.Enumerating all possible paths and finding their lengthis infeasible. However, this policy can be verified effi-ciently by viewing it as a variation of the longest pathproblem: for a given K, P4 is true if the longest pathbetween src and dst in the graph is ≤ K.

ILP. Finding the longest path between two nodes ina hedge graph is also NP hard [9]. To verify P4, wepropose another ILP whose objective is to maximize thenumber of inter device edges (dEdges) traversed by anadvertisement (Ai). Note again that the path traversed bythe advertisement is the opposite of traffic flow. But bothhave same path length. Notably, this ILP uses even fewerconstraints and variables compared to §4.2, and thus canrun even faster relative to Minesweeper.

Objective: maximize ∑i∈dEdges

Ai (2)

Again, we elide detailed constraints to the Ap-pendix D.

Single Path Constraints. We add constraints to en-sure that only one path gets advertised, that the dst sendsthe advertisement, and that the src receives it (Eqn 14and Eqn 15).

For other nodes, we add the flow conservation property,i.e. sum of incoming flows is equal to outgoing flows(Eqn 16).

Advertisement Constraints. Finally, we add con-straints on propogation of advertisements. Here, an ad-vertisement can be blocked on edge e if carries a blockedcommunity or satisfies the prohibited path constraints(Eqn 17).

5 Path specific policiesAs opposed to the properties in the previous section, theremaining properties that one may wish to verify requireknowing the exact path taken in the network. Considerthe policy that encodes a preference order among paths,say, pre f Path = P1 >> P2 >> P3. This states that P1should be taken by default; when path P1 fails, path P2(if available) should be taken; and if P1 and P2 both fail,then P3, if available, should be taken. A path, say P1,can become unavailable for multiple reasons: (a) a routerconfiguration along P1 was updated to withdraw or filterthe route to the dst; (b) a link along P1 failed. In all cases,we need to reason about what alternate paths materialize,and whether the materialized path is indeed the path thatwas meant to be taken. Simple graph traversal is thusinsufficient to verify this property. We need the actualpaths under various failure scenarios.

Griffin et al [8] observed that control plane protocolsessentially attempt to solve the stable paths problem.Based on this, to compute paths, Minesweeper [2] mod-els protocol interactions (advertisement generation/pro-cessing, best path selection, etc) as an instance of thestable paths problem. Furthermore, Griffin et al [8] pro-posed the Simple Path Vector Protocol (SPVP) for ob-taining a set of paths that form a solution to the stablepath problem.

Inspired by these two studies, we propose T PV P(Tiramisu PVP), a protocol that can compute actualnetwork paths taken, while avoiding prohibited paths.T PV P extends SPVP with rich messages and per-nodecomputation.

Whereas Minesweeper encodes the router’s path se-lection actions based on SMT constraints, T PV P modelsthem using a protocol. Thus, Tiramisu emulates the exe-cution of SPVP, whereas Minesweeper attempts to findthe SPVP output.

To find the path traversed by a specific traffic class,Tiramisu runs T PV P on its traffic class-specific graph.

5.1 Tiramisu Path Vector Protocol

Advertisements. Similar to SPVP, for each node u, andfor each of its peer v, T PV P uses rib− in(u⇐ v) vari-ables to keep track of the most recent route advertisementreceived from v. Each advertisement consists of (i) apath from the advertisement sender to the advertisementsource (dst of the traffic class), and (ii) the multi-metricpath cost to reach dst. Also, rib(u) represents the currentpath (best advertisement and best path cost) to reach dstfrom u.

Route Selection. In Minesweeper, the values ofrib− in were computed by applying import and export

8

filters on these advertisements. In T PV P, these are re-placed by an updateCost function. This function (i)rejects prohibited paths (§3.1), (ii) rejects paths blockedby communities (§4.1), and, (iii) computes the multi-atrribute path cost to reach dst through v. Each node hasa choices function (⊕) that provides a partial order overall rib− in(u⇐∗) paths. u then updates its rib(u) valueto the best choice/most preferred rib− in(u⇐∗).

rib− in also keeps track of the list of communities car-ried by the advertisements, and the updateCost functionupdates the community lists.

The updateCost and choices functions are closelymodeled based on the route import, route export androute selection constraints of Minesweeper and SPVP.These functions are either inferred from configuration orbased on standard conventions. E.g. choices could bebased on BGP selection policy (prefer highest local pre f ,shortest ASPathlength etc), and, updateCost specifieshow to calculate the cost of rib− in(u⇐ v) as a functionof best(v) and edge weight ewu→v (add osp f costs, setlocal pre f to ewu→v.l p etc).

Algorithm. Our final T PV P protocol is shown in Al-gorithm 3. In the initial state, T PV P sets the rib value ofall nodes except dst to null (line 2). rib(dst) is set to ε ,since it originates the advertisement (line 3). There arethree main steps in each iteration. First, for each nodeu, T PV P computes all its rib− in values based on theadvertisement sent by its neighbors (line 6 to 8). If thecurrent rib(u)’s value is different from the best advertise-ment, then T PV P updates rib(u) and sends a message toall peers of node u (line 9 to 11). This message containsthe updated rib(u).

T PV P terminates when none of the nodes receive anymessages about the new rib− in values (line 12, 13).This termination condition represents the converged stateof the newtork. Similar to SPVP, if network reaches aconverged state, then T PV P will find the best path to dstfor all nodes.

Theorem 2 Assuming the network control plane con-verges, T PV P always finds the exact path taken in thenetwork under any given failure scenario.

We leverage correctness of Minesweeper and SPVPto establish the correctness of T PV P as shown in Ap-pendix B.2.

We return to the correctness of T DFS.

Theorem 3 T DFS traverse all real network paths thatcould materialize under some failure scenario, and doesnot traverse any path that cannot materialize on anyfailure scenario.

Algorithm 3 Tiramisu Path Vector ProtocolInput:

G is the graphdst is the destination node

1: procedure T PV P(G,dst)2: set rib(i) values of all nodes i except dst to null3: set rib(dst) to ε , since dst originates the advertisement4: while true do5: for each u ∈ G.nodes do6: for each v ∈ peers(u) do7: rib − in(u ⇐ v) =

updateCost(rib(v),ewu→v)8: compute best(u) using choices([rib −

in(u⇐∗)])9: if rib(u) is different from best(u) then

10: rib(u) = best(u)11: send messages to all peers of node u

about change in rib(u)12: if no node receives any messages then13: break, since network has converged

We leverage the correctness of Theorem 1 and The-orem 2 to prove correctness of Theorem 3 as shownin Appendix B.3.

5.2 Tiramisu Yen’s algorithm

We return to determining how to verify the path prefer-ence property introduced earlier in this section. We firstobserve that there are similarities between analyzing pathpreference (P5) and finding the k shortest paths in a graph(the “k shortest paths problem” [18]). This is because, ink shortest paths problem, the kth shortest path is takenonly when k−1 paths have failed. Enumeration for allpossible failures of all k−1 shorter paths is tedious. Tohandle that, Yen [18] introduced a new algorithm to findthe k shortest paths. The algorithm uses dynamic pro-gramming to avoid enumerating failures/link removals.Next, we explain Yen’s algorithm in detail. Then, we pro-pose TY EN (Algorithm 4), a simple extension to Yen’salgorithm to verify P5. We show how in a simple fashionTY EN uses Yen’s algorithm to verify P5.

Yen’s algorithm. Yen uses two lists: listA (keepstrack of the shortest path seen so far) and listB (keepstrack of candidates for the next shortest path). At thestart, Yen finds the first shortest path P (line 2) from srcto dst using any shortest path algorithm (e.g. Dijkstra’s).

Next, it takes every node n in path P (line 11, 12),and finds the rootPath and spurPath of that node. TherootPath of n is the subpath of P from src to node n (line13). The spurPath is the shortest path from node n to dst(line 18) that satisfies the following two conditions: i) thepath must not have any node from rootPath (line 17); ii)the path must not traverse any outgoing edge e from n that

9

is part of any of the previous k−1 shortest paths havingthe same rootPath (line 15, 16). E.g. if A→ B→C→Dis the shortest path and A→ B is rootPath at node B, thenB→C→ E→ D spurPath is invalid.

Yen combines the rootPath and spurPath to form anew path nP (line 19). nP is added to listB if it doesn’talready exist in listA or listB (line 22). After traversingall nodes of path P, Yen adds P to listA (line 24). Next,Yen picks the shortest path from listB and reruns theprevious steps with this path as the new P. This continuestill Yen finds k paths.Algorithm 4 Tiramisu Yen

Input:G is the graphsrc, dst are source and destination nodeslevel is no. of path specified in path-preference policypre f Path, a map of preference level and path

1: procedure TY EN (G,src,dst, level)2: P← path from src to dst returned by T PV P3: P.level ← 14: P.eRemoved ← [], as best path requires no edge re-

moval5: listA← [], tracks paths already considered as P6: listB← [], tracks paths not yet considered7: do8: mostPre f ← most preferred path in pre f Path

whose edges don’t overlap with P.eRemoved9: if P 6= mostPre f then

10: return false, since path preference is violated11: for i← 0 to P.length - 1 do12: sNode← ith node of P13: rootPath← subpath of P from src to sNode14: for each sp ∈ listA do . paths in listA15: if sp has same rootPath at sNode then16: remove outgoing edge of sNode in sp,

so that path sp is not considered . also remove other edgesthat share the same hedge

17: remove all nodes and edges (hedge) of rootPathexcept sNode to avoid loops

18: spurPath← path from sNode to dst returnedby T PV P

19: nP← rootPath + spurPath20: nP.level← P.level +121: nP.eRemoved← all edges removed in this iter-

ation22: add nP to end of listB if np is valid and np /∈

[listA, listB]23: add back all nodes and edges to the graph24: add P to listA25: P← remove first path from listB26: while P.level < K27: return true, since loop didn’t find preference violation

Tiramisu Yen. The three failure modes highlighted at

the beginning of this section boil down to some specificedges being removed from the Tiramisu graph, causingthe path in question to cease to exist. Given this, we makea few modifications to Yen, resulting in an algorithm wecall TY EN, to work atop our multi-graph abstraction andcheck for path preference; simply put, like Yen, we checkif P2 is the next path chosen after P1, and so on (line 8to 10).

TY EN uses T PV P instead of Dijkstra to find the short-est path. In addition, TY EN associates each path witha variable, eRemoved. This keeps track of edges thatwere removed to prefer this path (line 21). During eachiteration of P, TY EN identifies the most prefered path inpathPre f that did not have an edge in P.eRemoved (line8). If this path varies from P, then preference is violated(line 9, 10).

Revisiting: Reachability < K with ACLs. The ILPin §4.2 for P3 is not accurate when data plane ACLsare in use. ACLs do not influence route advertisements.Hence, routers can advertise routes for traffic that endsup being blocked by ACLs. Recall that during graphcreation, Tiramisu removes edges that are blocked byACLs §3. This leads to incorrect mincut computation aswe show next:

Consider a src-dst traffic class in a network. Say thereare three network paths P1, P2 and P3 in increasingorder of cost that src learns of toward dst, and say P2has a data plane ACL on it. Suppose further than allthree paths are edge disjoint. If a link failure removesP1, the control plane would select path P2 to forwardto dst, but all packets from src are dropped at the ACLon P2. In this case, a single link failure (that took downP1) is sufficient to disconnect src and dst; that is, thetrue mincut is 1. In contract, Tiramisu would removethe offending ACL edge from the graph abstraction, andsince this preserves paths P1 and P3, Tiramisu wouldconclude that the mincut is 2, which is incorrect.

We address this issue as follows. Nodes can becomeunreachable when failures either disconnect the graphor lead to a path with an ACL. Thus, we compute twoquantities: (1) L: How many minimum failures causethe control plane to pick a path that first encounters ablocking ACL? (2) N: How many minimum failurescause disconnection in the Tiramisu graph with the ACLedge removed (as originally proposed)? The true min-cutvalue is min(L,N)?

Computing N in (2) is straightforward. For computingL in (1), we first construct a graph without removingedges for ACLs. Then, we run TY EN until we find thefirst path with a dropping ACL on it. Say this was the Mth

shortest. Then, we use an ILP to compute the minimum

10

% of NetworksProtocols/Modifiers University Datacenter Topology Zoo

eBGP 100% 100% 100%iBGP 100% 0% 100%OSPF 100% 97% 100%

Static routes 100% 100% 0%ACLs 100% 100% 0%

Route Filters 100% 97% 100%Local Prefs 50% 0% 100%

MPLS+VRF 100% 0% 0%VLAN 100% 0% 0%

Community 100% 100% 100%

Table 2: Configuration constructs used in networks

0 50 100 150Network size

0

100

200

300

400

# grap

h ve

rtices

a) Graph vertices

0 50 100 150Network size

0

200

400

# grap

h ed

ges

b) Graph edges

Figure 5: Size of multilayer graphs of all networksnumber of edge failures L that will cause the previousM−1 shortest paths to fail. If min(L,N)≥ K then P3 issatisfied.

So far, we covered five properties to verify. But, usingthe algorithms in this and the previous section as thebasis, Tiramisu can verify a variety of other propertieswhich Minesweeper can also verify. We list these, alongwith how Tiramisu verifies them, in Appendix A.

6 EvaluationOur implementation of Tiramisu is written in Java. Weuse Batfish [5] to parse router configurations. Fromthese, we generate our multilayer graphs (§3). We im-plemented all our verification algorithms (§4 and §5) inJava. Tiramisu uses Gurobi [1] to solve our ILPs. Inall, this amounted to ≈ 7K lines of code. We evaluateTiramisu on a variety of issues:

• How quickly can Tiramisu verify different policies?• How does Tiramisu perform relative to

Minesweeper [2]?• How does Tiramisu’s performance scale with network

size?

All our experiments were performed on machines with40 core 2.2 GHz Intel Xeon Silver Processors and 192

0 50 100 150Network size

0

100

200

300

400

# grap

h ve

rtices

a) Graph vertices

0 50 100 150Network size

0

200

400

# grap

h ed

ges

b) Graph edges

Figure 6: Size of multilayer graphs of all networks

0 50 100 150Network size

10

20

Time (μs)

base traffic-class

Figure 7: Graph generation time (all networks)

Uni1(9) Uni2(24) Uni3(26) Uni4(35)Universities

0

20

40

60

80

Time (m

s)

P1P3P4P5

Figure 8: Verify policies on university configs

GB RAM.

6.1 Network Characteristics

In our evaluation, we use configurations from (a) 4 uni-versity networks, (b) 34 real datacenter networks oper-ated by a large OSP, and, (c) 7 networks from topologyzoo dataset [12]. The university networks have 9 to35 devices. The university networks are the richest interms of configurations constructs. They support eBGP,iBGP, OSPF, Static routes, ACLs/Filters, community,local preference, MPLS+VRFs and VLANs. The data-center networks have between 2 and 24 devices. They donot employ local preference, MPLS or VLANs. Finally,the topology zoo networks have between 33 and 158 de-vices. The configs for these networks were syntheticallygenerated using NetComplete [3]. These generated con-figs do not have Static routes, ACLs, MPLS or VLANsas Netcomplete cannot model them. Table 2 shows whatpercentage of networks in these datasets support eachnetwork protocol/modifier.

Figure 6 characterizes the size of all the multilayergraphs generated by Tiramisu for these networks. It firstshows the number of nodes and edges used to representthe base multigraph of these networks. We observe twooutliers in both Figure 6a and Figure 6b. These occur fornetworks Uni−2 (24 devices) and Uni−3 (26 devices),from the university dataset. These networks have multi-ple VRFs and VLANs, and Tiramisu creates nodes (andedges between these nodes) per VRF/VLAN per rout-ing process. Note also that for the other networks, thenumber of routing processes per device varies. Hence,the number of nodes and edges do not monotonicallyincrease with network size.

11

10 20Network size

101102103

Time (m

s)

a) P1 - Time

10 20Network size

101102103

Time (m

s)

b) P2 - Time

10 20Network size

102

103

Time (m

s)

c) P4 - Time

10 20Network size

101102103

Time (m

s)

d) P5 - TimeMinesweeper Tiramisu

Figure 9: Performance under all failures: Tiramisu vs Minesweeper (datacenter networks)

10 20Network size

0

250

500

Spee

dup

a) P1 - Speedup

10 20Network size

0

500Sp

eedu

pb) P2 - Speedup

10 20Network size

0

20

40

Spee

dup

c) P4 - Speedup

10 20Network size

25

50

75

Spee

dup

d) P5 - Speedup

Figure 10: Speedup under all failures: Tiramisu vs Minesweeper (datacenter networks)

Policies. We consider five types of polices: (P1) al-ways unreachable, (P2) always waypointing, (P3) alwaysreachable with < K failures, (P4) always bounded length,and (P5) path preference. Recall from other sections:P1 and P2 use T DFS; P5 uses T PV P and TY EN algo-rithms; P3 uses ILP and TY EN.; And P4 uses anotherILP. Using these policies, we evaluate the performanceof all our algorithms.

6.2 Verification Efficiency

We examine how efficiently Tiramisu can construct andverify these multilayer graphs. First, we evaluate the timerequired to generate these graphs. We use configurationsfrom all the networks. Figure 7 shows the time takento generate the base graphs and per traffic class-specificgraph for all networks. Tiramisu can generate thesegraphs, even for large networks, in ≈ 30 µs. The timeto generate the traffic-class graph from the base graph isatmost 3 µs on average per traffic-class.

Next, we examine how efficiently Tiramisu can verifyvarious policies. Since the university networks are therichest in terms of configuration constructs, we use themin this experiment. Figure 8 shows the time taken toverify policies P1, P3, P4, and P5. In this and all theremaining experiments, the values shown are the mediantaken over 100 runs for 100 different traffic classes. Errorbars represent the std. deviation.

We observe that P1 can be verified in less than 3 ms.Since it uses a simple polynomial-time graph traversalalgorithm (T DFS), it is the fastest to verify among allpolicies. The time taken to verify P5 is higher than P1,because T PV P and TY EN algorithms are more complex,as they run our path vector protocol to convergence tofind paths (and in TY EN’s case the protocol is invokedmany times). Finally, P3 and P4, both use an ILP and, asexpected, are slowest to verify. However, they can stillbe verified in ≈ 80 ms per traffic class.

Although Uni2 and Uni3 have fewer devices thanUni4, they have more nodes and edges in their Tiramisugraphs (§6.1). Hence it takes longer to verify policies onthem.

6.3 Comparison with Minesweeper

Next, to put our performance results in perspective, wecompare Tiramisu with Minesweeper. In this experimentwe use the real datacenter networks. We consider policiesP1, P2, P4, and P5. In Minesweeper, we have to specifythe number of failures K; Minesweeper then verifies ifthe policy holds as long as there are ≤ K failures. Toverify a property under all failure scenarios, we set thevalue of K to number of physical links in the network - 1.Figure 9 (a, b, c, and d) shows the time taken by Tiramisuand Minesweeper to verify these policies. Figure 10 (a,b, c, and d) shows the speedup provided by Tiramisu foreach of these policies.

For policies that use T DFS (P1 and P2), Tiramisu’sspeedup is as high as 600-800X . For P4, the speedup isas high as 50X .

P5 is the only policy where speedup does not increaseswith network size. This is because larger networks havelonger path lengths and more possible candidate paths.Both of these affect the complexity of the TY EN al-gorithm. The number of times TY EN invokes T PV Pincreases significantly with network size. Hence thespeedup for P5 is relatively less, especially at larger net-work sizes.

Note that since the number of nodes and edges did notmonotonically increase with network size (§6.1), the timeto verify these policies does not monotonically increaseeither.

Next, we compare the performance of Tiramisu andMines- weeper for the same policies but without failures,e.g. “currently reachable” instead of “always reachable”.Tiramisu verifies these policies by generating the actual

12

10 20Network size

50

100

Spee

dup

a) P1 - Speedup

10 20Network size

304050

Spee

dup

b) P2 - Speedup

10 20Network size

10

20

Spee

dup

c) P4 - Speedup

10 20Network size

10

20

30

Spee

dup

d) P5 - Speedup

Figure 11: Speedup under no failures: Tiramisu vs Minesweeper (datacenter networks)

5 10 15 20 25Network size

100

101

102

Redu

ction ratio P3

P4

Figure 12: Variable reduction ratio - Tiramisu vs Minesweeper

Bics (33

)

Arnes (

34)

Latnet (

69)

Uninett

(69)

Gtsce (1

49)

Colt (15

3)

UsCarri

er (158

)

Network size

0

25

50

75

100

Time (m

s)

P1P3

P4P5

Figure 13: Performance on scale (topology zoo networks)

path using T PV P. Figure 11 (a, b, c, and d) shows thespeedup provided by Tiramisu for each of these policies.Even for no failures, Tiramisu significantly outperformsMinesweeper across all policies. Minesweeper has toinvoke the SMT solver to find a satisfying solution evenin this simple case.

To shed further light on Tiramisu’s benefits w.r.t.Mineswe- eper, we compare the number of variablesused by Mineswee- per’s SMT encoding and Tiramisu’sILP encoding to verify P3 and P4. In this experiment,we track reduction ratio, which is the number of vari-ables in Minesweeper divided by number of variablesin Tiramisu. This is shown in Figure 12. As expected,Tiramisu uses significantly fewer variables. Also, P3uses more variables than P4. Hence it has a lower re-duction ratio. Tiramisu uses integer variables only forthe aspects that matter towards the property in question,where Minesweeper uses binary and integer variablesthroughout its general encoding, irrespective of the prop-erty in question. This is one reason for Minesweeper’spoor performance.

6.4 Scalability

Until now, all our experiments were on small networks.In this section, we evaluate Tiramisu’s performance onlarge networks from the topology zoo. Figure 13 showsthe time taken to verify policies P1, P5, P3, and, P4.Tiramisu can verify these policies in < 0.12 s, even in

these large networks.As expected, verifying P1 (T DFS) is significantly

faster than all other policies, and it is very low acrossall network sizes. However, for larger networks, time toverify P5 (TY EN) is as high as P4. Again, this due tolarger networks having longer and more candidate paths.Large networks also have high diversity in terms of pathlengths. Hence, we see more variance in the time toverify P5 compared to other policies.

For large networks, the time to verify P3 is signifi-cantly higher than other policies. This happens becauseP3’s ILP formulation becomes more complex, in termsof number of variables, for such large networks.

7 Related WorkWe surveyed various related studies in detail, in earliersections. Here, we survey others that were not coveredearlier.

Aside from Minesweeper and ARC, there are othercontrol plane verification tools that attempt to verifypolicies against various environments (failures or ad-vertisements). ERA [4] symbolically represents con-trol plane advertisements which it propagates througha network and transforms it based on how routers areconfigured. ERA is useful to verify reachability againstarbitrary external advertisements, but it does not have thefull coverage of control plane constructs as Tiramisu orMinesweeper to analyze a range of policies. Bagpipe [16]is similar in spirit to Minesweeper and Tiramisu, but itonly applies to a network that only runs BGP. FSR [15]focuses on encoding BGP path preferences.

Batfish [5] and C-BGP [14] are control plane simula-tors. They analyze the control plane’s path computationas a function of a given environment, e.g., a given fail-ure or an incoming advertisement, by conducting lowlevel message exchanges, emulating convergence, andcreating a concrete data plane. Tiramisu also conductssimulations of the control plane; but, for certain poli-cies, Tiramisu can explore multiple paths at once viagraph traversal and avoid protocol simulation. For otherpolicies, Tiramisu only simulates a simplified protocol(SPVP) running over a multi-layer multi-attributed graph,and the simulations are conducted in parallel per trafficclass.

13

8 LimitationsOne limitation of graph-based control plane models isthat they cannot symbolically model advertisements,which is easier to do for SMT-based tools. Tiramisushares this drawback. This means that Tiramisu cannotexhaustively explore if there exists an external adver-tisement that could potential lead to a property viola-tion; Tiramisu can only exhaustively explore link failures.Tiramisu would have to be provided a concrete instanti-ation of an advertisement; in such a case, Tiramisu cananalyze the network under the given advertisement anddetermine if any policies can be violated.

A related issue is that Tiramisu cannot be applied toverify control plane equivalence: Two control planes areequivalent, if the behavior of the control planes (pathscomputed) is the same under all advertisements and allfailure scenarios.

In essence, while Tiramisu can replace Minesweeperfor a vast number of policies, it is not a universal replace-ment. Minesweeper’s SMT-encoding is useful to exploreadvertisements.

9 ConclusionWhile existing graph-based control plane abstractions arefast, they are not as general. SMT-based abstractions aregeneral, but not fast. In this paper, we showed that graphscan be used as the basis for general and fast network ver-ification. Our insight is that, rich, multi-layered graphs,coupled with algorithmic choices that are customizedper policy can help achieve the best of both worlds. Ourevaluation of a prototype (which we will release opensource) shows that we can offer 10-600X better speedthan state-of-the-art, scale gracefully with network size,and model all key features found in network configura-tions in the wild. This work does not raise any ethicalissues.

References[1] Gurobi. http://www.gurobi.com/, 2017.

[2] R. Beckett, A. Gupta, R. Mahajan, and D. Walker.A general approach to network configurationverification. In SIGCOMM, 2017.

[3] A. El-Hassany, P. Tsankov, L. Vanbever, andM. Vechev. Netcomplete: Practical network-wideconfiguration synthesis with autocompletion. In15th USENIX Symposium on Networked SystemsDesign and Implementation (NSDI 18). USENIXAssociation, 2018.

[4] S. K. Fayaz, T. Sharma, A. Fogel, R. Mahajan,T. D. Millstein, V. Sekar, and G. Varghese.

Efficient network reachability analysis using asuccinct control plane representation. InSymposium on Operating Systems Design andImplementation (OSDI), 2016.

[5] A. Fogel, S. Fung, L. Pedrosa,M. Walraed-Sullivan, R. Govindan, R. Mahajan,and T. Millstein. A general approach to networkconfiguration analysis. In Symposium onNetworked Systems Design and Implementation(NSDI), 2015.

[6] A. Gember-Jacobson, R. Viswanathan, A. Akella,and R. Mahajan. Fast control plane analysis usingan abstract representation. In SIGCOMM, 2016.

[7] M. Ghaffari, D. R. Karger, and D. Panigrahi.Random contractions and sampling for hypergraphand hedge connectivity. In Proceedings of theTwenty-Eighth Annual ACM-SIAM Symposium onDiscrete Algorithms, pages 1101–1114. SIAM,2017.

[8] T. G. Griffin, F. B. Shepherd, and G. Wilfong. Thestable paths problem and interdomain routing.IEEE/ACM Transactions on Networking (ToN),10(2):232–243, 2002.

[9] D. Karger, R. Motwani, and G. D. Ramkumar. Onapproximating the longest path in a graph.Algorithmica, 18(1):82–98, 1997.

[10] P. Kazemian, G. Varghese, and N. McKeown.Header space analysis: Static checking fornetworks. In Symposium on Networked SystemsDesign and Implementation (NSDI), 2012.

[11] A. Khurshid, X. Zou, W. Zhou, M. Caesar, andP. B. Godfrey. VeriFlow: Verifying network-wideinvariants in real time. In Symposium onNetworked Systems Design and Implementation(NSDI), 2013.

[12] S. Knight, H. X. Nguyen, N. Falkner, R. Bowden,and M. Roughan. The internet topology zoo. IEEEJournal on Selected Areas in Communications,29(9):1765–1775, 2011.

[13] H. Mai, A. Khurshid, R. Agarwal, M. Caesar, P. B.Godfrey, and S. T. King. Debugging the data planewith Anteater. In SIGCOMM, 2011.

[14] B. Quoitin and S. Uhlig. Modeling the routing ofan autonomous system with c-bgp. IEEE network,19(6):12–19, 2005.

14

[15] A. Wang, L. Jia, W. Zhou, Y. Ren, B. T. Loo,J. Rexford, V. Nigam, A. Scedrov, and C. Talcott.Fsr: Formal analysis and implementation toolkitfor safe interdomain routing. IEEE/ACMTransactions on Networking (ToN),20(6):1814–1827, 2012.

[16] K. Weitz, D. Woos, E. Torlak, M. D. Ernst,A. Krishnamurthy, and Z. Tatlock. Scalableverification of border gateway protocolconfigurations with an SMT solver. In ACMSIGPLAN International Conference onObject-Oriented Programming, Systems,Languages, and Applications (OOPSLA), 2016.

[17] G. G. Xie, J. Zhan, D. A. Maltz, H. Zhang,A. Greenberg, G. Hjalmtysson, and J. Rexford. Onstatic reachability analysis of ip networks. InINFOCOM 2005. 24th Annual Joint Conference ofthe IEEE Computer and Communications Societies.Proceedings IEEE, volume 3, pages 2170–2183.IEEE, 2005.

[18] J. Y. Yen. Finding the k shortest loopless paths in anetwork. management Science, 17(11):712–716,1971.

A Other PoliciesSome of the other policies that Tiramisu can verify arelisted below:

Always Chain of Waypoints (P6). Similar to way-pointing, we remove nodes associated with each way-point, one at a time. Using T DFS, we check if nodesassociated with one of the preceding waypoints in thechain can reach nodes associated with one of the follow-ing waypoints in the chain.

Equal Bound (P7). This policy checks that all pathsfrom src to dst are of the same length. The objective ofthe ILP in §4.3 can be changed to find the shortest pathlength. If the longest and shortest path length varies, thenthis policy is violated.

Always isolated (P8). Two traffic classes are alwaysisolated if they never traverse the same link. ARC [6] ver-ified this policy by checking if two traffic class-specificgraphs have any edge in common or not. Similarly, inTiramisu, we check if the two traffic class-specific graphshave any hedge in common.

Multipath Consistency (P9). Multipath consistencyis violated when traffic is dropped along one path butblocked by an ACL on another. To support multipathsin T PV P, we change the best variable to track multiplebest advertisements. Using T PV P, Tiramisu can identifythe number of best paths. We run T PV P on graphs with

and without removing edges for ACLs. If the number ofpaths varies with and without ACLs, then the policy isviolated.

Always no black holes (P10). Black holes occurwhen traffic gets forwarded to a router that does nothave a valid forwarding entry. Blackholes are causedby i) ACLs: router advertises routes for traffic that areblocked by their ACLs; ii) static routes: the next-hoprouter of a static route cannot reach dst. Tiramisu usesT DFS to check these conditions. For (i) Tiramisu firstcreates the graph without removing edges for ACLs. LetR be the router with a blocking ACL. If src can reachrouter R and R can reach dst (using T DFS), then traf-fic will reach router R under some failure, and then getdropped because of the ACL. For (ii) if src can reach therouter with static route and the next-hop router cannotreach dst, then the traffic gets dropped.

B ProofsB.1 Prohibited Paths

Any path with > 2 consecutive untainted nodes is a pro-hibited path

Theorem 1 The approach specified in §3.1 correctlyidentifies all prohibited path.

Proof: We will prove this by contradiction.In a router, each routing process will have its own

Routing Information Base (RIB) that has routes learnedby that routing process. The routes of all RIBs are alsoinserted in the Forwarding Information Base (FIB) of thatrouter. Hence, the “f” edge exists between all routingprocess node and the fib node for each router.

The progression of taints is modelled based on GRIBgraph of [17]. In GRIB, edge adjacency exists only forRIB adjacency. Similarly taints are spread only acrossRIB adjacent processes. Similar to [17], taints crossintra-device routing processes only with redistribution.

A prohibited path can exist only if you traverse a routerthat does not have a valid fib entry. Assume such a pathexists and is not identified by Tiramisu’s tainting strategy.Such paths have to satisfy the following three criterias: (i)there has to be a router where path goes from tainted nodeto untainted node, i.e. from node W representing routingprocess or FIB (e.g. Cbgp, C f ib in Figure 3) that hasforwarding entry, to node X representing routing processthat does not have forwarding entry (Cosp f ); (ii) nodeX has to be followed by a node Y (Dosp f ) representinganother router with the same routing process; and (iii)node Y has to be succeeded by node Z which will eitherbe a node representing node Y ’s router’s FIB (D f ib) or

15

another router with the same routing process as Y (Aosp f ).Here both these nodes have to be untainted. However,this ends up with a path with three untainted nodes, whichare classified by Tiramisu as prohibited. This contradictsthe assumption that this path is not identified by Tiramisu.

B.2 TPVP

Theorem 2 Assuming the network control plane con-verges, T PV P always finds the exact path taken in thenetwork under any given failure scenario.

Proof: We leverage correctness of Minesweeper andSPVP to establish the correctness of T PV P. In SPVP,there is no restriction on order in which messages areprocessed by different routers. As long as there existseven one stable path, SPVP will find it irrespective ofthe order of message processing. In T PV P (line 5), weprocess messages in a fixed round-robin order. Since anyordering of messages in SPVP leads to a valid solution,a fixed ordering of messages should also lead to a validsolution.

The body of the loop code of T PV P is equivalent toSPVP. The updateCost function is a function that wasnot mentioned in the original SPVP algorithm. In theoriginal SPVP algorithm, rib-in (u ⇐ v) representedpaths received from neighbor v. And the properties ofthose paths e.g. local-pref, ospf-weight etc were assignedbased on import and export policies. Minesweeper alsomodelled rib-in (u⇐ v) as import and export policy con-straints. Tiramisu’s updateCost function also updatespath properties based on import and export policy con-straints, similar to Minesweeper [2] and SPVP [8].

B.3 TDFS

Theorem 3 TDFS traverses all real network paths thatcould materialize under some failure scenario, and doesnot traverse any path that cannot materialize under anyfailure scenario.

Proof: Let path P be the path traversed by a particularsrc-dst pair in the actual network under some failure.Now assume this path is not traversed by T DFS becauseit does not exist in the graph. This contradicts Theorem 2,which showed that T PV P always finds the exact pathused in the actual network, which implies the path mustexist in the graph. Thus, T DFS will traverse all networkpaths that could materialize under some failure scenario.

Now assume there exists some path P′ in the actualnetwork that also exists in the graph. Also assume it isnot traversed by T DFS. This means this path P′ containsthree or more consecutive tainted nodes. This contradictsTheorem 1 as it cannot be a real network path. Thus,T DFS does not traverse any path that cannot materializeunder any failure scenario.

C Tiramisu Min-cutTable 3 lists the boolean indicator variables and functionsused in encoding of the ILP for the property in §4.2. Wenow provide the full ILP with detailed constraints; werepeat the description of the constraints to ensure easeof reading. Recall, edges in our graphs represented theflow of traffic from src to dst. For ease of understanding,in specifying the ILP, we reverse the edges to representflow of advertisement from dst to src.

Name Description

Var

iabl

e

Fe set as 1 if edge e failsAe set as 1 if advertisement propagates on edge eRn set as 1 if advertisement reaches node nBe set as 1 if edge e carries blocked communityPe set as 1 if edge e only propagates advertisement

that create prohibited pathsCn,c set as 1 if node n forwards community c

Func

tion

nodes returns all nodes of graphedges returns all edges of graphdEdges returns all inter-device edges of graphphyEdges returns all physical edges of graphsuEdges returns all edges of graph with untaint start and

end nodesoNodes returns all nodes except src and dstiE(n) returns incoming edges of node noE(n) returns outgoing edges of node niUE(n) returns incoming edges from untainted neigh-

bors of node niN(n) returns start nodes of all incoming edges of

node nac(n) returns communities added by node nrc(n) returns communities removed by node nbc(n) returns communities blocked on node noc(n) returns communities /∈ [ac(n),bc(n),rc(n)]start(e) returns start node of edge eend(e) returns end node of edge e

Table 3: Variables and Functions

ILP The objective of the ILP is to minimize the num-ber of physical link failures required to disconnect thesrc from dst. All edges belonging to hedge i will sharethe same Fi variable.

Objective: minimize ∑i∈phyEdges

Fi (3)

Advertisement Constraints We first discuss the con-straints added to represent reachability of advertisements.The base constraints state that the dst originates the adver-tisement. To disconnect dst from src, the advertisementmust not reach the src.

Rdst = 1 (4)

16

Rsrc = 0 (5)

For other nodes, we add the constraint that advertise-ment reaches n if it gets propagated (Ae) on any of itsincoming edge e.

∀n ∈ oNodes,Rn =∨

e∈iE(n)

Ae (6)

Note that logical AND, OR operators can be repre-sented as constraints in ILP.

Next, we discuss constraints on propagation of adver-tisements. An advertisement can propagate through anedge e, if it reaches the start node (n) of the edge (Rn), thephysical edge does not fail (¬Fe), the advertisement doesnot carry a community (¬Be) that is blocked on this node,and if edge e carries atleast one advertisement that doesnot create a prohibited path (¬Pe). This is represented as

∀n ∈ oNodes,∀e ∈ iE(n) :Ae = Rn∧¬Fe∧¬Be∧¬Pe (7)

Community Constraints The base constraints statethat each nodes that adds the community forwards thatcommunity and each nodes that removes that communitydoes not forward it.

∀n ∈ nodes,∀c ∈ ac(n) :Cn,c = 1 (8)

∀n ∈ nodes,∀c ∈ rc(n) :Cn,c = 0 (9)

For other nodes, we add the constraint that node nforwards a community c iff any of its inbound neighbors(iN(n)) forwards that community to node n.

∀n ∈ nodes,∀c ∈ oc(n) :

Cn,c =∨

i∈iN(n)

Ci,c (10)

Finally, we add the constraint that an edge e carries ablocked community iff the start node of edge e forwardsany community that is blocked by end node of edge e.

∀e ∈ edges :

Be =∨

c∈bc(end(e))

Cstart(e),c (11)

Prohibited Path Constraints An edge e, that onlypropagates advertisements that create prohibited paths,is an edge that satisfies the condition that (Eqn 12) thestart and end nodes of edge e are untainted (uEdges).

Furthermore the start node receives advertisement onlyon edges from untainted neighbors (Aue) and not on edgesfrom any other neighbor (Ane). Such edges always createsubpaths with three consecutive untainted nodes.

∀e ∈ uEdges : Pe =

∨ue∈iuE(start(e))

Aue

∧¬

∨ne∈¬iuE(start(e))

Ane

(12)

D Tiramisu Longest PathWe now specify the ILP corresponding to §4.3 in detail.For ease of reading, we also provide the full descriptionof the objective and the constraints.

Recall that to verify P4, we propose another ILPwhose objective is to maximize the number of inter de-vice edges (dEdges) traversed by an advertisement (Ai).Note again that, the path traversed by the advertisementis the opposite of traffic flow. Our objective is as follows:

Objective: maximize ∑i∈dEdges

Ai (13)

Single Path Constraints To ensure that only one pathgets advertised, and that the dst sends the advertisementand the src receives the advertisement, we add the con-straints in

∑out∈outEdge(dst)

Aout = 1 (14)

∑in∈inEdge(src)

Ain = 1 (15)

For other nodes, we add the flow conservation property,i.e. sum of incoming flows is equal to outgoing flows

∀n ∈ oNodes : ∑in∈iE(n)

Ain = ∑out∈oE(n)

Aout (16)

Advertisement Constraints Next, we add constraintson propagation of advertisements. An advertisement canbe bloc- ked on edge e if it satisfies the community (Be)and path prohibition (Pe) constraints. These are similarto Eqn 11 and Eqn 12.

∀e ∈ edges : Ae ≤ ¬(Be ∨ Pe) (17)

17