Consist

ent *Complete *

Well D

ocumented*Easyt

oR

euse* *Evaluated

*

OOPSLA*

Artifact *

AEC

Scalable Verification of Border Gateway Protocol Configurationswith an SMT Solver

Konstantin Weitz Doug Woos Emina TorlakMichael D. Ernst Arvind Krishnamurthy Zachary Tatlock

University of Washington, USA{weitzkon, dwoos, emina, mernst, arvind, ztatlock}@cs.washington.edu

AbstractInternet Service Providers (ISPs) use the Border GatewayProtocol (BGP) to announce and exchange routes for de-livering packets through the internet. ISPs must carefullyconfigure their BGP routers to ensure traffic is routed reli-ably and securely. Correctly configuring BGP routers hasproven challenging in practice, and misconfiguration has ledto worldwide outages and traffic hijacks.

This paper presents Bagpipe, a system that enables ISPsto declaratively express BGP policies and that automaticallyverifies that router configurations implement such policies.The novel initial network reduction soundly reduces policyverification to a search for counterexamples in a finite space.An SMT-based symbolic execution engine performs thissearch efficiently. Bagpipe reduces the size of its search spaceusing predicate abstraction and parallelizes its search usingsymbolic variable hoisting.

Bagpipe’s policy specification language is expressive:we expressed policies inferred from real AS configurations,policies from the literature, and policies for 10 JuniperTechLibrary configuration scenarios. Bagpipe is efficient:we ran it on three ASes with a total of over 240,000 lines ofCisco and Juniper BGP configuration. Bagpipe is effective:it revealed 19 policy violations without issuing any falsepositives.

Categories and Subject Descriptors D.2.4 [Software Engi-neering]: Software/Program Verification

Keywords Bagpipe, BGP, domain-specific language, solver-aided languages, correctness

1. IntroductionOver 3 billion people are connected to the Internet throughuniversity and corporate networks, regional ISPs, and nation-wide ISPs [18]. These networks, collectively known as Au-tonomous Systems (ASes), use the Border Gateway Protocol(BGP) to exchange route announcements, which describepaths that traffic can take across the Internet. To route traf-fic reliably and securely, ASes must configure their BGP-speaking border routers to implement policies restricting howroute announcements can be used and exchanged.

Router misconfigurations are common and have led tomany visible failures [8, 35, 28, 27]. For example, in 2008,in response to a government order, the Pakistan Telecom ASintended to block YouTube by announcing a non-existentYouTube route to ASes within Pakistan. Due to a misconfigu-ration, this route was also advertised to an AS outside Pak-istan, PPCC. PPCC forwarded the route to its neighbors. Thenon-existent route to YouTube then quickly spread through-out the Internet and was selected for packet forwarding bymost ASes. YouTube was then unavailable to most Internetusers, as their packets to YouTube were incorrectly forwardedto Pakistan Telecom. About two hours later, PPCC fixed theproblem by disconnecting Pakistan Telecom from the Inter-net [5]. If Pakistan Telecom had correctly implemented itspolicy to only block YouTube to ASes within Pakistan, or ifPPCC had correctly implemented its policy to only importroutes that an AS actually owns, this outage could have beenavoided.

Other failures could also have been prevented by correctlyimplementing appropriate BGP policies [15]. However, do-ing so with little to no tool support is difficult and expensive,particularly since large ASes maintain millions of lines of fre-quently changing configurations distributed across hundredsof routers [19, 40].

This paper presents Bagpipe1, which uses automatic veri-fication to prevent router misconfiguration. An AS operatorexpresses control-plane policies as declarative specifications.

1 Bagpipe is open-source, see http://bagpipe.uwplse.org

Then, Bagpipe verifies that the router configurations satisfythe specifications.

A straightforward implementation of Bagpipe would needto consider infinitely many possible network traces to deter-mine whether a BGP configuration violates a router policy;if no trace is a counterexample to the specification, then thepolicy holds. The novel initial network reduction enablesBagpipe to verify a specification by searching for counterex-amples over a finite space of network traces.

While the initial network reduction makes Bagpipe’ssearch space finite, the space is still far too large to permitbrute-force enumeration. Bagpipe partitions the search spaceby case splitting on the ranges of some symbolic variables,which enables parallel symbolic exploration using an SMT-based symbolic execution engine [39] to efficiently exploitrange information within each partition. Bagpipe also usespredicate abstraction to further reduce the search space,coalescing announcements with the same control flow inthe BGP configuration.

Bagpipe is an expressive, efficient, and effective verifi-cation tool. Bagpipe’s policy specification language is ex-pressive: specifications are rich enough to express policiesinferred from real AS configurations, express BGP policiesfound in the literature (such as the Gao-Rexford model [14]and prefix-based filtering [29]), and express policies for 10configuration scenarios from the Juniper TechLibrary [22,4]. Bagpipe is efficient: we applied it to three ASes withover 240,000 lines of BGP configuration written in the Ciscoand Juniper configuration language (which Bagpipe supportsout-of-the-box). Bagpipe is effective: it revealed 19 policyviolations without issuing any false positives.

This paper’s contributions include:

• A means of expressing AS-wide policies as declarativespecifications (Section 3).

• The initial network reduction, which enables Bagpipe toverify specifications by searching a finite set of networktraces (Section 4).

• An efficient verifier based on this reduction, which em-ploys a novel combination of search space partitioning,unused variable omission, symbolic execution, and predi-cate abstraction (Section 5).

• An evaluation of Bagpipe on 10 configuration scenariosand on 3 real ASes with over 240,000 lines of Cisco andJuniper BGP configuration (Section 6).

2. OverviewThis section provides background on BGP, shows examplepolicies that AS operators may need to guarantee, and illus-trates how Bagpipe automatically verifies such policies.

2.1 BackgroundThe core of the Internet is a network of routers that forwarddata packets toward their destinations. Routers learn of aroute — a path through other routers to a destination — via

1 # Control plane for router r.23 # The following RIBs contain the announcements4 # received, selected, and forwarded by the router r.5 # Initially, no such announcements are available.6 ∀ n pfx, adjRIBsIn[n][pfx] = notAvailable7 ∀ pfx, locRIB[pfx] = notAvailable8 ∀ n pfx, adjRIBsOut[n][pfx] = notAvailable9

10 # Process incoming announcement ann11 # from the source src for the prefix pfx.12 while (src, pfx, ann) = recvUpdateMessage():13 adjRIBsIn[src][pfx] = ann1415 # Import and select announcements.16 locRIB[pfx] = notAvailable17 for n in r.neighbors:18 annImp = IMPORT(r, n, pfx, adjRIBsIn[n][pfx])19 if better(annImp, locRIB[pfx]):20 locRIB[pfx] = annImp2122 # Export and forward announcements.23 for n in r.neighbors:24 annExp = EXPORT(r, n, pfx, locRIB[pfx])25 if annExp != adjRIBsOut[n][pfx]:26 adjRIBsOut[n][pfx] = annExp27 sendAnnBGP(n, pfx, annExp)

Figure 1. Simplified BGP Router Implementation. This pseudocodesketches the high-level behavior specified by the BGP standard,RFC 4271 [32]. AS operators can only configure the IMPORT and EXPORTrules. Section 3.1 describes restrictions on the IMPORT and EXPORT programsto ensure conformity with RFC 4271; for example, IMPORT must returnnotAvailable on announcements with AS routing loops.

the Border Gateway Protocol (BGP). Using BGP, a routerselects at most one route % per destination p and, onceselected, adds itself to % and forwards the announcementto its neighbors. The router then forwards packets for p tothe router from which % was received. The forwarding andselection of routes takes place on the control plane. It runsseparately and asynchronously from the data plane, on whichrouters forward data packets using the routes selected by thecontrol plane. Bagpipe verifies control plane policies thatcharacterize both (1) which routes may be selected and usedby the data plane and (2) which routes a router may forward.

The Internet’s routers are owned by Autonomous Systems(ASes) such as universities, corporate networks, regional ISPs,and nationwide ISPs; each router is operated by exactly oneAS. ASes are identified by globally unique AS numbers. Bag-pipe verifies control plane policies for a single AS becauseAS operators do not control their neighbors’ configurations.We call the routers owned and operated by the single ASunder consideration internal; other routers are external andmay behave arbitrarily.

Router control planes forward routes via update messages,which consist of a prefix and an announcement.

• A prefix is a set of destination IP addresses. Sets ofdestinations are represented in Classless InterdomainRouting (CIDR) notation. A set of destinations in CIDR

notation ip/n (e.g., 192.168.1.0/24) is referred to as aprefix, because it contains all IP addresses starting withthe same n bits as ip.

• An announcement contains metadata about the route in-cluding: AS-path, a list of the ASes the announcement haspreviously traversed which is used for avoiding routingloops as well as serving as a measure of “distance” onthe Internet; communities, a set of flags which are uninter-preted by the BGP protocol but can be used by ASes toexchange additional information about a route; and localpreference, a number that influences route selection.

Figure 1 sketches how each Internet router r managesthe control plane, as described by the BGP specification,RFC 4271 [32]. r maintains three tables, or Routing In-formation Bases, to track announcements it has received(adjRIBsIn), selected for routing packets on the data plane(locRIB), and forwarded to its neighbors (adjRIBsOut). Ini-tially (lines 6 to 8), all these tables contain only notAvailableto indicate that r has not received any announcements. r thenenters an infinite loop to process incoming BGP update mes-sages. Each incoming update message (line 12) includes theaddress src of the router that forwarded the update message,the prefix pfx that the update message is offering to route to,and the announcement ann. Receipt of an update messagetriggers a three-phase process.

1. r stores the received announcement in its adjRIBsIn rout-ing table for announcements for prefix pfx from neighborsrc (line 13).

2. The router applies the configurable IMPORT program to eachannouncement in adjRIBsIn for prefix pfx and each neigh-bor n (lines 17 to 18). IMPORT may transform or replacethe announcement, including returning notAvailable.The router chooses the best (possibly-transformed) an-nouncement (lines 19 to 20). An announcement withhigher local preference is considered better than an an-nouncement with lower local preference. If two announce-ments have the same local preference, other factors likethe length of their AS-path are considered. The routerstores the best announcement in locRIB[pfx], which isused by the data plane to forward packets.Since AS operators cannot directly control the better testthat selects announcements, they influence selection byconfiguring their IMPORT rules to either drop announce-ments or to modify the local preference so that the bettertest (line 19) will select their desired announcements.

3. For every neighbor, the router applies the configurableEXPORT program to the announcement that was selectedas best. If the result of EXPORT differs from the announce-ment that was most recently sent to that neighbor, thenEXPORT’s result is stored in adjRIBsOut and forwarded tothe neighbor. A router can retract a previously advertised

announcement by announcing notAvailable for the route’sdestination prefix.

To verify an AS configuration, it is not sufficient to checkthe IMPORT and EXPORT programs for each router individually.This is because internal routers establish invariants on theannouncements they forward to other internal neighbors (e.g.,the ISP Internet2 establishes the invariant that its internalrouters never send announcements for invalid prefixes). Theconfigurations of those neighbors often rely on such invari-ants. Thus, the verification task is to establish that the ASconfiguration correctly implements the policy for all routersin the AS at all times, for all possible sequences of incomingupdate messages.

Figure 2 illustrates an example network of several ASes.One AS is under consideration with three internal routers r0,r1, and r2. The external router e0’s data-plane is capable ofdirectly delivering packets with destinations in the prefix128.208.7.0/24. Note that the data plane delivers packetsalong a route in the opposite direction from which updatemessages for that route were forwarded through the controlplane. Consider the following example trace, i.e.,, sequenceof events happening in the AS:

1. The internal router r0 receives an update message m0from the external router e0, which indicates that e0 has aroute to the prefix 128.208.7.0/24.

2. r0 extends m0’s announcement with its own AS number,and selects it to route packets.

3. r0 forwards the selected announcement in the updatemessage m1 to its neighbors, including the internal routerr1.

4. r1 receives the update message m1 from r0.2

5. r1 selects the received announcement to route packets.6. r1 forwards the selected announcement in the update

message m2 to its neighbors, including the external routere0.

r0, r1, and e0 can use the selected announcements to forwardpackets with destinations in the prefix 128.208.7.0/24: forexample, 128.208.7.1 and 128.208.7.42 but not 128.208.8.0.

2.2 PoliciesAn AS operator decides on policies that restrict the announce-ments that the AS’s BGP routers’ control planes select andexchange. Some policies are used to ensure security prop-erties: for instance, a country’s ASes might want to ensurethat national traffic is routed within the country [33]. Otherpolicies are used to uphold business contracts, for example,to honor agreements that certain announcements should notbe publicly shared [35].

This section uses the BlockToExternal policy as a run-ning example. BlockToExternal prohibits internal routersfrom forwarding “classified” announcements to external

2 r1 does not extend m1 with its own AS number because it received theannouncement from an internal neighbor.

r0

r2

r1e0

e1128.208.7.0/24

m2m0

m1

AS

router

internalexternal

connection

update messagepropagationroute

Figure 2. BGP Routing Example. The AS under consideration (black)consists of three internal routers connected to five external routers. Theserouters use the Border Gateway Protocol (BGP) to establish a route betweene1 and e0. This route can be used to forward packets.

routers. An announcement is considered classified if it hadthe BTE community set when it was received by the AS.

Bagpipe’s specifications are assertions written in theRacket language that restrict the announcements selectedand exchanged by an AS’s internal routers. In Bagpipe, theBlockToExternal policy is expressed as:

(implies(external? receiver)(not (has-community? ’BTE (original sent))))

The policy asserts that if an announcement (sent) is for-warded by an internal router to one of its neighboring exter-nal routers (receiver ), then the announcement that gave riseto sent (i.e., the original announcement received by the ASbefore it was modified by import and export filters), must nothave contained the BTE community.

AS operators implement policies by configuring their AS’srouters. A BGP router is configured with IMPORT and EXPORTprograms that modify announcements in order to influencewhich ones are selected and forwarded (and therefore used toroute packets on the data plane). These programs are typicallywritten in either the Juniper or Cisco configuration language,which are loop-free imperative programming languages withdomain-specific syntax and semantics.

Consider the following snippet of Juniper configurationcode to implement the BlockToExternal specification forthe neighbor with IP 62.40.125.17.

neighbor 62.40.125.17 { — start neighbor configurationexport [RULE1 RULE2 ... BLOCK-BTE...]; — call export rules

...}policy-statement BLOCK-BTE { — define export ruleterm block-to-external {

from community BTE; — match announcementthen reject; }} — reject matched announcement

The export statement runs each passed rule (policy-statement)from left to right, stopping once a rule either accepts or re-jects the announcement. Each executed rule can also modifythe announcement. Note that if the AS operator does notcorrectly order statements, they may not fire on the announce-ments they are intended to check or modify. The BLOCK-BTErule rejects an announcement if it has the BTE community set,and otherwise does nothing.

Juniper/Ciscoconfiguration

specification Bagpipe specification holds /counterexample

SMT Solver

Figure 3. Bagpipe Workflow. Bagpipe takes a specification and an AS’srouter configurations as input. Bagpipe verifies that the configuration cor-rectly implements the specification using multiple concurrent SMT solvercalls, and then either indicates success or returns a counterexample that ASoperators can use to debug their configurations.

To manually verify that an AS’s router configurations im-plement the BlockToExternal specification, an AS operatormust check that:

• For every external neighbor, every router has an exportrule that drops announcements whose BTE community isset. These rules are similar to the BLOCK-BTE rule in theexample above.

• Each of these rules is executed: no preceding rule acceptsthe announcement. For example, the rules RULE1 and RULE2in the above example configuration should not acceptannouncements whose BTE community is set.

• No rule in any router clears the BTE community.

More complex policies, such as the Gao-Rexford policydiscussed later in this paper, are even harder to verify man-ually. Large ASes have millions of lines of configuration,making manual verification of even simple policies expensiveand error-prone.

2.3 BagpipeAs illustrated in Fig. 3, Bagpipe enables AS operators to ex-press their AS-wide policies as declarative specifications, andBagpipe then automatically verifies that their router configu-rations correctly implement such policies. Since AS operatorsgenerally do not have access to their neighboring ASes’ con-figurations, Bagpipe soundly assumes that external routersmay exhibit any behavior. Bagpipe verifies control planepolicies to ensure that inter-AS guarantees are upheld (e.g.,confidential announcements are not leaked) and to preventcontrol-plane performance issues. For example, ASes oftenreject update messages whose prefix is too long (correspond-ing to a too-small set of destinations) in order to avoid fillingtheir routing tables with too many routes and thus degradingperformance.3

To verify that an AS’s routers are safely configured,Bagpipe must ensure that every possible behavior of theAS satisfies the policy. Bagpipe models the behavior of anAS as a trace of announcement receipts, route selections,

3 From the data plane perspective, this behavior may seem unreasonable:if an AS would be willing to forward a packet with an IP in the prefix64.57.29.0/24 to a particular AS, it seems it should also be willing toforward a packet with an IP in the range 64.57.29.0/25 (since the latterrepresents a strict subset of the former). However, filling RIBs with manyannouncements for small prefixes can severely degrade performance.

and forwarding events in the network. Bagpipe models thepolicy in Racket as a boolean predicate over traces. Forexample, to verify the BlockToExternal specification fromSection 2.2, Bagpipe must guarantee that in every trace, nointernal router forwards a classified announcement. Notethat Bagpipe verifies policies even for oscillating and non-deterministic BGP networks; thus, the guarantees providedby Bagpipe hold even outside a BGP network’s steady state.

To verify a policy, it is not feasible to brute-force searchthe space of all possible traces for a counterexample, becausethe set of all traces is infinite. Not only can external routerssend arbitrarily long sequences of announcements, but theBGP protocol itself can oscillate (in which case routers keepannouncing and withdrawing routes indefinitely).

To address this challenge, Section 4 introduces the initialnetwork reduction (INR). In the initial network, all RIBs con-tain only notAvailable (as in Fig. 1 after line 8). Intuitively,the INR exploits the observation that if a router will everselect or forward a particular announcement, it will also doso the initial network. An AS’s behavior is thus “maximal”in the initial network. Bagpipe exploits this fact to reduce itssearch for a counterexample from the infinite set of tracesto a finite set of traces that process two arbitrary externalannouncements in the initial network.

The initial network reduction makes Bagpipe’s searchspace finite, but it is still too large for naive brute force search.Section 5 introduces four optimizations that Bagpipe im-plements to improve search performance. Instead of usingbrute-force search, Bagpipe searches the space symbolicallyusing an SMT solver (§5.1). Bagpipe uses predicate abstrac-tion to soundly reduce the number of announcements thatmust be considered (§5.2). Bagpipe hoists symbolic variablesso that search can be parallelized across many nodes in acluster (§5.3). Bagpipe avoids enumerating hoisted variableswhose values are not used (§5.4).

3. SpecificationsThis section first formalizes the behavior of the BGP controlplane in Section 3.1; and then uses the formalization todescribe a general framework for expressing BGP policyspecifications, and what it means for these specifications tohold in Section 3.2.

3.1 Control Plane FormalizationThis section and Figure 4 formalize the BGP control planediscussed in Section 2. A more complete formalizationappears in a technical report [41].

The set of routers is represented as a set of IP addresses,connected via bidirectional links over which routers exchangeannouncements.

A trace is a sequence of events within the AS underconsideration. There are three kinds of event. The eventrecv(r, i, p, a) means that a router r received an updatemessage from its neighbor i for prefix p with announcement

IP := [0, 255]× [0, 255]× [0, 255]× [0, 255] — ip addressesP := IP × [0, 32] — prefixes

R ⊆ IP — routersRi ⊆ R — internal routersRe ⊆ R — external routersin(r) = out(r) = neighbor(r) ⊆ R — router r’s neighbors

event := recv : (r : R)→ in(r)→ P → A→ event| slct : (r : R)→ P → A→ event| frwd : (r : R)→ out(r)→ P → A→ event

trace ⊆ list(event) — a trace is a legal sequence of events

notAvailable — placeholder used when announcement is not availableA := {pref : N, communities : P(N), aspath : list(asn)}

— BGP announcement; P stands for powerset.A := {current : A, original : A} ∪ {notAvailable} — tagged ann

imp : (r : R)→ in(r)→ P → A→ A — IMPORT program of Fig. 1exp : (r : R)→ out(r)→ P → A→ A — EXPORT program of Fig. 1

adjRIBsIn : trace → (r : R)→ in(r)→ P → A — active receivedlocRIB : trace → (r : R)→ P → A — active selectedadjRIBsOut : trace → (r : R)→ out(r)→ P → A — active fwded

Figure 4. Control Plane Formalization. A router r is connected via in-coming in(r) and outgoing out(r) links. A trace is a valid sequence ofevents within the AS under consideration. imp and exp refer to a router’sconfigurable import and export programs. In any state reachable via sometrace t, adjRIBsIn(t, r, i, p) contains the announcement most recently re-ceived for a prefix p by a router r from r’s neighbor i. locRIB(t, r, p)contains the announcement most recently selected for a prefix p by a routerr. adjRIBsOut(t, r, o, p) contains the announcement most recently for-warded for a prefix p by a router r to r’s neighbor o.

a. The event slct(r, p, a) means that a router r selectedan announcement a for prefix p. The event frwd(r, o, p, a)means that a router r forwarded an update message to itsneighbor o for prefix p with announcement a.

A valid trace must have the following 4 properties: 1)A router can receive an update message from an internalneighbor i only if there has previously been a correspond-ing forwarding event by i. A router can always receive anupdate message from an external neighbor, because Bagpipetreats external neighbors as “havoc”. 2) A router selects anannouncement in its locRIB if and only if it is chosen as theresult of the import and selection process (shown in Fig. 1 onlines 16 to 20). 3) A router can forward an update messageonly if its announcement is the result of applying the exportprocess (shown in Fig. 1 on lines 23 to 27) to a selected an-nouncement. 4) The receipt of an update message by a routermust be immediately followed by all the resulting select andforward events at that router. The end of Section 2.1 providesan example trace. A network state is reachable via some tracet if it is the result of executing every event in the trace t.A represents an announcement as specified by the BGP

protocol. It is a record consisting of an AS path aspath (theAS numbers of every AS traversed by the announcement),local preference pref (the better routine of Fig. 1 prefersannouncements with higher local preference), and a set ofcommunities communities (used by ASes to exchange addi-tional information about a route; a community is a number).

Bagpipe can be easily extended to track additional informa-tion in announcements, e.g., to model BGP extensions.

For reasoning, Bagpipe uses tagged announcements whichconsist of the current value of the announcement, plus theoriginal value of the announcement at the time when itentered the AS under consideration. Tracking this additionalprovenance information enables AS operators to write policyspecifications such as BlockToExternal .

A router r imports an announcement ai for prefixp received from neighbor i using the import programimp(r, i, p, ai). imp is a loop-free imperative program that ei-ther modifies or drops (by returning notAvailable) the passedannouncement. The export program exp is modeled similarly.These programs are called IMPORT and EXPORT in the BGPspecification and in Section 2.

For compliance with RFC 4271, there are some restrictionson how the IMPORT and EXPORT programs can operate. Forexample, both programs must map notAvailable inputs tonotAvailable outputs, IMPORT must mark announcements withAS routing loops as notAvailable, and EXPORT must preventannouncements received from internal neighbors from beingexported to other internal neighbors.

Each router r stores the most recently received, selected,and forwarded announcements. These are the active an-nouncements that can be used to forward packets. Specif-ically:

• adjRIBsIn contains the active received announcements:the most recently received announcement for each prefixand neighbor of r. A new announcement for some prefixfrom a neighbor always implicitly withdraws (and thusdeactivates) any previously-received announcements forthat prefix and neighbor.

• locRIB contains the active selected announcements: themost recently selected announcement for each prefix.

• adjRIBsOut contains the active forwarded announce-ments: the most recently forwarded announcement foreach prefix and neighbor of r. Older routes are implicitlywithdrawn by BGP.

Given a trace t, a router r, a neighbor i of r, and a prefix p,the function adjRIBsIn computes the network state resultingfrom the execution of every event in the trace t, i.e., thestate reachable via the trace t, and returns the most recentlyreceived announcement by r from i with prefix p in that state,or notAvailable if no announcement has been received. Notethat the adjRIBsIn formalism takes more arguments thanadjRIBsIn of Fig. 1. When applied to a trace t and a routerr, it corresponds to adjRIBsIn. The locRIB and adjRIBsOutare modeled similarly.

3.2 Policy SpecificationsThis section describes a general framework for expressingBGP policies specifications, and what it means for these spec-ifications to hold. Later sections show how Bagpipe automat-

V := {router ∈ R; prefix ∈ P ; sender ∈ R; received ∈ A;selected ∈ A; bestSender ∈ R; bestReceived ∈ A;receiver ∈ R; sent ∈ A}

spec : Type := V → bool

specHolds(τ : spec) :=∀ (t : trace) (r ∈ Ri) (p ∈ P ) (i ∈ in(r)) (o ∈ out(r)),let i∗ := inLocRIB(t, r, p)

ai := adjRIBsIn(t, r, p, i)a∗i := adjRIBsIn(t, r, p, i

∗)a∗l := locRIB(t, r, p)ao := adjRIBsOut(t, r, p, o)

in τ({router := r; prefix := p; sender := i; received := ai;selected := a∗l ; bestSender := i

∗; bestReceived := a∗i ;receiver := o; sent := ao}) = true

Figure 5. Policy Specification Definition. A policy specification τ : specis a predicate over a record of variables V , representing certain activeannouncements. specHolds(τ) defines what it means for a specification tohold.

ically verifies that specifications in this general frameworkhold.

A specification is an invariant over an AS’s active an-nouncements, i.e., an invariant over an AS’s routing informa-tion bases. Formally, a specification τ is a predicate over arecord of variables V . V represents certain values in router ’srouting information bases, namely an announcement receivedfrom a sender for prefix , an announcement selected forprefix which was received as bestReceived from bestSender ,and an announcement sent to a receiver for prefix .

The definition of a specification spec, and what it meansfor a specification τ to hold specHolds(τ), is given in Fig-ure 5. A specification τ holds (i.e., an AS’s routers correctlyimplement a specification), if and only if the invariant ex-pressed by τ is true for every router state reachable via anytrace. Formally, τ holds if and only if for any network trace t,router r, prefix p, neighbor i, and neighbor o, τ returns truewhen invoked with the most recently received announcementai for prefix p and neighbor i, the most recently selected an-nouncement a∗l for prefix p which was received as a

∗i from i

∗

(starred variables are associated with the selected announce-ment), and the most recently forwarded announcement ao forprefix p and neighbor o, along with r, p, i, and o.

The function inLocRIB(t, r, p) computes the neighbori∗. inLocRIB(t, r, p) first passes all announcements in theadjRIBsIn(t, r, i, p) to the imp program, and then deter-mines the neighbor with the “best” imported announcement.

Expressiveness Specifications are not arbitrary invariantsover all active announcements — specifications can onlyquantify over variables in V . In particular, a specificationcannot depend on routers other than r, announcements forprefixes other than p, announcements received from neigh-bors other than i and i∗, and announcements forwarded toneighbors other than o. For example:

• A specification cannot require routers to forward a re-ceived announcement for prefix p, if and only if an an-nouncement for some other prefix q has been selected.

• A specification cannot require routers to select a receivedannouncement from neighbor i, if and only if exactly k(where k ≥ 3) announcements from other neighbors havebeen received.

• A specification can require routers to select a receivedannouncement with either prefix p or q, if and only if thatannouncement has the IMPORTANT community set.

• A specification can require routers to forward an an-nouncement ao, if and only if ao is equal to the selectedannouncement a∗l .

As shown in the evaluation, even with these restrictions,Bagpipe still provides sufficient expressiveness for manyinteresting policies. This is due to two properties of BGP:

1) A BGP router r selects and forwards announcements fora certain prefix p, completely independent of any announce-ments for any other prefix p′ or any other router r′ (see Fig. 1).For example, the first disallowed policy above cannot be im-plemented, because the decision process for announcementsof prefix p cannot inspect announcements for prefix q.

2) The imp and exp programs can only consider oneannouncement at a time. This restriction is defined by theBGP specification:

[A router’s imp/exp programs] SHALL NOT use anyof the following as its inputs: the existence of otherroutes, the non-existence of other routes, or the pathattributes of other routes. [32]

For example, the second disallowed policy above cannot beimplemented, because an import rule can only consider asingle received announcement at a time.

Instead of interpreting a specification as a network stateinvariant, some (but not all) specifications can also be inter-preted as a the composition of an import specification, anexport specification, and a selection ranking.

An import specification πi restricts how routers can importreceived announcements. Formally, πi(r, p, i, ai, al) : boolholds if and only if for any network trace, πi returns true forany announcement ai which was received by router r fromneighbor i for prefix p, and which was imported as announce-ment al (al does not have to be selected). Note that an importspecification quantifies only over a single announcement, i.e.,an import specification restricts all announcements indepen-dently. An export specification πe restricts how routers canexport selected announcements, and is formalized similarlyto an import specification. Import and export specificationsresemble the domain-specific languages employed by Fre-netic [13], NetCore [30], and NetKat [1].

A selection ranking ≤ restricts which announcementsrouters can select. Formally,≤ (r, p, i, ai, i∗, a∗i ) : bool holdsif and only if for any network trace, router r, prefix p, andneighbor i, the announcement ai received from neighbor i

is ranked lower than the announcement a∗i received fromneighbor i∗ (i∗ is the neighbor from which r has selectedthe announcement). In contrast to much related work, theselection ranking is unique because it considers multipleannouncements simultaneously.

Many specifications τ can be decomposed into an importspecification πi, an export specification πe, and a selectionranking ≤ as follows:τ(v) : spec :=πi(router(v), prefix(v), bestSender(v),

bestReceived(v), selected(v)) ∧πe(router(v), prefix(v), receiver(v),

selected(v), sent(v)) ∧≤ (router(v), prefix(v), sender(v), received(v),

bestSender(v), bestReceived(v))

While some specifications cannot be decomposed intothis form, e.g., if the specification relates received(v) andsent(v), all specifications that we expressed for the evalua-tion of Bagpipe can.

The rest of this section describes examples of usefulpolicies, and shows how they are expressed as specifications.

Block To External Specification Section 2 described theBlockToExternal specification, which prohibits internalrouters of the AS under consideration from forwardingclassified announcements to any external routers. An an-nouncement is considered classified if it had the BTE (blockto external) community set at the time that it was received bythe AS. We can express this specification as:BlockToExternal(v) := receiver(v) ∈ Re →

BTE ∈ communities(original(sent(v)))

An AS’s router configurations correctly implement this spec-ification if and only if specHolds(BlockToExternal). Inlin-ing the definitions of specHolds and BlockToExternal , aswell as removing unused variables, leads to the formula be-low, which states that an AS’s router configurations correctlyimplement BlockToExternal if and only if the most recentlyforwarded announcement ao in any reachable router state ofany internal router r is not classified.∀ (t : trace) (r ∈ Ri) (p ∈ P ) (o ∈ out(r)),

let ao := adjRIBsOut(t, r, p, o)in o ∈ Re → BTE ∈ communities(original(ao))

Note that the BlockToExternal specification is an invarianton an AS’s adjRIBsOut . Further, BlockToExternal can bedecomposed into an export specification πe(r, p, o, al, ao) =o ∈ Re → BTE ∈ communities(original(ao)), and animport specification and selection ranking that always returntrue. We say that BlockToExternal is composed of only anexport specification.

No Martian Specification The BlockToExternal specifi-cation restricts which announcements an AS can forward.The NoMartian specification restricts which announcementsan AS can select.

The NoMartian specification prohibits internal routersfrom selecting a route announcement selected for martian

prefixes prefix , i.e., invalid prefixes such as the private prefix10.0.0.0/8 or the loop-back prefix 127.0.0.0/8 which shouldnot be used to forward packets over the Internet. Formally:

NoMartian(v) := martian(prefix(v))→ selected(v)=notAvailable

The specification holds iff specHolds(NoMartian), i.e.,:

∀ (t : trace) (r ∈ Ri) (p ∈ P ),let al := locRIB(t, r, p)in martian(p)→ al = notAvailable

Note that the NoMartian specification is an invariant on anAS’s locRIB , and is composed of only an import specifica-tion.

Gao-Rexford Specification According to the Gao-Rexfordmodel [14], a widely-used description of AS behavior, thereare three kinds of relationship that an AS can have with anyof its neighbors: customer , peer , or provider . Customerspay the AS to forward packets, peers neither charge nor paymoney to forward packets, and providers charge money toforward packets. To maximize profit, an AS’s routers shouldthus prefer an announcement from (i.e., a route through) acustomer over the announcement from a peer or provider,and should prefer the announcement from a peer over the an-nouncement from a provider. This preference can be capturedwith a relation

r0

r2

r1

e0

128.208.7.0/24 m1

m3

m6 m8

Figure 7. Initial Network Reduction Example.

RIBs to notAvailable (corresponding to the state after line 8in Fig. 1). We have proven this reduction in Coq. The fullformalization of the BGP semantics and the reduction areout of scope for this paper, but can be found in a technicalreport [41]. Here we focus on the reduction and its intuition.

To understand the intuition behind the initial networkreduction, consider an announcement a received by a routerr via some trace t. To be received by r, announcement a hadto be transmitted along some path (a sequence of connectedrouters) through the network. The initial trace t′ of t withrespect to a contains only those events in t that transmit a,but not those events that transmit any other announcements(i.e., t′ transmits a in the initial network without any otherannouncements that could interfere).

For an announcement to be transmitted along a path, allthe routers along the path have to select and forward theannouncement. This means that in trace t: 1) for every routeralong the path the announcement was selected because it wasbetter than all other announcements, and 2) the announcementwas forwarded because it was different from the previouslyforwarded announcement.

If an announcement was transmitted in t along the path ξ,it will also be transmitted in t′ along ξ because: 1) for everyrouter along the path the announcement is selected becausethere are no other announcements that could be better, and 2)the announcement is forwarded because it is different fromthe initial announcement notAvailable stored in all RIBs.

This implies that if Bagpipe verifies that a specificationholds for those announcements that can be transmitted viathe initial trace, then the specification also holds for thoseannouncements that can be transmitted via any trace. It thussuffices to only consider the finite set of initial traces, insteadof the infinite set of all traces. Thus, an AS’s behavior in theinitial network is “maximal” in a sense: if an announcementwill ever be selected or forwarded by a router in any networkstate, it will also be selected or forwarded in the initialnetwork.

To illustrate, consider the full trace t corresponding toFig. 7, where router e0’s data-plane can directly deliverpackets for destinations in 128.208.7.0/24:

1. r1 receives m1 from e0.2. r1 imports m1, resulting in a2, and selects a2.3. r1 exports a2, resulting in m3, and forwards m3 to r0.4. r0 receives m3 from r1.5. r0 imports m3, resulting in a5, and selects a5.

6. r0 receives m6 from e0.7. r0 imports m6, resulting in a7, and selects a7.8. r0 exports a7, resulting in m8, and forwards m8 to r2.9. r2 receives m8 (containing a8).

Black events are in the initial trace t′ of t with respect tothe announcement a8. Gray events are not in the initial trace t′

(these events are also gray in the figure). Subscripted variablesr, e, m, and a correspond to internal routers, external routers,update messages, and announcements respectively.

We focus on the announcement a8 received by routerr2, i.e., adjRIBsIn(t, r2, 128.208.7.0/24, r0). The announce-ment a8 is the result of transmitting the original updatemessage m6 along the path ξ = [e0, r0, r2] via the networktrace t.

It follows that a8 is also the result of transmitting m6along ξ via the initial trace t′ consisting only of the blackevents in trace t. t′ is still legal despite the fact that ro doesnot receive m3 because there is no better announcement thana7 for r0 to select, and the value of r0’s RIBs is notAvailableand thus different from a8

The following paragraphs and Fig. 6 use the above in-sight to precisely explain the initial network reduction whicheliminates specHolds’s quantification over all traces. As men-tioned earlier, we have proven this reduction in Coq, but thefull formalization of the BGP semantics and the reductionare out of scope for this paper. The initial network reductionproceeds in three steps.

1. Rewrite RIBs By the aforementioned insight, it sufficesto only consider the initial traces for received announcementsin the adjRIBsIn , but to verify a specification we have toalso consider the announcements in the adjRIBsOut andlocRIB . This step eliminates specHolds’s dependence onadjRIBsOut and locRIB . This is achieved by rewriting a)the forwarded announcements in the adjRIBsOut in termsof locRIB and b) the selected announcements in locRIB interms of adjRIBsIn . These rewrites are possible because theadjRIBsOut is computed from the locRIB , and the locRIBin turn is computed from the adjRIBsIn , by the algorithmdescribed in Fig. 1.

a) The announcement adjRIBsOut(t, r, p, o) forwardedby router r to some neighbor o is computed by applyingthe export program exp to the announcement locRIB(t, r, p)selected by r. adjRIBsOut(t, r, p, o) is therefore equal toexp(r, o, p, locRIB(t, r, p)).

b) The announcement locRIB(t, r, p) selected by router ris computed by first applying the import program imp to eachannouncement adjRIBsIn(t, r, i, p) received by a neighbori of r, and then selecting the “best” one (as described inFig. 1). Given the neighbor i∗ from which the “best” an-nouncement was selected, locRIB(t, r, p) is therefore equalto imp(r, i∗, p, adjRIBsIn(t, r, i∗, p)).

The function inLocRIB(t, r, p) computes the neighbori∗. inLocRIB(t, r, p) first passes all announcements in the

adjRIBsIn(t, r, i, p) to the imp program, and then deter-mines the neighbor with the “best” imported announcement.

2. Generalize Best Neighbor Note that the inLocRIB func-tion still depends on the trace t. This step eliminates thisdependence.

Consider the announcement a∗l selected by the router r.A router always chooses the announcement a∗l which haspref greater or equal to the pref of any other received andimported announcement al.

Therefore, if Bagpipe verifies that a specification holds forall received and imported announcements that have greater orequal pref than al, no matter from which neighbor they werereceived, the specification also holds for the announcementa∗l received from neighbor i

∗.This step applies this insight by strengthening specHolds

with the fact pref (imp(r, i, p, ai)) ≤ pref (a∗l ), and thengeneralizing i∗ to eliminate the dependence on t.

For those readers familiar with the BGP specification, notethat Bagpipe fully supports tie-breaking on announcementswith equal pref , e.g., tie-breaking on MED, and OSPF cost.This support stems from the fact that a specification musthold for all announcements with equal pref . This means thatBagpipe may falsely claim that a specification does not holdfor a selected announcement a∗l which can actually neverbe selected because a∗l ’s OSPF cost is higher than that ofall other announcements, but will never falsely claim that apolicy holds.

3. Generalize Received Announcements We are finallyready to apply the insight mentioned in the beginning ofthis section. This step replaces specHolds’s use of receivedannouncements in the adjRIBsIn , which requires quantifi-cation over all traces, with any announcements that can betransmitted in the empty network. Note that there are tworeceived announcements ai and a∗i that need consideration.

An announcement a received by router r from neighbori for some prefix p is transmittable in the empty networktransmittable(r, p, i, a) in two ways. Either a is the initialvalue notAvailable stored in r’s adjRIBsIn , or ai is theresult of transmitting some original announcement a0 throughthe initial network along some path ξ : path(i, r) that endsin router i followed by router r.

transmit(ξ, p, a0) computes the announcement that re-sults from forwarding an announcement a0 for prefix p alongsome path ξ in the initial network absent of any other an-nouncements, i.e., it applies the appropriate imp and expprograms of every router along ξ to a0.

Bagpipe exhibited no false positives in our evaluation. Onereason is that modulo pref tie-breaking, the initial networkreduction is complete for specifications that only depend oneither ai or a∗i (but not both), because if either ai or a

∗i can

be received by r via the initial trace, then by definition, thereexists a trace via which r receives either ai or a∗i . Examplesof specifications that depend on only ai or al are NoMartianand BlockToExternal . If a specification depends on both ai

and a∗i , it is possible that no trace exists that forwards both aiand a∗i to r, but we have not observed such cases in practice.

The goal of the initial network reduction was to eliminatespecHolds’s quantification over any infinite sets, specificallythe set of all traces. The initial network reduction achievesthis goal, because INR only quantifies over finite sets. The setof prefixes P is large but finite. The set of announcementsA is large but finite, as the BGP specification restricts themaximal announcement size to 4096 bytes. The set of routersRi, in(r), and out(r) are also finite.

For ASes in a full-mesh configuration, meaning that eachinternal router is directly connected to all other internalrouters4, the set of paths path(i, r) is also finite. This fol-lows from the fact that an internal router r either receivedannouncement ai from some external router i — in whichcase the path is [i, r] — or from an internal router i (otherthan r) which in turn received the announcement from anexternal router re — in which case the path is [re, i, r]. Theset of paths is thus:

path(i, r) := {[i, r] | i ∈ Re} ∪{[re, i, r] | i ∈ Ri \ {r} ∧ re ∈ Re ∧ re ∈ neighbor(i)}

Routing loops inside the AS are impossible, as routers do notforward announcements received from internal neighbors tointernal neighbors.

We refer to the formula resulting from the initial networkreduction as INR(τ). Because INR(τ) only quantifies overfinite sets, it enables automatic verification of specHoldsusing current solvers.

5. Bagpipe ImplementationBagpipe verifies a specification τ by searching for counterex-amples to INR(τ). While this search space is finite, it is stillfar too large to permit naive brute-force search. This sectiondescribes how Bagpipe searches this space efficiently.

5.1 Symbolic Search using RosetteBagpipe uses Rosette [39] to symbolically search for coun-terexamples. Rosette extends the Racket language with sym-bolic values, assertions, and a verify function. (verify e)attempts to assign a concrete value to every symbolic valuein e such that an assertion in e is violated. Rosette imple-ments verify by reducing the search for a failure-inducingassignment to a satisfiability query, which is then dischargedby an off-the-shelf SAT or SMT solver. Once the solver re-turns, its output is automatically lifted to a concrete value foreach symbolic value. These concrete values are then used toprovide counterexamples.

Figure 8 shows the implementation of Bagpipe in Rosette.The core of Bagpipe is a translation of INR(τ) to a Rosetteprogram. The universally-quantified variables in INR(τ) are

4 Some large ASes avoid the performance penalty of a full-mesh configu-ration by using route reflectors, routers that exist to propagate messagesbetween multiple connected components of an AS’s topology. Bagpipe doesnot currently support this optional extension of the BGP specification.

(define (bagpipe τ) (verify (begin(define r (symbolic Ri))(define p (symbolic P))(define i i∗ (symbolic (in r)))(define o (symbolic (out r)))(define ai a∗i (symbolic A))(define al (imp r i p ai))(define a∗l (imp r i

∗ p a∗i ))(define ao (exp r o p a∗l ))(assert

(implies(transmittable r p i ai) (transmittable r p i∗ a∗i )(≤ (pref al) (pref a∗l ))

(τ {router := r; prefix := p; sender := i; received := ai;selected := a∗l ; bestSender := i

∗; bestReceived := a∗i ;receiver := o; sent := ao}))))))

Figure 8. Bagpipe implementation in Rosette. The core of Bagpipe isa translation of INR(τ) to a program (begin ...). This program usessymbolic variables instead of universal quantification. Rosette’s verifyfunction attempts to assign a concrete value to every symbolic value inthe program such that an assertion in the program is violated. If no suchassignment is found, INR(τ) is valid, and the specification τ holds.

translated to symbolic values, which are used as inputs to theassertion that τ holds over all valid pairs of announcements.5

Users of Bagpipe express specifications in Racket asboolean predicates over active announcements. For example,a user would express the NoMartian specification fromSection 3 as follows:

(define NoMartian (v)(implies (martian (prefix v))

(= (selected v) notAvailable)))

In Bagpipe, routers are configured using the imp and expprograms. In practice, routers are configured using router con-figuration languages; most real-world routers use languagesdeveloped by Juniper and Cisco. An interpreter bridges thegap between Bagpipe and real-world configurations; it is aprogram that takes a router configuration and inputs (e.g.,router, prefix, announcement) and returns the result (an an-nouncement) of running the router configuration. Bagpipeincludes interpreters for Juniper and Cisco configurations.These interpreters consist of a parser that generates an AST,and an execution engine that can run an AST given someinputs. Rosette lifts the execution engine to a symbolic exe-cution engine that can run an AST symbolically on all inputs.Bagpipe also infers the network topology of the AS fromrouter configurations.

Bagpipe’s interpreters skip commands unrelated to BGP(e.g., configuration commands for other protocols includingIGMP, MPLS, and ISIS), and ignore low-level BGP config-uration details (e.g., maximum update message TCP packetsize). This could introduce unsoundness (e.g., if an AS op-erator accidentally configured maximum TCP packet sizeto be 0, then all update messages would be dropped). The

5 The translation of transmittable into Rosette is omitted due to spacereasons, but it is straightforward.

interpreters also currently do not handle some BGP-relatedcommands, such as IP broadcasting.

In our experiments, we found that Bagpipe’s currentinterpreters are sufficient to handle hundreds of thousands oflines of industrial BGP configurations. Extending Bagpipe’sinterpreters to support additional BGP-conforming featuresor configuration languages requires no changes in the mainalgorithm, because Bagpipe models import and export rulesas arbitrary functions. Such extensions may however requiresubstantial engineering efforts in the interpreters, as existingconfiguration languages are often proprietary and contain avast number of features.

5.2 Predicate AbstractionThe set of possible announcements is finite, since BGP re-stricts the maximal size of announcements to 4096 bytes.Even when represented symbolically, this is still a largesearch space. Therefore, Bagpipe also implements a formof predicate abstraction [16] by coalescing aspath andcommunities values that induce the same control flow inthe BGP configuration.

Bagpipe exploits the fact that the Juniper and Cisco config-uration languages use regular-expression predicates to branchon announcement attributes. The following example showstwo such predicates contained in the Internet2 configurations:

as-path PRIVATE ".* (64512-65535) .*";community LHCONE-DO-NOT-ANNOUNCE-AS members 65010:*;

PRIVATE matches all AS-paths that contain at least oneAS with an AS number in the range 64512 – 65535.LHCONE-DO-NOT-ANNOUNCE-AS matches all communities whosefirst 16 bits encode the number 65010. The set of predicatesin a configuration is finite and usually fairly small. All con-figurations of Internet2 combined, for example, contain only73 community predicates.

Bagpipe implements predicate abstraction by automat-ically discovering all regular-expression predicates usedin router configurations, and replacing the aspath andcommunities data contained in an announcement with abit-vector that contains a bit for every predicate over aspathor communities . These bit-vectors are represented symboli-cally during Bagpipe’s search for counterexamples.

This approach is sound, but incomplete. Consider forexample a configuration with two predicates: predicate Φmatches every aspath , and predicate φ matches exactlyone specific aspath . Bagpipe would explore a branch onΦ(a) ∧ ¬φ(a) which cannot be executed in reality becausefor every a, Φ(a) implies φ(a). We did not see such falsepositives in our evaluation.

5.3 Parallelization through HoistingBagpipe hoists certain symbolic values out of the programpassed to Rosette’s symbolic search, and enumerates theminstead of representing them symbolically. For example,Bagpipe hoists (symbolic Ri) out of the symbolic execution,

manually enumerating the set Ri instead. Rosette is theninvoked for each enumerated value.

(define (bagpipe τ) (for/each (r Ri)(verify (begin . . . r . . .))))

Bagpipe parallelizes the resulting loop across multiplenodes in a cluster. Bagpipe hoists all variables except for an-nouncements and prefixes, i.e., r, o, i, i∗, and some variablesin transmittable . Bagpipe hoists these sets because their rel-atively small size and minimal structure are not exploitableby the symbolic execution engine.

The number of calls made by Bagpipe to Rosette can becomputed by considering the magnitude of all of the sets thatBagpipe has to enumerate. To verify a policy, Bagpipe has toenumerate every internal router r, all the neighbors to whichr can forward an announcement ao, and all the paths alongwhich the two received announcements ai and a∗i could havebeen transmitted to r.

The number of neighbors to which r can forward an-nouncement ao is |out(r)|. There are three kinds of pathsalong which an announcement could have been transmittedto router r. (1) The announcement was directly transmit-ted from one of r’s external neighbors e to r. There aren(r) = |{e ∈ Re|e ∈ neighbor(r)}| such paths. (2) Theannouncement was transmitted from an external neighborthrough an internal neighbor i to r. There are

∑i∈Ri\{r} n(i)

such paths. (3) The announcement is the value notAvailablewith which the adjRIBsIn for an incoming neighbor wasinitialized. There are |in(r)| such cases. The total number ofcalls Bagpipe makes to Rosette are thus:

∑r∈Ri

|out(r)|

|in(r)|+ n(r) + ∑i∈Ri\{r}

n(i)

2

Assuming that an AS has at least one external neighbor,the asymptotic number of Rosette calls made by Bagpipeis O(|Ri|4|Re|3), where Ri is the set of internal routers andRe is the set of all external neighbors. This stems from thefact that O(|out(r)|) = O(|in(r)|) = O(|Ri| + |Re|), andO(n(r)) = O(|Re|).

5.4 Omitting Unused Specification ArgumentsThere are specifications that do not use all of their arguments;BlockToExternal , for example, uses neither the senderargument nor the received argument (whose value dependson sender ). Enumerating these unused arguments wouldresult in many calls to Rosette which are guaranteed tobe equivalent. To avoid this duplication, Bagpipe does notenumerate certain combinations of unused arguments, therebyreducing the number of calls to Rosette. In practice, wefound that useful specifications use one of the following threecombinations of unused variables (all supported by Bagpipe):

• Bagpipe does not enumerate sender and receiver if argu-ments sender , received , receiver , and sent are unused.

This is the case for all specifications composed of only animport specification (e.g., the NoMartian specification).In this case, Bagpipe calls Rosette O(|Ri|2|Re|) times.

• Bagpipe does not enumerate sender if arguments senderand received are unused. This is the case for all specifica-tions composed of only an export specification (e.g., theBlockToExternal specification). Bagpipe calls RosetteO(|Ri|3|Re|2) times.

• Bagpipe does not enumerate receiver if argumentsreceiver and sent are unused. This is the case for allspecifications composed of only a selection ranking (e.g.,the GaoRexford specification). Bagpipe calls RosetteO(|Ri|3|Re|2) times.

6. EvaluationThis section evaluates Bagpipe by answering the followingquestions. Expressiveness: Can Bagpipe specify policies forcommon configuration scenarios? Efficiency: How long doesBagpipe take to verify specifications? Effectiveness: Howmany bugs does Bagpipe find, and how many false positivedoes Bagpipe produce?

6.1 Juniper TechLibrary ScenariosWe evaluated Bagpipe on 10 configuration scenarios from theJuniper TechLibrary. Specifically, we evaluated Bagpipe on10 scenarios described in the Basic BGP Configuration [4]and Configuring Routing Policies [22] sections of the JuniperTechLibrary, which is the official technical documentation forJuniper products. We used all scenarios from the Basic BGPConfiguration, but did not use 25 scenarios from ConfiguringRouting Policies because: 5 scenarios require extensions oroptional features of the BGP specification RFC 4271 thatare inconsistent with Bagpipe’s model of BGP (specificallyexporting routes that are not selected, and delaying selec-tion of announcements to reduce oscillation), 5 scenariosare unrelated to BGP verification (e.g., logging the numberof forwarded announcements), and 15 scenarios require cur-rently unsupported Juniper features (e.g., policy subroutines)that we believe could be implemented in Bagpipe withoutchanging Bagpipe’s model of BGP.

Each scenario describes a set of AS operator objectives,and it provides router configurations for an entire exampleAS that achieves these objectives. For each scenario, weexpressed a specification and verified it against the scenario’sconfigurations. Bagpipe verified each specification in lessthan one minute.

The following list provides the name of each scenario,along with a description and code size of the specificationthat we expressed and verified (no line is longer than 80characters):

1. Configuring Internal BGP Peering. All internal routersshare their route announcements with all other internalrouters (5 lines).

2. Using AS Path Regular Expressions. Routers block an-nouncements whose AS path matches a given set of regu-lar expressions (4 lines).

3. Disabling Suppression of Route Advertisements. Routeannouncements are sent back to the neighbor from whichthey were received (5 lines).

4. Configuring Policy Chains and Route Filters. The prefixof exported announcements matches the given chainedroute filters (10 lines).

5. Configuring a Conditional Default Route Policy. A defaultroute is exported (3 lines).

6. Configuring Communities in a Routing Policy. Routers setappropriate local preferences according to an announce-ment’s communities (8 lines).

7. Rejecting Known Invalid Routes. Known invalid routesare rejected (3 lines).

8. Configuring External BGP Peering. A router is connectedonly to external routers in a given set (4 lines).

9. Configuring Routing Policy Prefix Lists. The prefix ofexported announcements matches the given prefix lists(12 lines).

10. Using Routing Policy to Set a Preference Value for BGPRoutes. Any local preference set by external neighbors isreplaced with a default value (3 lines).

To ensure Bagpipe detects specification violations, wealso modified each scenario’s configurations to violate itsobjectives. In each case, Bagpipe detected the violation inunder a minute.

Section 3 showed that Bagpipe supports policies found inthe literature, such as the Gao-Rexford model [14] and prefix-based filtering [29]. Section 6.2 shows that Bagpipe can alsoexpress policies inferred from real AS configurations.

6.2 Real AS ConfigurationsWe inferred and verified specifications from the configura-tions of three ASes: the nation-wide ISP Internet2, the re-gional ISP BelWü, and the local ISP Selfnet. These configu-rations total over 240,000 lines of Cisco and Juniper code.

For each inferred specification, Figure 9 summarizesthe time required to verify the specification, the numberof searches that were performed by Rosette (which areperformed in parallel), the arguments passed to the policythat are not used (see Section 5.4), and the number of importand export programs that violate the specification. Bagpipedid not produce false positives on any benchmark.

Timings are on Amazon EC2 with 2 instances of typec3.8xlarge, each with 32 virtual-cores and 60 GB of memory.The experiments ran for a total of 82h, the cost for which isabout $30 using EC2 spot instances.

Nationwide ISP: Internet2 The Internet2 AS connectseducational, research, and government institutions spreadthroughout the US. We have access to the full configura-tion of Internet2’s 10 BGP routers [19]. These routers areconnected to 274 external neighbors. The configurations to-

tal 100,651 lines of Juniper code. We verified four policyspecifications for Internet2, described below.

Internet2’s configurations contain checks to block theimport of announcements with martian prefixes. We thusinferred that it is Internet2’s policy to never import martianprefixes. It takes Bagpipe 1,178s (20min) to verify thatInternet2 correctly implements the NoMartian specification.

Internet2 contains dedicated sanity checks in 237 out ofthe 274 imp programs. After removing these sanity checksfrom the configurations, it takes Bagpipe 1,194s (20min) tocheck the NoMartian specification for Internet2 (indicatedby no checks in Fig. 9). Because Internet2 performs a largevariety of other checks on announcements, the specificationcontinues to hold for 189 neighbors. This result impliesthat Bagpipe’s ability to verify specifications is not onlyuseful to increase confidence in the correctness of routerconfigurations, but also to safely remove unnecessary sanitychecks — 152 in this case.

From Internet2’s configurations, it appears to be Inter-net2’s policy not to forward an announcement to externalrouters if the announcement has the BTE community set. Ittakes Bagpipe 28,594s (8h) to check the BlockToExternalspecification. The configurations of 5 neighbors do not adhereto the BlockToExternal specification. We notified Internet2of these violations, but have not yet received a response.

Internet2 appears to operate according to a refined versionof the Gao-Rexford model discussed in Section 3. We man-ually classified neighbors either as customer or peer usingInternet2’s pricing structure [20] and comments found in theconfigurations. None of Internet2’s neighbors is a provider .We refined the usual GaoRexford specification to support In-ternet2’s advanced policies, such as blocking announcementsfor invalid prefixes and allowing both customers and peersto influence Internet 2’s preference of an announcement bysetting certain communities. For example, a peer can set theHIGH_PEERS community to increase an announcement’s pref-erence. It takes Bagpipe 260,790s (72h) to check the refinedGaoRexford specification. The configurations of 14 neigh-bors do not adhere to the refined GaoRexford specification.We have contacted Internet2 about these violations, but havenot yet received a response.

Regional ISP: BelWü We have access to the BGP relatedconfigurations for three BGP routers6 of the regional ISPBelWü [3]. These routers are connected to 300 externalneighbors. The configurations total 143,657 lines of Ciscocode.

At the time of our experiments, it was BelWü’s policy tomonitor all announcements for martian prefixes by taggingthem with a particular community, and to use this monitoringinformation to evaluate the impact of martian filtering onBelWü’s existing routes. Because of good results, BelWü isplanning to enable strict martian filtering in the near future.

6 Our experiments for these configurations are to demonstrate scaling, andassume that only these three routers are operating in the AS.

AS Policy Time Solver Calls Unused Arguments Violations False PositivesInternet2 (nation-wide) NoMartian 1,178s (20min) 3,114 sender and receiver 0 0Internet2 (nation-wide) NoMartian (no checks) 1,194s (20min) 3,114 sender and receiver N/A 0Internet2 (nation-wide) BlockToExternal 28,594s (8h) 115,330 sender 5 0Internet2 (nation-wide) GaoRexford 260,790s (72h) 971,680 receiver 14 0BelWü (regional) NoMartian 2,106s (35min) 1516 sender and receiver N/A 0BelWü (regional) TagMartian 2,165s (36min) 1516 sender and receiver 0 0BelWü (regional) RemoveOwnASN 1,838s (31min) 1516 sender and receiver 0 0Selfnet (local) StaticExport 2s 9 none 0 0

Figure 9. Real AS Configuration Case Study Results. Solver Calls is the number of calls to Rosette’s solve function. Unused Arguments indicates the argumentspassed to the policy that are not used. Violations is the number of specification violations found. Bagpipe did not issue false positives in any experiment.

Bagpipe took 2,106s (35min) to check the NoMartianspecification and, as expected, revealed that BelWü importsmartian prefixes (from 268 neighbors). We also expressedthe specification TagMartian , which requires BelWü totag all announcements for martian prefixes. Bagpipe took2,165s (36min) to verify that BelWü correctly implementsTagMartian .

It is BelWü’s policy to remove, from all received an-nouncements, communities that contain the ISP’s own ASnumber. We call this policy RemoveOwnASN . It takes Bag-pipe 1,838s (31min) to verify that BelWü correctly imple-ments RemoveOwnASN .

The configurations did not indicate BelWü’s Gao-Rexfordrelationships (customers, peers, and providers), and we didthus not verify a GaoRexford policy. Verifying such a policywould require 619,258 solver calls.

Local ISP: Selfnet We also analyzed the 66 lines of Juniperconfiguration for the sole BGP router of the local ISP Self-net [34]. This router has only a single BGP neighbor, and theISP’s policy is to only export announcements with the pre-fixes that it actually owns. We call this policy StaticExport .Bagpipe took 2s to verify that Selfnet correctly implementsStaticExport .

6.3 Potential False PositivesBagpipe has three potential sources of false positives:

1) When two announcements are tied on local preference,BGP uses complex rules to choose between them. Bagpipeconservatively and soundly models tie breaking as non-deterministic choice, rather than completely modeling thedetails of lower-level protocols like OSPF that these rules use(Section 4).

2) Bagpipe uses predicate abstraction to represent an-nouncements. Removing this source of incompleteness woulddramatically increase the size of Bagpipe’s search space (Sec-tion 5.2).

3) The initial network reduction forwards announcementsin the initial network, which is sound but could lead to falsepositives (e.g., when two announcements can be individuallyforwarded in the initial network but interfere with each otherin reality). We do not know how to eliminate this source

of incompleteness while keeping the search space finite(Section 4).

We found, via manual inspection, that none of the counterexamples returned by Bagpipe during our evaluation weredue to false positives.

7. Related WorkIn this section, we address related work in network config-uration checking. We also briefly discuss software-definednetworking and prior work on SMT-based tools.

Network Analysis rcc [11] is a tool to find bugs in BGPconfigurations, which has been adopted by many AS admin-istrators. It attempts to find violations of route validity andpath visibility by inferring inter-AS relationships from theconfiguration itself (the input to the tool is a set of configura-tions from all of the routers in an AS). Unlike Bagpipe, rccdoes not provide strong guarantees about the checked config-urations; there are both false positives and false negatives inthe configuration errors flagged by the tool.

Batfish [12] is a Datalog-based network configurationanalysis tool. Router configurations, topology descriptions,and a particular set of received BGP announcements aretranslated into Datalog facts which are processed by Datalogrules to generate routing-tables. Z3 is then used to verify first-order properties over the generated routing-tables. Bagpipeand Batfish make different design decisions, and are thus ableto verify different properties. Bagpipe verifies a restrictedset of routing-table invariants (described in Section 3.2) withrespect to any set of received announcements, whereas Batfishverifies arbitrary first-order logic formulas over routing-tableswith respect to a particular set of BGP announcements.Great care has been taken in Bagpipe that all invariantscan be translated to SAT formulas which can be decidedin exponential time, whereas Batfish’s formulas are generallyundecidable.

Header Space Analysis (HSA) [23] verifies a data plane’spacket forwarding behavior in a similar way to Bagpipe(which verifies a control plane’s announcement forwardingbehavior). Contrary to Bagpipe, HSA uses a custom symbolicsearch algorithm instead of an SMT solver; and HSA verifiesspecifications that restrict the forwarding of packets inde-pendent of any other packets in the network (like Bagpipe’s

import/export specifications), but cannot verify specificationsthat restrict the forwarding of packets depending on otherpackets in the network (like Bagpipe’s selection rankings).

BGP Simulation C-BGP [31] is a BGP simulator. Given atopology and a set of configurations, it determines how trafficwill be routed. AS operators can use it both for debuggingexisting problems and for testing potential new configurations.C-BGP and Bagpipe are potentially complementary; an ASoperator could test configurations using C-BGP and thenverify them using Bagpipe to guarantee that the networkis configured to correctly handle any set of received pathannouncements.

SDN Software defined networking is a new paradigm forlocal networks in which router configuration is controlled bya single program running on a master router. There has beena large amount of work on verifying the behavior of software-defined networks, including language support [30, 1], model-checking [2, 9], and full formal verification [17]. SDN hasthus far not been used to control BGP-speaking borderrouters, but even if current BGP configuration languagesare supplanted by SDN, tools like Bagpipe will still be usefulto ensure that configurations respect AS policies.

SMT-Based Tools SAT and SMT solvers have been appliedin a wide range of automated tools for bug finding [7, 6, 10],verification [25, 26, 37], program synthesis [36, 24], andfault localization [21]. Bagpipe builds on Rosette [38], aprogramming language designed for easy creation of suchtools. In particular, Rosette is equipped with a symboliccompiler that can efficiently reduce a verification, synthesis,or fault localization query about all bounded executions ofa program to an SMT formula. Because the initial networkreduction enables Bagpipe to treat BGP configurations asfinite programs, we can use Rosette’s bounded reasoningfacilities for sound and scalable verification of BGP policies.

8. ConclusionWe presented Bagpipe, a tool to automatically verify thatrouter configurations correctly implement AS operators’ BGPpolicies. To make this verification possible, we introduced theinitial network reduction, which reduces verification of BGPpolicies from checking an infinite set of traces to checking afinite set of initial traces, thus enabling Bagpipe to use currentconstraint solvers effectively. Building on this reduction,the Bagpipe implementation additionally employs a novelcombination of search space partitioning, unused variableomission, symbolic execution, and predicate abstraction. Weevaluated Bagpipe on 10 configuration scenarios from theJuniper TechLibrary and three ASes with a total of over240,000 lines of configuration, finding that Bagpipe will scaleto the complexity and scope of real-world AS configurations.

Future work will extend Bagpipe to support additionalBGP features (e.g., route reflectors), incrementalize verifica-tion for configuration updates, and use Rosette’s synthesis

features to automatically generate configurations that cor-rectly implement policies.

AcknowledgmentsWe thank Tim Kleefass and Sebastian Neuner from BelWü,as well as Jann Haber, Hannes Rist, and Christoph Wurmfrom Selfnet for sharing their BGP configurations and pro-viding valuable feedback. We also thank the reviewers fortheir insightful comments. This material is based upon worksupported by the National Science Foundation GraduateResearch Fellowship under Grant No. DGE-1256082. Thismaterial is based on research sponsored by DARPA underagreement numbers FA8750-12-2-0107, FA8750-12-C-0174,FA8750-15-C-0010, and FA8750-16-2-0032. The U.S. Gov-ernment is authorized to reproduce and distribute reprintsfor Governmental purposes notwithstanding any copyrightnotation thereon.

References[1] C. J. Anderson et al. “NetKAT: Semantic Foundations

for Networks”. In: POPL. 2014.[2] T. Ball et al. “VeriCon: Towards Verifying Controller

Programs in Software-defined Networks”. In: PLDI.2014.

[3] BelWü. https://www.belwue.de/.[4] BGP Feature Guide for the OCX Series. 2015.[5] M. Brown. Pakistan hijacks YouTube. http://research.

dyn.com/2008/02/pakistan-hijacks-youtube-1/. 2008.[6] C. Cadar, D. Dunbar, and D. Engler. “KLEE: unassisted

and automatic generation of high-coverage tests forcomplex systems programs”. In: OSDI. 2008.

[7] E. Clarke, D. Kroening, and F. Lerda. “A Tool forChecking ANSI-C Programs”. In: TACAS. 2004.

[8] J. Cowie. China’s 18-Minute Mystery. http://research.dyn.com/2010/11/chinas-18-minute-mystery/. 2010.

[9] M. Dobrescu and K. Argyraki. “Software DataplaneVerification”. In: NSDI. 2014.

[10] J. Dolby, M. Vaziri, and F. Tip. “Finding bugs effi-ciently with a SAT solver”. In: FSE. 2007.

[11] N. Feamster and H. Balakrishnan. “Detecting BGPConfiguration Faults with Static Analysis”. In: NSDI.2005.

[12] A. Fogel et al. “A General Approach to NetworkConfiguration Analysis”. In: NSDI. 2015.

[13] N. Foster et al. “Frenetic: A Network ProgrammingLanguage”. In: ICFP. 2011.

[14] L. Gao and J. Rexford. “Stable Internet Routing With-out Global Coordination”. In: SIGMETRICS. 2000.

[15] S. Goldberg. “Why Is It Taking So Long to SecureInternet Routing?” In: Queue (2014).

[16] S. Graf and H. Saïdi. “Construction of Abstract StateGraphs with PVS”. In: CAV. 1997.

[17] A. Guha, M. Reitblatt, and N. Foster. “Machine-verifiedNetwork Controllers”. In: PLDI. 2013.

[18] International Telecommunication Union Statistics.2014.

[19] Internet2 Configurations. http://vn.grnoc.iu.edu/Internet2/configs/configs.html.

[20] Internet2 Fees. http : / / www. internet2 . edu / about - us /membership/.

[21] M. Jose and R. Majumdar. “Bug-Assist: assisting faultlocalization in ANSI-C programs”. In: CAV. 2011.

[22] Junos OS: Routing Policies, Firewall Filters, and TrafficPolicers Feature Guide for Routing Devices. 2016.

[23] P. Kazemian, G. Varghese, and N. McKeown. “HeaderSpace Analysis: Static Checking for Networks”. In:Proceedings of the 9th USENIX Conference on Net-worked Systems Design and Implementation. 2012.

[24] A. S. Koksal et al. “Synthesis of Biological Modelsfrom Mutation Experiments”. In: POPL. 2013.

[25] K. R. M. Leino. “Dafny: An Automatic Program Veri-fier for Functional Correctness”. In: LPAR. 2010.

[26] K. R. M. Leino. This is Boogie 2. Tech. rep. 2008.[27] D. Madory. Chinese Routing Errors Redirect Russian

Traffic. http://research.dyn.com/2014/11/chinese-routing-errors-redirect-russian-traffic/. 2014.

[28] D. McConnell. Chinese company ‘hijacked’ U.S. webtraffic. http: / /www.cnn.com/2010/US/11/17/websites .chinese.servers/. 2010.

[29] D. Meyer, J. Schmitz, and C. Alaettinoglu. Applicationof Routing Policy Specification Language (RPSL) onthe Internet. 1997.

[30] C. Monsanto et al. “A Compiler and Run-time Systemfor Network Programming Languages”. In: POPL.2012.

[31] B. Quoitin and S. Uhlig. “Modeling the Routing of anAutonomous System with C-BGP”. In: IEEE Network(2005).

[32] Y. Rekhter, T. Li, and S. Hares. A Border GatewayProtocol 4 (BGP-4). RFC 4271. 2006.

[33] L. Schaefer. Deutsche Telekom: ’Internet data madein Germany should stay in Germany’. http://www.dw.com/en/deutsche-telekom-internet-data-made-in-germany-should-stay-in-germany/a-17165891. 2013.

[34] Selfnet. https://selfnet.de/.[35] D. Slane. 2010 Report to Congress of the U.S.–China

Economic and Security Review Commission. 2010.[36] A. Solar-Lezama et al. “Combinatorial Sketching for

Finite Programs”. In: ASPLOS. 2006.[37] P. Suter, A. S. Köksal, and V. Kuncak. “Satisfiability

modulo recursive programs”. In: SAS. 2011.[38] E. Torlak and R. Bodik. “A Lightweight Symbolic

Virtual Machine for Solver-aided Host Languages”. In:PLDI. 2014.

[39] E. Torlak and R. Bodik. “Growing Solver-aided Lan-guages with Rosette”. In: Onward! 2013.

[40] D. Turner et al. “California Fault Lines: Understand-ing the Causes and Impact of Network Failures”. In:SIGCOMM. 2010.

[41] K. Weitz et al. Bagpipe: Verified BGP ConfigurationChecking. Tech. rep. 2016.