OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing...

10
OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad Agarwal Chen-Nee Chuah Randy H. Katz Computer Science Division Dept. of Electrical & Computer Engineering Computer Science Division University of California, Berkeley University of California, Davis University of California, Berkeley [email protected] [email protected] [email protected] Abstract— An increasing number of ASes have been connecting to the Internet through the BGP inter-domain routing protocol. With increas- ing stress on the scale of this system and increasing reliance on Internet connectivity, more participants demand additional functionality from inter- domain routing that BGP cannot handle. For example, we believe that the recent trend towards multihomed stub networks exhibits a likely intent to achieve fault tolerant and load balanced connectivity to the Internet. How- ever, BGP today offers route fail-over times as long as 15 minutes, and very limited control over incoming traffic across multiple wide area paths. More research literature and news media are calling for stemming malicious or erroneous routing announcements. We propose a policy control architec- ture, OPCA, that runs as an overlay network on top of BGP. OPCA allows an AS to make route change requests at other, remote ASes to achieve faster route fail-over and provide capabilities to control traffic entering the local AS. The proposed architecture and protocol will co-exist and interact with the existing routing infrastructure and will allow for a scalable rollout of the protocol. I. I NTRODUCTION A. Trends in Inter-Domain Routing The Border Gateway Protocol (BGP) [1] is the de-facto inter- domain routing protocol between Autonomous Systems (ASes) that achieves global connectivity while shielding intra-domain routing details and fluctuations from the external view. Recent studies of BGP [2], [3] have indicated a significant growth in BGP routing tables, an increase in route flapping and unneces- sarily specific route announcements. The large growth in the number of ASes that participate in BGP peering sessions has been fueled by stub ASes. Our analysis of the BGP data from Routeviews [4] reveals that at least 60% of these stub ASes are multi-homed to two or more providers, i.e., they announce BGP routes via multiple upstream ASes. This trend towards increased connectivity to the Internet and participation in inter-domain routing is placing more stress on the BGP infrastructure. The scale of routing is increasing, and more features are being expected out of it than BGP was de- signed to handle. For instance, this increasing trend towards multi-homing is intended as a solution to achieve two goals: fault tolerance and load balancing on inter-domain routes. B. Features Absent in BGP As an illustration, Figure 1 compares two scenarios where a stub AS is (a) single-homed and (b) multi-homed to three providers. The stub AS, W, in Figure 1(b) can choose to have its traffic go primarily through ISP X. If the link to ISP X fails, Sharad Agarwal and Chen-Nee Chuah are also affiliated with the IP & In- terworking Division of the Sprint Advanced Technology Laboratories. This re- search was supported by Cisco Systems and the California MICRO Program, with matching support provided by Ericsson, Nokia, and Sprint. or when there are failures along the path through ISP X, W can failover to ISP Y or Z. If it were singly homed, as in case (a), it could only protect itself against upstream link failures by pur- chasing multiple redundant links to ISP X. In addition, W can load-balance its outgoing traffic by selecting the best route to the destinations via one of the three providers. Routescience [5] and others automate outgoing traffic balancing by selecting specific BGP announcements heard from different providers. Achieving connectivity by subscribing to multiple providers is likely to be expensive, but Mortimer’s study [6] suggests that reliability is a deciding factor. However, the effectiveness of multi-homing is limited by the slow convergence behavior of BGP. Inter-domain routes can take upto 15 minutes [7] to fail- over in the worst case. For companies that rely on Internet con- nectivity to conduct online transactions, such a long outage can have a severe financial impact. Furthermore, BGP allows an AS little control over how the incoming traffic enters its network. As more networks connect to the Internet via BGP, the like- lihood of router misconfiguration will increase. With more par- ticipants, the chance of having a malicious or compromised par- ticipant grows. As has been observed in the past [8], a single incorrect routing announcement can seriously impact data traf- fic. As yet, no protocol exists for detecting and stemming such bogus route announcements. As more and more applications are enabled on the Internet, traffic will grow and may become more unpredictable. Higher traffic use may incur higher transit costs for stub networks that rely on ISPs for Internet service. During periods of abnormal traffic patterns or even link failures, it may be advantageous for two entities that have a business relationship for exchang- ing traffic to temporarily modify this agreement to improve con- gestion or reduce transit costs. As yet, no protocol exists for negotiating and applying such temporary agreements. C. Solution Instead of overloading BGP with protocol extensions, we pro- pose to address these problems by developing an Overlay Policy Control Architecture (OPCA) running on top of BGP to facili- tate policy exchanges. Our architecture relies on knowing AS relationships and the AS level hierarchy. While OPCA can be used to address the shortcomings of the current infrastructure, we focus on two main goals: to support fast, fine grained management of incoming traffic across multiple incoming paths, and to reduce the fail-over time of inter-domain paths.

Transcript of OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing...

Page 1: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 1

OPCA: Robust Interdomain Policy Routing andTraffic Control

Sharad Agarwal† Chen-Nee Chuah† Randy H. KatzComputer Science Division Dept. of Electrical & Computer Engineering Computer Science Division

University of California, Berkeley University of California, Davis University of California, [email protected] [email protected] [email protected]

Abstract— An increasing number of ASes have been connecting to theInternet through the BGP inter-domain routing protocol. With increas-ing stress on the scale of this system and increasing reliance on Internetconnectivity, more participants demand additional functionality from inter-domain routing that BGP cannot handle. For example, we believe that therecent trend towards multihomed stub networks exhibits a likely intent toachieve fault tolerant and load balanced connectivity to the Internet. How-ever, BGP today offers route fail-over times as long as 15 minutes, and verylimited control over incoming traffic across multiple wide area paths. Moreresearch literature and news media are calling for stemming malicious orerroneous routing announcements. We propose a policy control architec-ture, OPCA, that runs as an overlay network on top of BGP. OPCA allowsan AS to make route change requests at other, remote ASes to achieve fasterroute fail-over and provide capabilities to control traffic entering the localAS. The proposed architecture and protocol will co-exist and interact withthe existing routing infrastructure and will allow for a scalable rollout ofthe protocol.

I. INTRODUCTION

A. Trends in Inter-Domain Routing

The Border Gateway Protocol (BGP) [1] is the de-facto inter-domain routing protocol between Autonomous Systems (ASes)that achieves global connectivity while shielding intra-domainrouting details and fluctuations from the external view. Recentstudies of BGP [2], [3] have indicated a significant growth inBGP routing tables, an increase in route flapping and unneces-sarily specific route announcements. The large growth in thenumber of ASes that participate in BGP peering sessions hasbeen fueled by stub ASes. Our analysis of the BGP data fromRouteviews [4] reveals that at least 60% of these stub ASes aremulti-homed to two or more providers, i.e., they announce BGProutes via multiple upstream ASes.

This trend towards increased connectivity to the Internet andparticipation in inter-domain routing is placing more stress onthe BGP infrastructure. The scale of routing is increasing, andmore features are being expected out of it than BGP was de-signed to handle. For instance, this increasing trend towardsmulti-homing is intended as a solution to achieve two goals:fault tolerance and load balancing on inter-domain routes.

B. Features Absent in BGP

As an illustration, Figure 1 compares two scenarios wherea stub AS is (a) single-homed and (b) multi-homed to threeproviders. The stub AS, W, in Figure 1(b) can choose to haveits traffic go primarily through ISP X. If the link to ISP X fails,

†Sharad Agarwal and Chen-Nee Chuah are also affiliated with the IP & In-terworking Division of the Sprint Advanced Technology Laboratories. This re-search was supported by Cisco Systems and the California MICRO Program,with matching support provided by Ericsson, Nokia, and Sprint.

or when there are failures along the path through ISP X, W canfailover to ISP Y or Z. If it were singly homed, as in case (a),it could only protect itself against upstream link failures by pur-chasing multiple redundant links to ISP X. In addition, W canload-balance its outgoing traffic by selecting the best route to thedestinations via one of the three providers. Routescience [5] andothers automate outgoing traffic balancing by selecting specificBGP announcements heard from different providers.

Achieving connectivity by subscribing to multiple providersis likely to be expensive, but Mortimer’s study [6] suggests thatreliability is a deciding factor. However, the effectiveness ofmulti-homing is limited by the slow convergence behavior ofBGP. Inter-domain routes can take upto 15 minutes [7] to fail-over in the worst case. For companies that rely on Internet con-nectivity to conduct online transactions, such a long outage canhave a severe financial impact. Furthermore, BGP allows an ASlittle control over how the incoming traffic enters its network.

As more networks connect to the Internet via BGP, the like-lihood of router misconfiguration will increase. With more par-ticipants, the chance of having a malicious or compromised par-ticipant grows. As has been observed in the past [8], a singleincorrect routing announcement can seriously impact data traf-fic. As yet, no protocol exists for detecting and stemming suchbogus route announcements.

As more and more applications are enabled on the Internet,traffic will grow and may become more unpredictable. Highertraffic use may incur higher transit costs for stub networks thatrely on ISPs for Internet service. During periods of abnormaltraffic patterns or even link failures, it may be advantageousfor two entities that have a business relationship for exchang-ing traffic to temporarily modify this agreement to improve con-gestion or reduce transit costs. As yet, no protocol exists fornegotiating and applying such temporary agreements.

C. Solution

Instead of overloading BGP with protocol extensions, we pro-pose to address these problems by developing an Overlay PolicyControl Architecture (OPCA) running on top of BGP to facili-tate policy exchanges. Our architecture relies on knowing ASrelationships and the AS level hierarchy. While OPCA can beused to address the shortcomings of the current infrastructure,we focus on two main goals:

• to support fast, fine grained management of incoming trafficacross multiple incoming paths, and• to reduce the fail-over time of inter-domain paths.

Page 2: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 2

Together, these goals serve to improve routing and traffic in thecurrent inter-domain routing structure, and allow it to scale bet-ter to the growing number of multi-homed ASes.

OPCA consists of a set of intelligent Policy Agents (PAs) thatcan be incrementally deployed in all the participating AS do-mains. The PAs are responsible for processing external policyannouncements or route-change requests while adhering to lo-cal AS policies, and enforcing necessary changes to local BGProuters. These PAs communicate with one another via a newOverlay Policy Protocol (OPP). Such an overlay architecture al-lows ASes to negotiate the selection of inter-domain paths forincoming traffic with remote ASes, leading to more predictableload-balancing performance. In addition, an AS can requestrouting changes to other ASes to expedite fail-over. Our archi-tecture does not require any modifications to the BGP protocolor to existing BGP routers.

We will review related work in the next section. Section IIIdescribes the design of our architecture. In Section IV, we ex-plain the rationale behind our design of OPCA. We follow thiswith a description of the applications of our architecture in Sec-tion V. We end with some deployment and scaling issues inSection VI and conclusions in Section VII.

Internet

X

W W

Internet

YXZ

(a) (b)

Fig. 1. (a) Company W with ISP X. (b) Company W with ISP’s X, Y and Z

II. RELATED WORK

Huston [9] suggests ways to address the problems of load bal-ancing and route fail-over. There are two main approaches: (a)extending the current BGP protocol/implementations or (b) usealternate routing through overlay networks or replace BGP witha new interdomain routing protocol.

Various BGP-based solutions have proposed to limit the ad-vertisement scope of route announcements [10], [11], [12]. Forexample, BGP can be modified to allow bundling of routes or tospecify aggregation scopes. These proposals may limit the ill-effects of multi-homing but do not solve the issues of fast fail-over and inbound load balancing that we are concerned with.Pei [13] proposes modifications to BGP to reduce convergencetime by identifying which withdrawals are due to actual failuresand filtering out announcements of invalid or conflicting pathsduring the transient periods. He employs a new community at-tribute to convey routing policy information over the traditional

BGP session. However, it does not address the issue of wide-area traffic balancing. These schemes require a new versionof BGP to be deployed or many BGP routers to be reconfig-ured. This is difficult to accomplish given the widespread use ofBGP. Mortier [14] proposes a new route attribute that expressesa price for carrying traffic on the advertised route that one neigh-bor charges another. While this scheme adds another interestingattribute that can be used in route selection, it does not addressthe goals of our work in reducing route failover times and man-aging incoming traffic.

All the afore-mentioned approaches rely on “in-band” signal-ing, i.e., they embed and distribute policy information in BGProuting messages. In this paper, we explore an orthogonal ap-proach by introducing “out-of-band” signaling through OPCAto permit more flexibility and control of the policy distributionsand negotiations between AS domains. This results in more pre-dictable performance for inbound traffic engineering. In fact,OPCA could potentially leverage these other BGP modificationsto obtain accurate connectivity information in a more efficientmanner, and based on this, make better policy decisions.

In the early days of BGP, Estrin [15] proposed a centralizedrouting arbiter that collects all routing entries, centrally com-putes the “best” routes, and re-distributes the final routing en-tries. We believe that such an architecture is not deployed inthe Internet today due to its complexity and scalability issues.Alternative routing architectures have been proposed, such asRON [16], Nimrod [17], and BANANAS [18]. RON is an over-lay network that uses active probing and global link state ina fully meshed network to customize routing between overlaynodes. RON is designed for applications with a small number ofparticipating nodes and cannot scale to the number of ASes thatexist today. Nimrod [17] was proposed in 1996 as an alterna-tive inter-domain routing architecture. It would distribute link-state information and support three forms of routing: MPLS-like flow routing, BGP-like hop by hop routing and data packetspecified routing. BANANAS [18] also distributes link state in-formation and allows a sender to specify the full path for eachpacket. Although sender specified routing can help achieve loadbalancing and fault tolerance, packets will no longer go throughthe fast path in routers (such as Cisco Express Forwarding) andwill require more processing and hence delay. Nimrod and BA-NANAS also introduce the difficulty of timely link-state propa-gation which has not yet been addressed. These solutions pro-pose to change the underlying routing protocol itself to routetraffic either on optimal or source-dictated paths. Our approachreuses the existing BGP routing protocol and builds an overlaylayer to achieve explicit control over policy distributions andload balancing.

Frameworks and protocols for distributing policies within adomain that are based on MPLS, DiffServ or IntServ have beenproposed, e.g., COPS and Bandwidth Broker [19], [20], [21],[22]. MPLS Fast-Reroute maintains backup paths and switch-ing entries for every link computed from link state informationflooded through the local network. In our solution, we focuson the inter-domain case and do not rely on the widespread de-ployment of DiffServ or MPLS, but instead rely on the alreadywidespread deployment of BGP.

Page 3: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 3

PADirectory

PA

MI PD

PA

PA

MI PD

AS W

AS X

AS Y

AS Z

RMAP

Other AS domains/networks

in the Internet

E-BGP

EBGP E-BGP

E-BGP speaker

E-BGPsessions

Fig. 2. Overlay Policy Control Architecture

III. OVERLAY POLICY CONTROL ARCHITECTURE (OPCA)

A. Overview

The Overlay Policy Control Architecture (OPCA) is designedto support fault-tolerant and efficient wide-area routing. Ourapproach leverages intra- and inter-AS measurement and mon-itoring systems that are commonly deployed in ASes to obtainsimple data on traffic load changes and network performance.As shown in Figure 2, OPCA consists of five components: Pol-icy Agents (PAs), Policy Databases (PDs), Measurement Infras-tructures (MIs), the PA directory and the AS Topology and Re-lationship Mapper (RMAP). Policy Agents (PAs) are intelligentproxies that reside in each AS domain that agrees to participatein the overlay policy distribution network. Most Internet Ser-vice Providers (ISPs) today have deployed a measurement in-frastructure to monitor the performance of their network. Theyalso maintain a database of service level agreements (SLAs) andpeering arrangements. We assume that OPCA can reuse any ofsuch existing PDs and MIs within an AS domain. The PA direc-tory and RMAP are new query-services introduced by OPCA.The next section describes each of these components in detail.We have completed the design of the PA protocol and the RMAPcomponent. We will reuse existing MIs and use the already de-ployed DNS for the PA directory. We are currently implement-ing the PA and PD and designing the evaluation methodology.

B. Components of OPCA

B.1 Policy Agent (PA)

Intelligent Policy Agent (PAs) are introduced in each AS do-main that employs OPCA to form an overlay policy distribu-tion network. The PAs are responsible for processing externalpolicy announcements, processing local AS policies, and en-forcing these policies at border routers participating in externalBGP sessions. This implies that PAs should have administra-tive control over the E-BGP speakers within their respective ASdomains. The E-BGPs are dynamically (re)configured to reflectpolicy changes, and continue to perform route selection based

on these policies. However, some form of synchronization maybe needed with PAs to prevent simultaneous conflicting changesfrom network operators. A centralized or distributed PA direc-tory is needed to allow distant PAs (PAs belonging to differentASes) to communicate with each other. Each PA should be ac-cessible at an IP address and port that can be reached from dis-tant ASes.

A routing policy will be influenced by traffic measurementsbetween the local AS and one or more distant ASes that areimportant, such as application level customers. To impose achange on the routing between these sites, a local PA will haveto negotiate with the remote PA, and possibly other intermedi-ate policy agents. Contacting the appropriate intermediate PAswill be important, since conflicting routing and filtering policiesin other ASes along the route can severely impact the routingchange. Having an understanding of the relationships betweenASes along prospective routes [23] will be important to ensurethe effectiveness of protocol.

The protocol that the PAs use to communicate with one an-other is the new overlay policy protocol (OPP). The design ofPAs and the OPP protocol is subject to certain constraints:• The PAs should communicate with BGP speakers via conven-tional means, and should not require any modifications to therouters. This is important for the acceptability and deploymentof this protocol.• The PAs should detect and identify policy conflicts at run-time, and avoid potential BGP oscillations or divergence.• OPP should co-exist with the widely deployed IGP/EGP to-day such as BGP, IBGP, IS-IS or OSPF.• The use of OPP should not increase BGP route flapping andthe number of routing table entries.• The correctness and effectiveness of OPCA should not relyon every AS employing PAs. We strive to support incrementaldeployment, i.e., early adopters of the new PAs should not be ata disadvantage compared to those continuing to use only BGP,even though the utility of the new system may increase as moreASes subscribe to OPCA.

B.2 Policy Database (PD)

The policy database is a local repository of information thatwill help the local PA decide how to change routing for its do-main. The PD should provide the following data:• Ordered list of remote ASes containing the local domain’smain customers. This list identifies the target ASes that the PAshould focus its load balancing efforts at. This list can easilybe formed by examining the logs of the service provided by thelocal domain [24].• List of local application servers. The PA needs to know whichset of IP addresses serve content or any other service to the re-mote customers. The majority of traffic will likely be destinedfor or originate from these local servers. The PA will be con-cerned with the load balancing and fail-over of routes to theseaddresses.• Pricing constraints and SLAs. The PA will try to balance traf-fic across multiple ISP links evenly weighted by actual link ca-pacity. However, the local domain may prefer to weight trafficby pricing structures imposed by the SLAs it is operating under.If this is the case, the PA will need to be informed about these

Page 4: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 4

price constraints.

B.3 Measurement Infrastructure (MI)

Most ISPs and customer bases already employ some form of ameasurement infrastructure (MI). Some may use it to verify Ser-vice Level Agreement (SLA) specifications with a third party, ormay use it to manage their network. In our architecture, we as-sume that such an infrastructure already exists in each domainemploying OPCA. The MI helps the PA keep track of the effectsof the PAs alterations to routes and decide when such an alter-ation is necessary. The MI should provide the following data:• E-BGP link characteristics. The PA needs data on each of thelinks connecting the local AS to the Internet. This data shouldinclude actual link capacity and current bandwidth load. Thisallows the PA to decide which links are underutilized and toverify the effect of policy changes that it imposes. PA controltraffic will also go over these links. Simple SNMP statistics oreven periodic polls of the Cisco IOS interface byte counters willsuffice.• Customer-server traffic characterization. The MI shouldalso provide data outlining characteristics of traffic (such astotal bandwidth, average latency) that each main customersends/receives to/from each local server. This helps the PA un-derstand the current traffic distribution and how to best redis-tribute this load. Tools such as Cisco Netflow should be able toprovide such data.

B.4 PA Directory

The PA directory is a means by which a local domain’s PAcan locate the address for a distant AS’s PA. This is necessarybecause some ASes may not have PAs, since we do not rely onimmediate wide scale deployment of OPCA, and those that dohave PAs can place them anywhere inside their network. Thedesign of the directory is not a goal of our work. The systemcan use one or multiple directories, which may or may not becoherent with each other. From the architecture’s point of view,there is logically one PA directory. A simple solution such asusing the already deployed DNS can be used. The PA directory’sfully qualified domain name is known to all PAs and PAs needonly make DNS queries to locate other PAs.

B.5 AS Topology & Relationship Mapper (RMAP)

The RMAP is a repository of the inter-AS relationships andInternet hierarchy. These relationships determine how routingand traffic flows on the Internet as governed by route exportrules. Export rules dictate that when sending routes to a cus-tomer AS, a provider has to send it all the routes it knows of.When sending routes to a provider or peer, an AS does notsend other peer or provider routes. In the RMAP, we deducethese relationships from multiple BGP routing tables by apply-ing heuristics. Our heuristic begins by ranking the ASes basedon their distance in the AS paths from the AS where we col-lect each routing table. We then infer whether an AS-AS linkrepresents a peer-peer or a provider-customer relationship basedon the relative ranks of each AS pair that shares a link. Us-ing pruning, greedy ordering and weak cuts designed to exposethe different business classes of Internet service providers, we

also infer the hierarchical structure of the AS topology that ex-ists today. Details of our algorithm can be found in our priorwork [23].

The RMAP is an essential component to OPCA. The architec-ture is indifferent to whether there is only one global RMAP or ifmultiple RMAPs exist for different ASes. The only requirementis that the RMAP have access to enough diverse BGP routingtable dumps as to be mostly accurate. We will revisit this issuein Section IV. PAs need the RMAP to find the likely path a dis-tant AS uses to reach it and the intermediate AS relationshipsthat determine how routes propagate to the distant AS. This alsohelps to determine if a policy request will conflict with a distantAS’s local routing policy.

C. Overlay Policy Protocol

C.1 Message Propagation

OPP carries inter-PA messages from the source PA to the des-tination PA over UDP by rendezvousing first through the PA di-rectory. Therefore, PA control messages do not have to throughevery PA in the intermediate ASes the same way as BGP an-nouncements propagate from one BGP speaker to another, be-cause the originating PA has the burden of determining the ap-propriate destination PA to contact directly. The advantages ofthis approach are• eliminate intermediate PA processing• keep convergence time of PA messages lower than that ofBGP messages• give originating PA more control• give more privacy to originating PAs

C.2 Connections and Sessions

We choose unreliable, connectionless UDP over reliable,connection-based TCP because the reverse route may be un-available during link failure. Consider the case in Figure 5where X uses the path X → F → C → A to reach A. Supposethe C → A link fails and A wants X to failover to the F → E →

B → A path. A’s PA has to contact F’s PA to make the routingchange. A already has an alternate path to F via B and can useit to send the message to F. However, until F receives the PA re-quest and implements it, it will not have a working return path toA to send a response. This is why OPP uses UDP messages be-cause TCP connections cannot be setup in some circumstances.In some conceivable scenarios of multiple concurrent outages,even multi-hop PA communication may be required. Duringlink failure, a PA may not be able to receive replies from thePD. If there are DNS servers available via backup links, thenthis is not an issue. We are exploring how a PA can proactivelycache the locations of various other PAs that it would need tocontact during link failure.

C.3 OPP Messages

Since all the OPP messages will be connectionless UDP mes-sages, they will have to identify the sending PA’s IP address,UDP port and AS number. In Figure I, we list the messagesthat OPP will support for fast failover and incoming traffic loadbalancing. The error codes will include such errors as invalidrequest, conflict with peering policy, unknown address range

Page 5: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 5

TABLE I

OPP MESSAGES

Message Description

PA locate(AS) Message from a PA to the PA directory requesting the address of a PA in a remote ASe.g., PA locate(25)

PA locate reply(AS,ipaddr,port,timeout) Reply from the PA directory, containing the IP address and port of the requested PAe.g., PA locate reply(25,169.229.62.121,8919,6000) and the number of seconds after which the reply is invalid

PA route(prefix) Request from a PA to another PA for the best route chosen for a particular prefixe.g., PA route(128.32.0.0/16)

PA route reply(prefix,AS Path) Reply from a PA giving the AS path chosen for a network prefixe.g., PA route reply(128.32.0.0/16,11423 25)

PA block(prefix,AS1,AS2) Request from a PA to another PA to block all announcements for a particular prefixe.g., PA block(128.32.0.0/16,25,11423) (or all addresses if null) that contain “AS1 AS2” anywhere in the AS path

PA block reply(error code,prefix,AS1,AS2) Reply from a PA giving the status of a previous block requeste.g., PA block reply(0,128.32.0.0/16,25,11423)

PA select(prefix,X,Y) Request from a PA to another PA to select the announcement for a particular prefixe.g., PA select(128.32.0.0/16,7018,50) from AS-X when re-announcing to AS-YPA select reply(error code,prefix,X,Y) Reply from a PA giving the status of a previous select request

PA select reply(1,128.32.0.0/16,7018,50)

and address range ownership verification failed. Each PA willhave the option of using publicly available address allocationregistries to verify that a request to change routing for an ad-dress range came from the AS that owns it. Further, we expecta deployed system to use a public key authentication system toprovide additional protection against malicious use. The PA di-rectory will have a public key that is known widely. All repliesfrom the PA directory will have to be signed by its private keyand will contain the public key of the PA who’s address is beingqueried. All PAs that receive requests will verify that they weresigned by the private key belonging to the originating PA.

We are continuing to define all the error codes and additionalmessages that will be needed for augmenting peering relation-ships, detecting bogus routes and blocking bogus routes.

IV. OPCA DESIGN RATIONALE

In this section, we discuss the design rationale behind OPCA.We begin by examining why using a separate control path fromBGP helps us achieve our goals. We also discuss the use ofmultiple BGP views to improve the accuracy of OPCA.

A. Separating Policy and Routing

AS

AS AS

AS ASRoute

AnnouncementRoute

Announcement

Traffic

RouteArbiter

AS AS

ASAS

(a) (b)

Fig. 3. (a) Current Routing Scenario (b) Route Arbiter Scenario

The key design constraint of the current inter-domain rout-ing system as shown in Figure 3(a) is that domains advertising

reachability in BGP have little control over how those adver-tisements propagate and are applied. This influences how trafficreaches the advertising domain. The end AS has better aggre-gate knowledge on the characteristics of the traffic entering itsnetwork but can do little to change routing at distant ASes toaffect the traffic flow.

The alternate structure shown in Figure 3(b) has been pro-posed in the research literature as we described in Section II.While in this scenario an end AS may be able to influence whichroutes are used to reach it, there are many issues that need to beresolved, such as scalability of link state information, computa-tional complexity, full disclosure of routing policies and loss ofroute selection control for each AS.

ASAS

AS

AS AS

PA

PA PA

PAPA

PA PA

Fig. 4. Our Approach

Instead we propose to separate routing and policy as shown inFigure 4. We continue to use the BGP infrastructure to dissem-inate and select routes, thereby avoiding issues such as the linkstate propagation, route selection control, computational com-plexity and full disclosure of routing policies. We augment thesystem with an overlay protocol to explicitly request routing pol-icy changes between ASes directly, where the two ASes can ne-gotiate policy conflicts and exchange more routing and trafficinformation.

In theory, we could propose our own BGP community at-tribute and distribute policy control information along with BGProutes. However, the advertiser will not know a priori which re-mote BGP speakers adhere to the new community specification,which makes the results of such approaches less predictable.

Page 6: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 6

Furthermore, the system would suffer from the same BGP con-vergence times.

The routing policy conveyed by BGP is essentially “controlsignaling” that attempts to influence the route selection processat the remote router. The basic routing functionality only re-quires exchange of connectivity information (e.g., AS Path andNexthop). Therefore the key premise behind OPCA is that sep-arating the policy (the control part) from routing (basic connec-tivity information) will provide much more flexibility in con-trolling inter-domain paths, which can potentially lead to betterperformance, new services, and prevention of bogus routes.

The specific advantages of this approach include:

• Reduced response time for failure recoveryWhen there is a failure or a change in routing, BGP today cantake up to 15 minutes to converge to a new route. By the timetraffic is forwarded correctly on the new path, network condi-tions could be vastly different from what prompted the changein the first place. During transient fail-over periods, traffic canget caught in routing loops or become black-holed, resulting inincreased packet loss or delay jitter. This can adversely impactthe perceived quality of networked applications. OPCA helps tohasten failure recovery by determining the distant AS that seesboth (or more) possible paths. It allows the local AS to contactthis distant AS and switch the route instead of waiting for nor-mal BGP failure recovery. Hence, the response time of routechanges in the data plane can be greatly reduced.• Enhanced traffic engineering performanceTwo major difficulties in inter-domain traffic engineering are ASPath asymmetry and localized routing decisions. For example,when an AS-X supplies hints of how AS-Y can reach it via a setof AS paths, it is up to AS-Y to decide which path to choose, andhence could diminish the intended load balancing results. Theoutgoing path from AS-X to AS-Y can be different from the re-turn path, and may experience different performance in termsof delay and losses. OPCA allows ASes to directly negotiaterouting policies with one another and determine the best possi-ble inter-domain path. Using the same example, once AS-X andAS-Y reach a consent, OPCA will configure their BGP routersto select the agreed-upon BGP route. The intermediate ASescan also be updated if necessary.• Augmenting peering relationshipsTwo ASes are said to have a peering relationship if they agree toexchange traffic directly without exchanging money. However,this also implies certain routing constraints that do not allowtransit traffic to flow through a peering link. Through a separatelayer of policy distribution and negotiation, OPCA can allowtwo ASes to allow temporary passage of certain transit traffic inexchange for money to reduce congestion or mitigate networkpartitions.• Better support for detecting and handling misconfigured orbogus routesIn OPCA, participating PAs communicate with one another viaan overlay protocol. Authentication mechanisms can be easilyadded during the initial hand-shakes to verify route announce-ments from a particular AS. A PA can configure its BGP routersto reject bogus or misconfigured routes that are not “authenti-cated” by a specific originating AS. Also, if an AS detects abogus route, it can use OPCA to inform other ASes and stem

the spread of the bogus route by requesting route filtering at anAS close to the source of the bogus route.

B. Accuracy, Correctness and Security

Our architecture’s core algorithm relies on inferring the AS-level topology, AS relationships, the likely paths between twopoints and the path convergence points. The efficiency of ouralgorithm will rely partly on the accuracy of these inferencesfrom the RMAP.

We built the RMAP because no oracle exists that can bequeried about the AS structure of the Internet and the relation-ship between any pair of ASes. Due to the lack of such an or-acle, we cannot check the accuracy of our inferences. We haveverified our inferences by comparing them to additional routingtable views [23] and in most cases found fewer than 3% errors.We also have contacted two major ISPs and verified the list ofAS relationships that involve them. We exploit access to mul-tiple routing tables to improve both the completeness and theaccuracy of our inferences. If more ASes participate in this ar-chitecture, then more views can be used. Since peering relation-ships between ASes do not change rapidly, the RMAP does notneed to be recomputed often with fresh BGP tables.

The RMAP improves the efficiency of the protocol by quicklyidentifying likely route paths, route convergence points, androuting relationships between ASes at and near the convergencepoints. If no RMAP existed, OPCA can continue to function,but with much more signaling overhead. A PA would have tocontact several distant PA’s to find out where routes that it isannouncing are propagating and find convergence points for itsroutes. Once found, the PA would have to make several policychange requests because many of them may be denied due toconflicts with neighboring AS relationships.

PAs will not apply routing policies that conflict with their ownlocal peering policies. In this way, the ill effects of having an in-accurate RMAP or having malicious PAs can be kept to a min-imum. The OPP protocol will only allow simple requests to bemade such as a request for route information, for route selec-tion change or termination of route propagation. None of theserequests would violate the BGP protocol. The BGP protocolalready allows each AS to use practically any form of route se-lection by the application of local policies. We can incorporate astandard form of authentication and address ownership verifica-tion to the PA protocol as is proposed for BGP itself [25]. Thiswould ensure that an AS can only influence routing for its ownaddress space and limit OPCA’s exposure to misconfigured ormalicious PAs.

V. APPLICATIONS OF OPCA

In this section, we briefly describe how OPCA achieves fastfail-over of inter-domain routes and inbound load-balancing.

A. Improving Route Fail-Over Time

From Labovitz [7], we know that the fail-over time for BGProutes can be as long as 15 minutes. He showed this throughexperiments where he injected routes into BGP speakers on theInternet and measured their fail-over time. It is currently infea-sible to measure the route fail-over times of all the routes on theInternet to determine how common this long convergence time

Page 7: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 7

EF

XD

C B

A

Fig. 5. Routing Scenario

is. However, Labovitz has shown that the fail-over time dependson the length of the longest alternative path. Consider the trivialexample shown in Figure 5. An arrow from AS A to B indi-cates that B is A’s provider. A double-ended arrow indicates apeer-peer relationship. Suppose X is using the path X → F →

E → B → A to reach A. If the A → B link fails, A would wantX to quickly switch over to using the route through C, becausemost of A’s important customers reside in X. Traditional BGPfail-over would involve the following sequence of steps:1. B’s BGP session with A resets2. B sends a route withdrawal to E and D3. E receives the route withdrawal, re-runs route selection,chooses the previously heard route through D and announcesit to F4. F receives the announcement, chooses it and announces it toX5. D receives the route withdrawal, re-runs route selection andsends a withdrawal to E6. E receives the withdrawal, re-runs route selection and sendsa withdrawal to F7. F receives the withdrawal, re-runs route selection, choosesthe previously heard route through C and announces it to X

Instead, in OPCA, A’s PA can contact F’s PA directly andrequest the change, thereby shortcutting the long BGP conver-gence. The following sequence of events would take place:1. A stops receiving traffic on the link from B and assumes thelink is down2. A’s PA queries the RMAP, determines that the best feasibleroute for X is through C and that F is the best control point3. A’s PA sends PA locate(F) to the PA directory and receivesPA locate reply(F,ipaddr,port,timeout)4. A’s PA sends PA block(prefix,A,B) to F’s PA via the link A-C5. F’s PA applies the change at it’s BGP routers, sendsPA block reply(0,prefix,A,B) to A’s PA6. F sends the new route to X

Alternatively, if F does not have a PA, A can contact E andhave it block the incorrect routes. This is a very simple exam-ple. A more complicated scenario will be more common, wherevarious routing pathologies can cause normal BGP route con-vergence to be especially delayed. It is important to note that inthe case of BGP, the route announcements add to intermediaterouters’ CPU and memory load, and are subject to dampening.On the other hand, the OPP messages are only exchanged be-tween OPCA components, and do not experience per-hop BGP

delay. Once a PA successfully convinces a remote PA (usually atan aggregation point) to switch to an alternate route, the remotePA can configure the local BGP router(s) to select this new pathas the best route. In the example shown above, the time takenfor the system to stabilize is the time taken to send and processOPCA messages from Step 2-5 and for F to propagate new BGProutes to X. Hence, the overhead cost of OPCA is fixed regard-less of the length of Internet paths.

The advantage of our architecture is that PAs can directlycommunicate with other PAs when necessary instead of usinga multi-hop, dampened protocol like BGP. This allows the ben-efits of a fully meshed communication architecture, without thepitfalls because the actual routes still propagate through the un-derlying hierarchical BGP structure. It is important to note herethe role of the RMAP component. It is important for A’s PAto know what the topology, as shown in Figure 5, is in order tocontact the relevant remote PAs. Also, it needs to know if alter-nate routes are feasible. In this scenario, X → F → C → A andX → F → E → B → A and X → F → E → D → B → A areall valid. However, if B and E were peers instead of customerand provider, then the second path would be invalid based oncommonly followed route export rules [23].

B. Incoming Traffic Load Balancing

Direct PA to PA communication also helps achieve incomingtraffic balancing. Consider again the example in Figure 5 whereAS A is load balancing its links to C and B, primarily for someapplication level customer X. With respect to X, F is the aggre-gation point for routes to A via either C or B. Pathological routeupdates need not be sent in BGP by A to “game” route selectionat F. Instead, A’s PA will contact F’s PA directly and requestspecific route selection at F. A’s PA has to first determine thestructure of this graph and that F is the aggregation point. It alsohas to determine whether its route selection requests will con-flict with F’s local route selection policies based on F’s peeringarrangements. The RMAP helps the PA in these respects.

C. Other Applications

OPCA can be applied to solve other problems beyond ourstated goals of fast fail-over and traffic balancing. It can be usedto query the BGP table at remote ASes to help in the diagnosis ofrouter misconfigurations or find the source of malicious BGP ad-vertisements. It can be used to request filtering of certain routesto prevent the spread of bogus BGP announcements. It can alsobe used to also arrange per address prefix micro-peering [26].We do not explore these applications here.

VI. PERFORMANCE EVALUATION AND FUTURE WORK

A. OPCA Deployment

Ideally, we would like to deploy OPCA in all ASes to max-imize the benefits of the architecture, but this may not be prac-tical. Instead, we consider the case of incremental deploymentand we want to analyze the marginal utility of introducing PAs inselected ASes. We find that most of the benefits can be gainedby deploying PAs at a) multi-homed stub ASes, and b) ASeswhere most inter-domain routes converge. This is because theseconvergence points control which of the multiple possible routes

Page 8: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 8

TABLE II

INFERRED RELATIONSHIPS FOR 23,935 AS PAIRS

Relationship # AS pairs Percentage

Provider-customer 22,621 94.51%Peer-peer 1,136 4.75%Unknown 178 0.74%

TABLE III

DISTRIBUTION OF ASES IN THE HIERARCHY

Level # of ASes

Dense core (0) 20Transit core (1) 129Outer core (2) 897Small regional ISPs (3) 971Customers (4) 8898

are used and propagated to their neighbors. Due to the AS levelhierarchy that exists today [23], most of these aggregation pointslie in the core of the Internet. We will describe the AS hierarchythat is obtained from the RMAP and analyze the scalability ofOPCA in Section VI-B. If OPCA offers enough of a gain overthe current routing architecture, we believe that stub ASes willurge their upstream providers to deploy PAs, and such pressurewill cascade up to the core of the Internet.

B. Preliminary Results

B.1 RMAP Implementation

We have completed the design of the RMAP as described inSection III-B.5. Table II shows the relationships that we inferredbetween the 23,935 AS pairs that we extracted from several rout-ing tables [23]. Table III shows the AS hierarchy that we in-ferred among the 10,915 ASes that were present in the routingtables.

Using our hierarchy characterization, we can also study thegrowth of multihoming. We use our current list of customerASes and identify multihomed customer ASes as those makingroute announcements via multiple upstream ASes. Note thatthese values are lower bounds as some ASes and links may notbe visible from the BGP tables that we use. We use routingtables from many time periods [4] to show the change in multi-

0

2000

4000

6000

8000

10000

12000

14000

01Jan

1998

01Jan

1998

01Jan

1999

01Jan

1999

01Jan

2000

01Jan

2000

01Jan

2001

01Jan

2001

01Jan

2002

01Jan

2002

0

2000

4000

6000

8000

10000

12000

14000

No.

Multihoming of ASes

ASesCustomers ASes

Multihomed Customer ASesDoubly Homed Customer ASes

Triply Homed Customer ASes>=4 Homed Customer ASes

Fig. 6.

0 2 4 6 8 10 12

010

0020

0030

0040

0050

0060

00

Number of Paths

Num

ber

of C

usto

mer

AS

es

Different Paths to InnercoreOrthogonal Paths to Innercore (approx)

Fig. 7. Orthogonality of Customer AS → Dense Core Paths

homed customer ASes over time. In Figure 6, we show that thelarge growth in the number of ASes has been fueled by customerASes, most of which are multihomed.

B.2 Scalability Analysis

We also need to show that the system will scale to the num-ber of ASes that exist today and can accommodate projectedfuture growth based on today’s trends. If stub ASes can chooseamong multiple providers for fail-over, they will want to chooseproviders that do not also have the same upstream provider. Thiswill produce a set of AS paths are as uncorrelated during failureas possible. In this case, most of the aggregation points wheremulti-homed paths converge will be in the dense core of the In-ternet that consists of about 20 large ISPs [23].

If only 20 ASes receive the PA requests from all the stubASes, then their PAs may not be able to keep up with the re-quest load. OPCA is flexible in that each AS can have multiplePAs, where PA requests are segregated by the requesting AS orIP address range, as shown in Figure 4. This will reduce theload on each PA. However, an additional synchronization stepmay be needed to coordinate configuration changes at the sameBGP router. If a small number of ASes receive most of the PArequests, they may be able to do more intelligent global opti-mizations across the different requests.

In Figure 7, we attempt to measure the diversity of AS levelpaths on the Internet. We first use BGP routing tables from Oc-tober 19, 2002 to generate the AS level hierarchy and inter-ASrelationships using our prior work [23]. We then analyze thepaths present in the BGP routing tables collected on October 19from the ASes in Table IV. We split up all the paths that we seein the BGP tables into uphill paths from the customer ASes tothe dense core ASes. We do not distinguish between the densecore ASes, so we treat two paths that are identical except forthe last dense core AS hop as identical paths. We see a totalof 193,778 different paths from any customer AS to the densecore ASes. In Figure 7, the “Different Paths to Dense Core” lineshows how many different paths exist from any one customer ASto the dense core. The “Orthogonal Paths to Dense Core” showsan underestimate of the number of paths for each customer ASwhere only the starting AS is the same and the rest of the pathis different before reaching the dense core. We see that for justunder half of the customer ASes, only one path to the densecore exists, while for about another half of the customer ASes,approximately two orthogonal paths exist to the dense core.

Page 9: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 9

TABLE IV

TELNET LOOKING GLASS SERVERS

AS # Name1 Genuity

50 Oak Ridge National Lab.210 Utah Education Network553 Belwue, Germany852 Telus

1838 AT&T, Cerfnet3257 Tiscali International3582 University of Oregon3741 Internet Solutions, South Africa3967 Exodus, US4197 Global Online Japan5388 Energis Squared5511 France Telecom6539 Group Telecom, Canada7018 AT&T, US8220 COLT Internet8709 Exodus, Europe

15290 AT&T, Canada

0 5000 10000 15000 20000

010

0020

0030

0040

0050

0060

0070

00

Number of Paths

Num

ber

of C

usto

mer

AS

es

Different Paths to InnercoreOrthogonal Paths to Innercore (approx)

0 2 4 6 8 10

010

0020

0030

0040

0050

0060

0070

00

Number of Paths

Num

ber

of C

usto

mer

AS

es

Different Paths to InnercoreOrthogonal Paths to Innercore (approx)

Fig. 8. Enumerated Orthogonality of Customer AS → Dense Core Paths (sec-ond graph zooms in on left portion)

Since BGP does not distribute link state, we may be seeingonly a small percentage of the possible paths in these routingtables. Using our AS relationship inferences [23] from Oc-tober 19, we can enumerate all the possible paths from cus-tomer ASes to the dense core that do not violate peer-peer orcustomer-provider export rules. We have enumerated 1,321,470such paths. We plot the orthogonality of these paths in Fig-ure 8. We still see that just under half of the customer ASeshave only a single orthogonal path to the dense core and most ofthe other half have only two orthogonal paths to the dense core.The graphs exhibit a heavy tail, so there are a few customer ASeswith many orthogonal paths to the dense core.

These results indicate how much of a load the ASes in thedense core can expect from customer ASes using the OPP ar-chitecture. Few customer ASes will have the need to switch be-tween several BGP paths, and most that do will switch betweentwo paths. We are continuing to examine whether customerASes have more orthogonal paths to the second tier of ISPs (thetransit core) than the number of orthogonal paths all the wayto the dense core. However, if OPP provides a significant im-provement to the current routing scenario, more customer ASesmay multihome in a way to achieve more orthogonal paths to thedense core and the graphs may shift to the right. The architec-ture should still allow the scaling techniques that we previouslydescribed to handle this possible scenario.

C. Proposed Emulation Study

We plan on evaluating our architecture in an emulationtestbed. We will use about 50 Linux PCs connected via ahigh speed LAN. We will run multiple software Zebra BGProuters [27] and PAs on each machine which we can connectover TCP into any arbitrary AS-level topology, extracted fromreal, sample scenarios we have observed [23]. We will injectthe BGP routing tables that we have collected into the softwarerouters. We can then induce faults by shutting down specificrouters and measure the convergence time. We plan on usingNIST Net [28] to allow us to emulate wide area packet delays.We will use fictitious traffic traces to evaluate OPCA’s load bal-ancing capabilities.

One main metric of success is the reduction in the time to fail-over from a primary route to a backup route for a destinationprefix. The other metric is the time it takes to shift incomingtraffic from one route to another. We will also quantify howeffective OPCA can be given the current AS-level structure ofthe Internet. That is, the current deployment of ASes and inter-AS links will determine how many diverse paths exist for eachstub AS and how many upstream aggregation ASes exist. Thiswill also determine how well OPCA will scale.

VII. SUMMARY

We believe that the motivation behind the large increase inmulti-homed stub ASes is in achieving fast fail-over and trafficload balancing. The increased participation in BGP also makesmore urgent the need to quickly identify and block misconfig-ured or malicious BGP route announcements. Better connectiv-ity to the Internet may also be obtained if micro-peering is sup-ported for specific address ranges for short periods of time toavoid congestion. BGP today offers slow fail-over, limited con-trol over incoming traffic, little or no automated mechanismsfor bogus route detection and prevention and does not supportthe dynamic change in route and traffic filtering between peers.We address these issues by developing an overlay control ar-chitecture that will coexist with and interact with the existingBGP infrastructure. We have explained how the protocol al-lows our goals to be met and outlined some design decisionsthat affect deployment, scaling, choice of control path and accu-racy. We plan on developing and testing this system in an em-ulation testbed running software BGP speakers and using realBGP routing data.

Page 10: OPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing ...sahara.cs.berkeley.edu/papers/ACK03.pdfOPENARCH 2003 1 OPCA: Robust Interdomain Policy Routing and Traffic Control Sharad

OPENARCH 2003 10

ACKNOWLEDGEMENTS

We would like to thank the anonymous reviewers for their de-tailed and helpful feedback. We would also like to thank IonStoica (UC Berkeley), Anthony D. Joseph (UC Berkeley), Ya-suhiko Matsunaga (NEC), Supratik Bhattacharyya (Sprint ATL)and Jennifer Rexford (AT&T Research) for their valuable feed-back on early versions of this work.

REFERENCES

[1] J. W. Stewart, BGP4: Inter-Domain Routing in the Internet. Addison-Wesley, 1998.

[2] T. Bu, L. Gao, and D. Towsley, “On routing table growth,” in Proc. IEEEGlobal Internet Symposium, 2002.

[3] G. Huston, “Analyzing the Internet’s BGP Routing Table,” Cisco InternetProtocol Journal, March 2001.

[4] “University of Oregon RouteViews project.”http://www.routeviews.org/.

[5] “Route Science Web Page.”http://www.routescience.com/.

[6] R. Mortimer, “Investment and Cooperation Among Internet BackboneFirms,” Ph.D. dissertation, Economics Department, UC Berkeley, 2001.

[7] C. Labovitz, R. Wattenhofer, S. Venkatachary, and A. Ahuja, “The impactof Internet policy and topology on delayed routing convergence,” in Proc.IEEE INFOCOM, April 2001.

[8] “NANOG: North America Network Operators Group mailing list.”http://www.merit.edu/mail.archives/nanog/.

[9] G. Huston, “Architectural requirements for inter-domain routing in the In-ternet,” Internet Draft 01, Internet Architecture Board, May 2001.

[10] G. Huston, “NOPEER community for route scope control,” InternetDraft 00, August 2001.

[11] O. Bonaventure, S. Cnodder, J. Haas, B. Quoitin, and R. White, “Control-ling the redistribution of BGP routes,” Internet Draft 02, February 2002.

[12] T. Hardie and R. White, “Bounding longest match considered,” InternetDraft 02, November 2001.

[13] D. Pei, X. Zhao, L. Wang, D. Massey, A. Mankin, S. F. Wu, and L. Zhang,“Improving BGP convergence through consistency assertions,” in Proc.IEEE INFOCOM, 2002.

[14] R. M. Mortier, “Multi-timescale internet traffic engineering,” in IEEECommunication Magazine, October 2002.

[15] D. Estrin, J. Postel, and Y. Rekhter, “Routing arbiter architecture,” 1994.[16] D. G. Andersen, H. Balakrishnan, M. F. Kaashoek, and R. Morris, “Re-

silient overlay networks,” in Proc. ACM SOSP, October 2001.[17] I. Castineyra, N. Chiappa, and M. Steenstrup, “The Nimrod Routing Ar-

chitecture,” RFC 1992, Network Working Group, August 1996.[18] S. K. et al, “BANANAS: A new connectionless traffic engineering frame-

work for the internet,” RPI Technical Report, 2002.[19] R. Neilson, J. Wheeler, F. Reichmeyer, and S. Hares, “A discussion of

bandwidth broker requirements for Internet2 Qbone deployment.”[20] R. Yavatkar, D. Pendarakis, and R. Guerlin, “A framework for policy-

based admission control,” RFC 2753, IETF, January 2000.[21] D. Durham, J. Boyle, R. Cohen, S. Herzog, R. Rajan, and A. Sastry, “The

COPS (COmmon Open Policy Service) Protocol,” RFC 2748, IETF, Jan-uary 2000.

[22] D. O. Awduche, A. Chiu, A. Elqalid, I. Widjaja, and X. Xiao, “A Frame-work for Internet Traffic Engineering,” draft 2, IETF, 2000.

[23] L. Subramanian, S. Agarwal, J. Rexford, and R. H. Katz, “Characterizingthe Internet hierarchy from multiple vantage points,” Proc. IEEE INFO-COM, 2002.

[24] B. Krishnamurthy and J. Wang, “On network-aware clustering of webclients,” in Proc. ACM SIGCOMM, August 2000.

[25] S. K. Charles, “Secure Border Gateway Protocol (S-BGP) — Real WorldPerformance and Deployment Issues.”

[26] R. Mortier and I. Pratt, “Efficient network routeing,” unpublished projectproposal, 2000.

[27] “GNU Zebra, free routing software.” http : //www.zebra.org/.[28] “NIST Net Emulation Web Site.”

http://snad.ncsl.nist.gov/itg/nistnet/.