Bas Kreukniet, Sr Network Specialist at SURFSARA

NL-T1 Expectations, findings, and innovation

Geneva Workshop 10 Februari 2014

Outline

1. Expectations from NL-T1 grid administrators2. Findings while connecting to the LHCONE3. Innovation: Ethernet OAM and NSI

Expectations from NL-T1 administrators

Advice from grid administrators NL-T1:

• Bulk data with simple applications (grid-FTP). “Keep it simple” • Network provisioning from application is

considered “complex”• Don’t rush to merge LHCOPN with LHCONE

NL-T1 connected to LHCONE since 20 Jan 2014

BGP routing starts at the connected site.

BGP routing for T1’s:

Not only a “NREN thing” – it already starts at your organisation!

Focus on some BGP topics for connecting sites to LHC networks

LHCOPN connectivity

LHCONE connectivity

128.142.0.0/16 *[BGP/170] 1w2d 04:07:47, MED 120, localpref 100 AS path: 20965 20641 513 I > to 62.40.126.161 via xe-4/0/1.2012

NL-T1AS1162

GÉANTAS20965

LHCONE-RS CERN

AS20641

CERNAS513

LHCONE.inet.0: 133 destinations, 133 routes (133 active, 0 holddown, 0 hidden)+ = Active Route, - = Last Active, * = Both

Preferred route: LHCOPN, LHCONE or Internet

• primary: direct T1–T1 over LHCOPN• secondary: T1–T1 over LHCOPN via another T1 • tertiary: LHCONE• quaternary: Internet

Preferred route: route (a)symmetry

If everyone makes this choice, all connections are symmetric.But: sites may have different preferences:

10 Gb/s 10 Gb/s

1 Gb/s 100 Gb/ssite 1 site 2

ISP-B

ISP-A

Tie-breaker between LHCOPN and LHCONE

CNAF routers at NL-T1

LHCOPN131.154.128.0/17 *[BGP/170] 14:58:18, localpref 100 AS path: 34878 137 I

LHCONE131.154.128.0/17 *[BGP/170] 1w2d 04:23:40, MED 120, localpref 100 AS path: 20965 137 I

LHCOPN / LHCONE route preference

BNL routes at NL-T1

LHCOPN130.199.185.0/24 *[BGP/170] 2w5d 09:35:43, MED 10, localpref 100 AS path: 513 43 I > to 192.16.166.73 via xe-1/1/0.0 [BGP/170] 3d 01:15:22, MED 51, localpref 100 AS path: 39590 513 43 I > to 109.105.124.17 via xe-2/1/0.0

LHCONE130.199.185.0/24 *[BGP/170] 1w2d 04:27:43, MED 120, localpref 100 AS path: 20965 293 43 I > to 62.40.126.161 via xe-4/0/1.2012

Problems and concerns regarding BGP

• A site advertising his routes has no control who to send a route to. At best they can give hints with BGP communities.

• The site receiving a route decides which route to accept and how to accept.

BGP in LHC networks

Only As is not enough. More info needed. community for origin of a route T1/T2?Or even better: -site connected to LHCOPN-site connected to LHCONE- Site connected to both:Type A “prefer LHCONE for this route”Type B “prefer LHCOPN for this route”- Specials: dedicated link between two (T1) sites. (“VPN”or “private link”)

Route Preference Solutions

• Idea: Tag routes with two types of communities: • One for origin or source• One for destinations• See also: BGP hinting by Martin Sweeny (Indiana U)

• BGP Always-compare-MED always on. We sometimes add metrics on incoming routes.

Operational Issues

• We received routes over LHC from a site,but traffic we send is blackholed by that site

• The site was still reachable over the Internet.• This happened to us twice recently: on LHCOPN (accidental route

redistribution) and LHCONE (incoming IP filter).• Configuration errors will be made (we’re also just human)

• LHCOPN Link NL-T1 – TRIUMF link is still unstable• 31 outages last 4 months

Monitoring remains important!

LHCONE and LHCOPN layer 2 monitoring

Ethernet OAM monitoring or Layer 2 monitoring

NIKHEF, TRIUMF to participate as Measurement PointsLooking for T1’s and T2 to participate

Advantages:• Layer2 keep-alive, ping and traceroute• Interdomain, intervendor solution• L2 devices can be made visible• unidirectional fibercuts can be signalled

LHCONE Innovation

NL-T1 likes to partcipate in NSI for LHCONE

NSI experiences so far:• Use-case: Life Science Grid (LSG) in Holland

makes use of NSI since autumn 2013. • Freek contributed to standard• Sander wrote NSI client and implemented it for

“Cloud Bypassing” in the Life Science Grid.

Cloud Bypassing on Life Science Grid (LSG)

Compute clusters at ±10 locations in the Netherlands

Cloud Bypassing on Life Science Grid (LSG)

Offloading campus networks (some only have 1 Gb/s Internet)

LHCONE Innovation: Offloading is cheaper

Internet full routing (incl backup): € 8k – €10k per month for 10 Gb/sLHCONE or dynamic lightpath: € 2k – 3k per month for 10 Gb/s

Questions?

Erik Ruiter

Bas Kreukniet

Diederik Vandevenne

Sander Boele

Farhad Davani

Freek Dijkstra