Network Connectivity and Projects – a Perspective from the UK

Richard Hughes-Jones PPNCG

SuperJANET4 StatusConnectivity to EuropeConnectivity to the USGrid Networking Projects

SuperJANET4: Backbone and Access links

Worldcom supplied the transmissionUKERNA layer the IP service

Core PoP IP router at WorldcomBackbone Access Router at MANs

Access Links:Large MAN 2.5 Gbit -> 10-20 GbitMedium MAN 622 Mbit -> 2.5 Gbit

4 node DWDM development netDeployment Status:

Backbone Oct 00 Routers Nov 00All sites Mar 01Proved to be StableConstant growth of traffic

SuperJANET4: ping rtt Core routers Jun 01

SuperJANET4: ping rtt Core routers

Lancaster Glasgow

MAN / LAN Issues

Bristol Cambridge

London MAN Upgrade

UDP Throughput

Mbit/s

UDP Packet loss

%

UDPmon Tests

Manchester – London

MAN was 155 Mbit ATM

1st Oct Time interval in Weeks Richard HJ

ICFAMON Plot from RAL to CERN for 19th Oct to 1st Nov 2001

Connectivity to Europe – Access links (1)

UK Access link 155 Mbit ATMSustained rate 130 MbitContract to end of Nov 01Bad news for users !

Traceping Oxford to CERN for 31st October 2001

Connectivity to Europe – Access links (2)

loss around ten155-gw.ja.net router

J. Macallister

Connectivity to Europe : GeantStart mid November 2001UKERNA switched off TEN-155 3 Dec 2001Monitoring – see later

Connectivity to US

6 * 155 Mbit links

US link about to go to 2.5 G bitIP commodity attached in London Research traffic over 2.5G bitPeer in Hudson St. 622 Mbit to Esnet622 Mbit to Abilene.

Monitoring: US Traffic

UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US17 Jul 2000 (300kbit/s) 7 Dec 2001 (900kbit/s)

peak is 88% of total BW 930 Mbit10 minute averages 10 minute averages

Last 7 days 1 hour averages Last 7 days 1 hour averages

Monitoring: US Traffic - TrendsJan 99 – Nov 02

UKERNA – US Traffic Daily totals Mbytes

Weekend

Xmas 99

Weekday

Xmas 01

Peak load ~88 %

J. Macallister

Grid Network Projects

DataGridGridPP

Extending DataGrid to BaBar, D0, CDF,… in UK

DataTAG Foci: Network resarch, Grid inter-operability

MB-NG UK funded 2.5Gbit Development Network

Optical Switching

Network Monitoring

Several tools in test – plugged into a coherent structure: PingER, RIPE one way times, iperf, UDPmon, rTPL, GridFTP,

and NWS prediction engine continuous tests for last few months to selected sites:

DL Man RL UCL CERN Lyon Bologna SARA NBI SLAC … The aims of monitoring for the Grid:

to inform Grid applications, via the middleware, of the current status of the network – input for resource broker and scheduling

to identify fault conditions in the operation of the Grid to understand the instantaneous, day-to-day, and month-by-month

behaviour of the network – provide advice on configuration etc.

Report written on LDAP scheme for publishing the network information

Will be used by UK e-science sites and non HEP WPs

UDPmon Measuring Throughput with UDP

Send a burst of UDP frames spaced at regular intervalsVary the frame size and the frame transmit spacingRecord

The time to send and the time to receive the frames The number received, the number lost, number out of order The received inter-packet spacing

Zero stats

OK done

●●●

Get remote statistics

Send statistics

Send data frames atregular intervals ●●●

Time to send

Time to receive

Inter-packet time

Local NetworkMonitoring

Store & Analysisof Data (Access)

Access to current and historic dataand metrics via the Web, i.e. WP7NM Pages, access to metric forecasts

Backend LDAP script to fetch metricsMonitor process to push metrics

localLDAPServer

Grid Application access viaLDAP Schema to- monitoring metrics; - location of monitoring data.

PingER(RIPE TTB)

IperfERUDPmon

rTPLNWS

etc

LDAPSchema

Grid AppsGridFTP

Network Monitoring Architecture

Robin Tasker

Network Monitoring Components

Ping Netmon UDPmon iPerf Ripe

Cronscript

plot

Table

LDAP

raw

control Cronscript

controlCronscript

plot

Table

LDAP

raw plot

Table

LDAP

raw

WEB Display AnalysisGrid BrokerPredictions

Web I/f

Scheduler

Tool

Clients

LDAP

raw

LDAP

raw

Ping & UDP throughput MAN-RAL From 20 Oct 01

PingER rtt (ms)

dl – RAL

1000 byte packet

Forecast

UDPmon Zero packet loss!

UDPmon throughput Mbit/s

man – RAL

300 * 1400 byte frames

Ping & UDP throughput MAN-CERNFrom 20 Oct 01

PingER rtt (ms)

dl – cern

1000 byte packet

Forecast

UDPmon throughput Mbit/s

man – cern

300 * 1400 byte frames

iperf & UDP throughput MAN-CERN From 20 Oct 01

Iperf throughput

ucl – cern

262144 byte buffer

Forecast

UDPmon throughput Mbit/s

man – cern

300 * 1400 byte frames

iperf & UDP throughput MAN-SARA From 20 Oct 01

Iperf throughput

ucl – sara

262144 byte buffer

Forecast

UDPmon throughput Mbit/s

man – sara

300 * 1400 byte frames

iperf & Pinger UK-Bologna From 20 Oct 01

Iperf throughput

ucl – Bologna

262144 byte buffer

Forecast in green

PingER rtt (ms)

dl – Bologna

1000 byte packet

Forecast

iperf throughput UCL-SARA From 1 Nov 01 – Geant Operational

Iperf throughput

ucl – sara

262144 byte buffer

Geant Enabled Routing Stable

Brief outline of work plan

High Performance High Throughput: Understanding the end system HW, best way to monitor traffic and protocol packets, Type and effect of background traffic, Throughput vs rtt, throughput vs window-size, Throughput using multiple TCP streams, and effect on the Network GridFTP and BBftp at Gigabit, Effect of different TCP stacks, Use of non-TCP protocols, and effect on the Network Show and tell demos

Bandwidth Reservation and QoS Identification of traffic classes and middleware components

requiring QoS Study of traffic differentiation and traffic engineering techniques demonstration of QoS & Bandwith Reservation

The EU DataTAG project

EU Transatlantic Gigabit project.Status: approved for 4 M ECU start on 1 Dec 2001Partners: CERN/PPARC/INFN/UvA. IN2P3 sub-

contractorThe main foci are:Grid Network Research including:

Provisioning (CERN) Investigations of high performance data transport (PPARC) End-to-end inter-domain QoS + BW / network resource

reservation Bulk data transfer and monitoring (UvA)

Interoperability between Grids in Europe and the US PPDG, GriPhyN, DTF, iVDGL (USA)PPDG, GriPhyN, DTF, iVDGL (USA)

DataTAG project

AbileneAbilene

ESNETESNET

MRENMREN

STAR-TAP

UKUKSuperJANET4

ITGARR-

B

ITGARR-

B

GEANT

NewYork

Olivier Martin

NLSURFne

t

NLSURFne

t

2.5 Gbit lambda between CERN and StarlightPOS 2nd half 2002 – WDM later

STAR-LIGHT

CERN

2.5 Gbps SDH “Lambda”100/1000 Mbps Ethernet

Amsterdam

GSR

GSRGSR

GSR

GSR

λ Broker

Application

CERN

StarLight

Italy

UK

Optical Switching - DataTAG

7600

IwireAbileneESnet

MB – NG SuperJANET Testbed

MB - NG

SuperJANET4ProductionNetwork

Leeds

RAL /UKERNA

RAL /UKERNA

SJ4 DevC-PoP

Reading

SJ4 DevC-PoP

Warrington

Manc MCC

ULCC UCL

SJ4 DevC-PoPLondon

Gigabit Ethernet2.5 Gbit POS10 Gbit POS

E-science core projectMB - NG

Project to investigate and pilot: end-to-end traffic engineering and management over multiple

administrative domains – MPLS in core diffserv at the edges.

Managed bandwidth and Quality-of-Service provision. (Robin Tasker)

High performance high bandwidth data transfers. (Richard HJ)

Demonstrate end-to-end network services to CERN using Dante EU-DataGrid and the US DataTAG.

Status: approved with requested funds start on 1 Dec 2001 Partners:CISCO, CLRC, Manchester, UCL, UKERNA plus

Lancaster and Southampton (IPv6) A technical meeting held – draft spec. of the HW required Would like to use real Grid traffic – as demonstrator:

CDF UCL-RAL

BaBar Man-RAL

And Finally in case it helps:

GridPP and DataGrid have been working on packaging Globus v2 as a .rpm kit.Makes installation simpler.Available from DataGrid WP6 web:

marianne.in2p3.fr/datagrid/testbed1/globus

Conclusion & Status

SuperJANET4 stable working well Access links to Campus becoming very inportantNetwork Links and peering to US good overallUS-UK bandwidth upgrade overdue Geant access to Europe OK (at the moment !)Considerable UK interest in the Internet2 tests Grid: Lots of interesting work to do …