Tier3 Network IssuesRichard CarlsonMay 19, 2009rcarlson@internet2.edu

Internet2 overviewMember organization with a national backbone infrastructureCampus & Regional network membersNational and International peersTiered connection modelCampus Regional BackboneShared IP and Circuit based infrastructureAssistance with technical and non-technical problems

Basic PremiseApplications performance should meet your expectations!

If they dont you should complain!

However, you must complain effectively!

Realistic ExpectationsWhat the ATLAS physicists needs to defineHow large is a datasetHow long should it take to move this datasetHow often will this dataset be renewedThis data can be turned into network infrastructure requirements

Data movement over REN networks

Link SpdByts/hourFair Sharexfer 1 TB100 Mbps45 GB/h34 GB/h28 hours1 Gbps450 GB/h120 GB/h8 hours10 Gbps4.5 TB/h1 TB/h1 hour

Basic Connectivity TestsPing Confirms that remote host is upSome network operators block these packetsTracerouteIdentifies the routers along the pathSame blocking problem as aboveRouters treat TR packets with lower priority

Advanced user toolsExisting NDT toolAllows users to test network path for a limited number of common problemsExisting NPAD toolAllows users to test local network infrastructure while simulating a long path

Network Diagnostic Tool (NDT)Measure performance to users desktopIdentify real problems for real usersNetwork infrastructure is the problemHost tuning issues are the problemMake tool simple to use and understandMake tool useful for users and network administrators

NPAD/pathdiagA new tool from researchers at Pittsburgh Supercomputer CenterFinds problems that affect long network pathsIdentifies host tuning and network infrastructure problems

NDT/NPAD user interfaceWeb100 based servers (requires patched Linux kernel)Web-based JAVA applet allows testing from any browserCommand-line client allows testing from remote login shell Client installed in OSG VDT

Initial NDT testing shows Duplex Mismatch at one end

NPAD Sample results

Long Path ProblemE2E application performance is dependant on distance between hosts Full size frame time at 100 MbpsFrame = 1500 BytesTime = 0.12 msecIn flight for 1 msec RTT = 8 packetsIn flight for 70 msec RTT = 583 packets

Long Path ProblemH1H2H3X1 msec H1 H270 msec H1 H3

TCP Congestion AvoidanceCut number of packets by Increase by 1 per RTTLAN (RTT=1msec)In flight changes to 4 packetsTime to increase back to 8 is 4msecWAN (RTT = 70 msec)In flight changes to 292 packetsTime to increase back to 583 is 20.4 seconds

*Example - PNNL Throughput Problem950+ Mbps from remote sites to PNNL966 Mbps328 Mbps930 MbpsMeasured Speeds shows problem when PNNL sends

*PNNL Throughput Problem950+ Mbps from remote sites to PNNL966 Mbps6 msec328 Mbps76 msec930 Mbps23 msecInteresting: RTT increases by a factor of 3 and speed decreases by the same factor

*PNNL Throughput Problem950+ Mbps from remote sites to PNNL966 Mbps 6 msec0.0094%6.04% ooo 328 Mbps 76 msec0.0049%5.15% ooo930 Mbps 23 msec0.0045%5.5% oooFinally: look at loss rate and packet reordering (ooo) rate, problem exists in Seattle PNNL metro net

Network Admin ToolsBWCTL Bandwidth ControlAllows single person operation over wide area testing environmentRuns NLANR iperf programOWAMP One way Delay MeasurementAdvanced ping commandAllows single person operation over wide area testing environment

Under Active DevelopmentEmerging PerfSonar toolAllows users to retrieve network path data from major national and international REN network

PerfSonar Next Steps in Performance MonitoringNew Initiative involving multiple partnersESnet (DOE labs)GEANT (European Research and Education network)Internet2 (staff and connectors)

PerfSONAR ServicesMeasurement Archive (MA)Measurement Point (MP)Lookup Service (LS)Topology Service (TS)Authentication Service (AS)

USATLAS Throughput Monitoring

Traceroute Visualizer

Finding a ServerWhat? You dont have one running at your site?

Install the Internet2 Network Performance Toolkit Knoppix Disk

PSC Tuning Page

ESnet Tuning Page

ConclusionsPrimary tools still usefulAdvanced tools are being developedDeveloping tools will make things even easierDemand 10 MB/s as the minimum acceptable throughput rate

Why is Internet2 involved?*