Tier3 Network Issues

of 29 /29
Tier3 Network Issues Richard Carlson May 19, 2009 [email protected]

Embed Size (px)


Tier3 Network Issues. Richard Carlson May 19, 2009 [email protected] Internet2 overview. Member organization with a national backbone infrastructure Campus & Regional network members National and International peers Tiered connection model Campus  Regional  Backbone - PowerPoint PPT Presentation

Transcript of Tier3 Network Issues

  • Tier3 Network IssuesRichard CarlsonMay 19, [email protected]

  • Internet2 overviewMember organization with a national backbone infrastructureCampus & Regional network membersNational and International peersTiered connection modelCampus Regional BackboneShared IP and Circuit based infrastructureAssistance with technical and non-technical problems

  • Basic PremiseApplications performance should meet your expectations!

    If they dont you should complain!

    However, you must complain effectively!

  • Realistic ExpectationsWhat the ATLAS physicists needs to defineHow large is a datasetHow long should it take to move this datasetHow often will this dataset be renewedThis data can be turned into network infrastructure requirements

  • Data movement over REN networks

    Link SpdByts/hourFair Sharexfer 1 TB100 Mbps45 GB/h34 GB/h28 hours1 Gbps450 GB/h120 GB/h8 hours10 Gbps4.5 TB/h1 TB/h1 hour

  • Basic Connectivity TestsPing Confirms that remote host is upSome network operators block these packetsTracerouteIdentifies the routers along the pathSame blocking problem as aboveRouters treat TR packets with lower priority

  • Advanced user toolsExisting NDT toolAllows users to test network path for a limited number of common problemsExisting NPAD toolAllows users to test local network infrastructure while simulating a long path

  • Network Diagnostic Tool (NDT)Measure performance to users desktopIdentify real problems for real usersNetwork infrastructure is the problemHost tuning issues are the problemMake tool simple to use and understandMake tool useful for users and network administrators

  • NPAD/pathdiagA new tool from researchers at Pittsburgh Supercomputer CenterFinds problems that affect long network pathsIdentifies host tuning and network infrastructure problems

  • NDT/NPAD user interfaceWeb100 based servers (requires patched Linux kernel)Web-based JAVA applet allows testing from any browserCommand-line client allows testing from remote login shell Client installed in OSG VDT

  • Initial NDT testing shows Duplex Mismatch at one end

  • NPAD Sample results

  • Long Path ProblemE2E application performance is dependant on distance between hosts Full size frame time at 100 MbpsFrame = 1500 BytesTime = 0.12 msecIn flight for 1 msec RTT = 8 packetsIn flight for 70 msec RTT = 583 packets

  • Long Path ProblemH1H2H3X1 msec H1 H270 msec H1 H3

  • TCP Congestion AvoidanceCut number of packets by Increase by 1 per RTTLAN (RTT=1msec)In flight changes to 4 packetsTime to increase back to 8 is 4msecWAN (RTT = 70 msec)In flight changes to 292 packetsTime to increase back to 583 is 20.4 seconds

  • *Example - PNNL Throughput Problem950+ Mbps from remote sites to PNNL966 Mbps328 Mbps930 MbpsMeasured Speeds shows problem when PNNL sends

  • *PNNL Throughput Problem950+ Mbps from remote sites to PNNL966 Mbps6 msec328 Mbps76 msec930 Mbps23 msecInteresting: RTT increases by a factor of 3 and speed decreases by the same factor

  • *PNNL Throughput Problem950+ Mbps from remote sites to PNNL966 Mbps 6 msec0.0094%6.04% ooo 328 Mbps 76 msec0.0049%5.15% ooo930 Mbps 23 msec0.0045%5.5% oooFinally: look at loss rate and packet reordering (ooo) rate, problem exists in Seattle PNNL metro net

  • Network Admin ToolsBWCTL Bandwidth ControlAllows single person operation over wide area testing environmentRuns NLANR iperf programOWAMP One way Delay MeasurementAdvanced ping commandAllows single person operation over wide area testing environment

  • Under Active DevelopmentEmerging PerfSonar toolAllows users to retrieve network path data from major national and international REN network

  • PerfSonar Next Steps in Performance MonitoringNew Initiative involving multiple partnersESnet (DOE labs)GEANT (European Research and Education network)Internet2 (staff and connectors)

  • PerfSONAR ServicesMeasurement Archive (MA)Measurement Point (MP)Lookup Service (LS)Topology Service (TS)Authentication Service (AS)

  • USATLAS Throughput Monitoring

  • Traceroute Visualizer

  • Finding a ServerWhat? You dont have one running at your site?

    Install the Internet2 Network Performance Toolkit Knoppix Disk

  • PSC Tuning Page

  • ESnet Tuning Page

  • ConclusionsPrimary tools still usefulAdvanced tools are being developedDeveloping tools will make things even easierDemand 10 MB/s as the minimum acceptable throughput rate

    Why is Internet2 involved?*