Network Monitoring, WAN Performance Analysis,

& Data Circuit Support at Fermilab

Phil DeMar

US-CMS Tier-3 Meeting

Fermilab

October 23, 2008

Active Wide-Area Network Monitoring

PerfSONAR: distributed network monitoring infrastructure Supported by US-LHC T1 sites and Internet2 community

PerfSONAR-PS: Active monitoring package Web services collection built on trusted monitoring tools:

• ping, BWCTL(iperf), owamp, NPAD, NDT toolkit• Web service interface for pulling data into other monitoring tools

Zero configuration; out of box deployment• Based on Knoppix Live CD bootable disk• Optional software bundle deployment

Modest hardware requirements for on-site deployment

PerfSONAR Deployment Status

US-Atlas moving ahead with perfSonar-PS at T1 & T2s: Two dedicated systems per site; one each for latency & b/w testing Systems are spec’ed devices, $628 each (Koi computer) Utilize Knoppix disks & standard configurations

We’ve recommended the same model for US-CMS

Current PerfSONAR-PS deployment: Both US-LHC Tier-1s (FNAL & BNL) UNL (CMS), U-Mich (ATLAS); U-Delaware; Internet-2; ESnet Complete active monitoring matrix of the above

Background information

PerfSONAR-PS project -http://code.google.com/p/perfsonar-ps/

Tour of perfSONAR-PS service is available - http://code.google.com/p/perfsonar-ps/wiki/CodeTour

Knoppix Live CD bootable disk info - http://code.google.com/p/perfsonar-ps/wiki/NPToolkit

Appliance PCs: Vendor: KOI Computing – (630) 627-8811 Spec: 1U Intel Pentium Dual-Core E2200 2.2GHz System Cost: $628/each

Performance Analysis Support

In 1999, Matt Mathis coined the term ‘Wizard’s Gap’ Today, it’s still an issue

Users often don’t know about: Common OS tuning issues for

WAN data movement Wide-area network path, its characteristics, available tools

Its still an end-to-end problem And the world is still short on wizards

Our structured analysis methodology seeks to put some of the wizardry into structured process

Disks Operating System

ApplicationsCPU

MEM

Disks Operating System

Network Applications

CPUMEM

Network

R/S

Router RouterCable

NIC NIC

12

3

4

5

6

78

9

Network Application Performance Factors !!!

1’2’

3’

4’

5’

6’

7’

• Network Delay• Bandwidth• Packet Drop Rate

• CPU speed• MEM Size• System Load• Disk I /O Speed • Operating System

• R/W buffer size• Disk cache size

• NIC Speed

End System

R/S

LAN

WAN

Disks Operating System

ApplicationsCPU

MEM

Disks Operating System

Network Applications

CPUMEM

Network

R/S

Router RouterCable

NIC NIC

12

3

4

5

6

78

9

Network Application Performance Factors !!!

1’2’

3’

4’

5’

6’

7’

• Network Delay• Bandwidth• Packet Drop Rate

• CPU speed• MEM Size• System Load• Disk I /O Speed • Operating System

• R/W buffer size• Disk cache size

• NIC Speed

End System

R/S

LAN

WAN

Find the performance problem area(s)

Performance Analysis Methodology

Structured approach to performance analysis

Model the process like medical diagnosis Collect the physical characteristics Run diagnostic tests Record everything; develop a history of the analysis

Strategic approach: Sub-divide problem space:

• Application-related problems• Host diagnosis and tuning• Network path analysis

Then divide and conquer

7’R/S

Router RouterCable

8

9

WAN

`

Network End System

`

BR

BREnd-to-end Path

LAN LANPacket Trace

Diagnosis Server

WAN

Network End System Diagnosis Server

Network PathDiagnosis ServerNESDS

NPDS

PTDS

NES NESDS NPDSNES

PTDS

NESDS

NPDS

NPDS

NES

BR Border Router

Network Performance Analysis Architecture

PTDS

Host diagnosis Script that pulls system configuration Network Diagnostic Tool (NDT)

• Faulty network connections & NICs, duplex mismatches

Network path diagnosis OWAMP to collect and diagnose one-way network path statistics.

• Packet loss, latency, jitter Other tools such as ping, traceroute, as needed

Packet trace diagnosis Port mirror on border router(s) Tcpdump to collect packet traces Tcptrace to analyze packet traces Xplot for visual examination.

Performance Analysis Tools…

Round-trip time Sequence of routers along the paths One-way delay, delay variance One-way packet drop rate Packet reordering

Network path characteristics collected

Step 1: Definition of the problem space

Step 2: Collect host information & network path characteristics

Step 3: Host tuning & diagnosis

Step 4: Network path performance analysis Route changes frequently? Network congestion: delay variance large? Infrastructure failures: examine the counter one by one Packet reordering: load balancing? Parallel processing?

Step 5: Evaluate packet trace pattern

Network Performance Analysis Methodology

Tier2/Tier3 Sites worked with

UERJ (Brazil) IHEP (China) RAL (UK) University of Florida IFCA (Spain) TTU (Texas) CIEMAT (Spain) Belgium OWEA (Austria) CSCS (Swiss)

An available service for CMS Tier-2/3 sites A work-in-progress at this point Focus is on process as well as results Willing to work with others in this area

Future areas of effort: Incorporate into work flow & content management system Make use of perfSonar monitoring infrastructure

https://plone3.fnal.gov/P0/WAN/netperf/methodology/

How to get hold of us: Send email to WAN@FNAL.GOV Wide Area Work Group video-conf meetings every other Friday

Performance Analysis Status & Summary

Strategic Direction Toward Circuits

DOE High Performance Network Planning Workshop established a strategic model to follow: High bandwidth backbones for

reliable production IP service• ESnet

Separate high-bandwidth network paths for large scale science data flows • Science Data Network

Metropolitan Area Networks (MAN) for local access• Fermi LightPath a cornerstone

for Chicago area MAN

ESnet4: Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits)

Cle

vela

nd

Europe(GEANT)

Asia-Pacific

New York

Chicago

Washington DC

Atl

anta

CERN (30+ Gbps)

Seattle

Albuquerque

Au

str

ali

a

San Diego

LA

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN (30+ Gbps)Canada(CANARIE)

Asi

a-Pac

ific

Asia Pacific

GLORIAD (Russia and

China)

Boise

HoustonJacksonville

Tulsa

Boston

Science Data Network Core

IP Core

Kansa

s

City

Au

str

ali

a

Sunnyvale

Production IP core (10Gbps)

SDN core (20-30-40-50 Gbps)

MANs (20-60 Gbps) or backbone loops for site access

International connections

USLHCNet

Topology of circuit connections

Circuits utilize MAN infrastructure: 10GE channel(s) reserved for routed IP

service (purple) LHCOPN circuit (orange) to CERN SDN channels for E2E circuits to CMS

Tier-2/3 (shades of green)

Circuits based on end-to-end vLANs Direct BGP peering with remote site

Multiple provider domains is the norm Deployed technology varies by

domains involved Complexity is higher than IP service

FNAL Alternate Path Circuits

Supported since 2004

Serve a wide spectrum of experiments CMS Tier-2s are heavy

users

Implemented on multiple technologies But based on end-to-end

layer-2 paths

Usefulness has varied

E2E Circuit Summary

FNAL currently supporting E2E circuits to Tier0 & Tier2s A few Tier3s

Today, circuits are largely static configurations

Dynamic circuit services are becoming available Driven largely by Internet2 DCN services

Alternate path support services also emerging Lambda Station (FNAL) TeraPaths (BNL)

Contact WAN@FNAL.GOV for help or information