KEK Network

Qi Fazhi

2004.8

KEK SWL2/L3 Switch for

outside connections

Central L2/L3 Switch A

Netscreen

Firewall

Super Sinet Router

10GbE

2 x GbE

2 x GbE

IDSGbE

KEK GW2 x GbE

KEK SWL2/L3 Switch for

outside connections

Central L2/L3 Switch B

Netscreen

Firewall

2 x GbE

2 x GbE

2 x GbE

4 x GbE

NOODSwitch to other places

NOODSwitches to Buildings

4 x GbE

10GbE

Tskuba Center

10GbE

Other Univ. HEP Lab.

GbE

4 x GbE4 x GbE

up to: 14 Vlan, Based MAC Static & DHCP(for wireless)

IDSGbE

KEK’s firewall architecture

L2/L3 Switch for outside connections

Central L2/L3 Switch

Firewall

10GbE

10GbE/GbE

10GbE

2 x GbE

2 x GbE

Securitymonitor

High Throughput Access RouteHigh Throughput Access Route

Regular flowsRegular flowsIDSGbE

KEK

universities/institutesin Asia-Pacific

universities in Japan

SLAC,FNAL,BNL,

CERN,DESY,SDSC,

.....

SuperSINET

SuperSINETSuperSINET

APAN/Transpac

APAN

Since 2004.1

SuperSINET’s links to US/EU

SuperSINETNII’s router at NewYork

OC48 x 4

Gèant

Abilene

ESnet

OC48

OC48

GbE

GbE

GbE

GbE

GbE ISP(10Gbps)

MAN LAN10GbE Switch

POS Channel

10GbE

10GbE

10GbE

10GbE

GbE x 3

IEEAF Atlantic

SLAC is on ESnet

CERN is on GEANT

WAN Performance Measurement and Monitor

-- Study and Work at KEK

Qi Fazhi

2004-07

Study

• WAN Performance Terminology

• WAN Performance Measurement Tools

• How to get larger throughput

Terminology / Elements be relationship to WAN Performance

• Bandwidth– Physical bandwidth, or capacity (C)– Available bandwidth (A)

• Throughput– Maximum throughput– Achievable throughput

• Latency / RTT

TERMINOLOGY•Latency

•Packet Loss

•Bandwidth:the speed that a network element can forward traffic.

(is independent of end hosts and protocol type)

–Physical/capacity:maximum number of bits per second a network element can transfer(is determined by the slowest network element along the path)

–Available:the capacity minus utilization over a given time interval.

•Throughput:amount of data that is successfully sent from one host to another via a network(may be limited by every component along the path from source host to destination host)

–Maximum throughput

–Achievable throughput

Network Performance Measurement Tools

• Iperf/Netperf: tools for measuring end-to-end TCP/UDP performance

• pipechar: hop-by-hop bottleneck analysis tool

• pchar: hop-by-hop performance measurement tool

• traceroute: lists all routers from current host to remote host

How to get larger throughput

Decrease Latency / RTTPacket Loss

Choose the best path Routing coordination

Larger Throughput

TCP Tuning?

Latency / RTT Packet Loss

Parallel stream TCP

TCP Tuning

• Why?– Bandwidth increases– TCP shortage

• How?– Modify the TCP Windows Size– Modify the System Buffer Size– ……– Some New Protocol: FAST

TCP

• TCP is adaptive• It is constantly trying to go faster• It slows down when it detects a loss

• How much it sends is controlled by windows• When it sends is controlled by received

ACK’s(or timeouts)

Summary of techniques to maximize TCP WAN

throughput • congestion window (CWND):TCP uses it to determine how

many packets can be sent at one time, The larger the congestion window size, the higher the throughput.

• TCP “slow start” and “congestion avoidance” algorithms determine the size of the congestion window

• The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket (System)

• For each socket, there is a default value for the buffer size, which can be changed by the program using a system library call just before opening the socket.

• Kernel enforced maximum buffer size.

Why Tuning?

Bandwidth*Delay Product

• Bandwidth * Delay = number of bytes in flight to fill path

• The optimal buffer size is twice the bandwidth*delay product of the link(recover from errors)

• buffer size = 2 * bandwidth * delay = bandwidth * RTT • bandwidth :get it from pipechar or pchar • RTT:get it from Ping• Example:ping time is 50 ms ; the end-to-end network consists of

all 100 BT Ethernet and OC3 (155 Mbps).then TCP buffers should be .05 sec * (100 Mbits / 8 bits) = 625 KBytes

• most of today UNIX OS's by default have a maximum TCP buffer size of only 256 KB! So we should do some tuning……

buffer Windows throughputBW*Delay

Important Points About TCP

• Throughput• congestion window :determine how many packets can be sent at one

time ,the larger the congestion window size, the higher the throughput. • the size of the congestion window :slow start (constantly trying

to go faster)and congestion avoidance (slows down when it detects a loss)• maximum congestion window :is related to the amount of buffer

space that the kernel allocates for each socket

• buffer size :• 2*Bandwidth*Delay Products （ BDP ） = bandwidth * RTT

Summary

Routing coordinationKEK<->IHEP

• Before Change Status– Go through Russia and USA

• KEK->IHEP: KEK->Sinet->Abilene->RBNet->CSTnet->IHEP

• IHEP->KEK: IHEP->CSTnet-> RBNet ->Abilene ->Sinet ->KEK

– RTT is very large

• After Change Status– KEK->IHEP: KEK->SINet->APAN-JP->CERnet->NSFC->CSTnet->IHEP

– IHEP->KEK(not changed): IHEP->CSTnet-> RBNet ->Abilene ->Sinet ->KEK

– For the reason of not changing IHEP->KEK route,the RTT is still large,but there are some changes in the throughput

KEK-IHEP Route Tuning

KEK2505

Sinet2907 Abilene11537 APAN_JP7660

RBnet5568

Dargon_tap9407 Cernet4538 NSFC9406 CSTNet7497

AS1239

AS9405

AS3356

Level 3 Sprint

APAN_JP&APAN_CN Link

IHEP3460

KEK->I HEP(Vi a USA/ RUSSI A)

0

0. 2

0. 4

0. 6

0. 8

1

1. 210 15 20 30 40 50 60 70 80 90 100

110

120

130

140

150

160

Test Durati on (Seconds)

Thr

ough

put (

Mbps

)

KEK->I HEP(Vi a APAN)

0

0. 5

1

1. 5

2

2. 5

10 15 20 30 40 50 60 70 80 90 100

110

120

130

140

150

160

Test Durati on (Seconds)

Thr

ough

put (

Mbps

)

Parallel TCP Stream KEK->IHEP

KEK->I HEP(Vi a APAN)

0

0. 5

1

1. 5

2

2. 5

10 15 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160

Test Durati on (Seconds)

Throu

ghpu

t(Mb

ps)

One TCP Stream Parallel TCP Streams

WAN Performance Measurement/Monitor

• Target– Packet loss & latency– Throughput

• Tools

• Mechanism

WORK

Test PC Data

Remote RemoteRemote

WWW

Traceroute/Ping iperf netperf

HTTP

Ping/iperf/netperf

Mechanism (Base on www)

CMDResult

System Structure

WAN Performance Measurement and Monitor

WAN Performance Measurement WAN Performance Monitor

Performance data collect

Data File

Performance data analysisPerformance measurement resultPerformance measurement command

The System Interface

Measurement: (RTT & Packet Loss/traceroute & ping)

Traceroute and Ping Result

Traceroute and Ping Result with graph

ms

Measurement: Throughput with Iperf

Measurement Result: Throughput with Iperf

Measurement: Throughput with Netperf

Measurement Result: Throughput with Netperf

Monitor: RTT and Packet loss with ping

Monitor Result: RTT and Packet loss with ping

Monitor: Throughput with iperf

Monitor Result: Throughput with iperf

README

• INSTALL– creat a user account: monitor

– cd ~monitor

– tar xvf monitor.tar

– edit /usr/local/apache/conf/httpd.conf

– edit the ~monitor/ping_list.txt and ~monitor/iperf_list.txt

– edit the /etc/crontab

– http://serveraddress/monitor/

Useful links

• http://www.apan.net

• http://ws.arin.net/cgi-bin/whois.pl