1

Masaki Hirabaru and Yasuhiro Koyama{masaki,koyama}@nict.go.jp

APEC-TEL APGrid WorkshopSeptember 6, 2005

e-VLBI: Science over High-Performance Networks

2

Radio Telescopes

NICT Kashima Space Center 34m

Onsala Space Observatory 20m (left)

Perks 64m (right)Australia Telescope National Facility

MIT Haystack 18m

Shanghai25m

3

• Geophysics and Plate Tectonics

VLBI ApplicationsVLBI Applications

鹿島

ハワイ

アラスカ

5700km5700km

5400km5400km

4700km4700km

鹿島－ハワイの基線長変化

- 400

- 200

0

200

400

1984 1986 1988 1990 1992 1994

年

基線

長（ｍ

ｍ）

アラスカ－ハワイの基線長変化

- 400

- 200

0

200

400

1984 1986 1988 1990 1992 1994

年基

線長

（ｍｍ

）

鹿島－アラスカの基線長変化

- 400

- 200

0

200

400

1984 1986 1988 1990 1992 1994

年

基線

長（ｍ

ｍ）

-63.5 0.5 mm/year

-46.1 0.3 mm/year

1.3 0.5 mm/year

Kauai

Fairbanks

Kashima

Kashima-Kauai Baseline Length

Fairbanks-Kauai Baseline Length

Kashima-Fairbanks Baseline Length

4

VLBI Applications (2)VLBI Applications (2)

Halca （ Muses-B ）

NGC4261

Radio Telescope Satellite Radio Telescope Satellite ‘Halca’ and its images‘Halca’ and its images

Earth Orientation ParametersEarth Orientation Parameters

• Radio Astronomy : High Resolution Imaging, Astro-dynamics• Reference Frame : Celestial / Terrestrial Reference Frame• Earth Orientation Parameters, Dynamics of Earth’s Inner Core

5

VLBI (Very Long Baseline Interferometry)

•e-VLBI geographically distributed observation, interconnecting radio antennas over the world

ASTRONOMYGEODESY

•Gigabit / real-time VLBI multi-gigabit rate sampling

dela

y

radio signal from a star

correlator

A/D clockA/D

　Internet

clock~Gbps

~Gbps

A B

A

B

d

Large Bandwidth-Delay Product Network issue

6

VLBI System Transitions

K5 Data AcquisitionTerminal

1st Generation

2nd Generation

1983~Open-Reel TapeHardware Correlator 1990~

Cassette TapeHardware Correlatore-VLBI over ATM

3rd Generation

2002~PC-based SystemHard-disk StorageSoftware Correlatore-VLBI over Internet

K3 Correlator (Center)K3 Recorder (Right)

K4 Terminal

K4 Correlator

64Mbps

256Mbps1 ～ 2Gbps

7

Recent e-VLBI System DevelopmentsRecent e-VLBI System DevelopmentsK5 by NICTK5 by NICT

ADS1000(1024Msample/sec 1ch 1bit or 2bits)

ADS2000(64Msample/ch·sec, 16ch, 1bit or 2bits)

IP-VLBI Board(~16Msample/ch·sec, ~4ch, ~8bits)

PC : Data AcquisitionCorrelation

VSI

Correlatorother DAS

Internet

PC-VSI Board(Supports VSI-H specifications) VSI

VSI-ERTP/RTCP

8

Motivations• MIT Haystack – NICT Kashima e-VLBI Experiment

on August 27, 2003 to measure UT1-UTC in 24 hours– 41.54 GB NICT → MIT 107 Mbps (~50 mins)

41.54 GB MIT → NICT 44.6 Mbps (~120 mins)– RTT ~220 ms, UDP throughput 300-400 Mbps

However TCP ~6-8 Mbps (per session, tuned)– BBFTP with 5 x 10 TCP sessions to gain performance

• HUT – NICT Kashima Gigabit VLBI Experiment

- RTT ~325 ms, UDP throughput ~70 MbpsHowever TCP ~2 Mbps (as is), ~10 Mbps (tuned)

- Netants (5 TCP sessions with ftp stream restart extension)

There was bandwidth available but we could not utilize.

9

• Observing Bandwidth Data rate

(Precision of Time Delay)-

1

(SNR)1/2

• Wave Length / Baseline Length Angular Resolution

• Baseline Length (EOP Precision)-1

VLBI - CharacteristicsVLBI - Characteristics

Faster Data Rate = Higher Sensitivity

Longer Distance = Better Resolution

10

Long Distant Rover Control

(at least) 7 minutes one way delay

Image

Command

EarthMars

When operator saw collision, it was too late.

11

Long-Distance End-to-End Congestion Control

Merge (Bottleneck) A+B > C

Overflow

Sender(JP)

Receiver(US)

Feedback

BWDP: Amount of data sent but not yet acknowledged64Kbps x 200ms = 1600B ~ 1 Packet

1Gbps x 200ms = 25MB ~ 16700 Packets

200ms round trip delay

A

B

C

12

Average TCP Throughput less than 20Mbps

Q=50

ExampleHow much speed can we get?

ReceiverSenderHigh-Speed

Backbone

L2/L3SW

1G 100M

Delay at light speed: 100ms

1G

13

Analyzing Advanced TCP Dynamic Behavior in a Real Network(Example: From Tokyo to Indianapolis at 1G bps with HighSpeed TCP)

The data was obtained during e-VLBI demonstration at Internet2 Member Meeting 　 in October 2003.

Throughput

RTT

Window Sizes

Packet Losses

14

　

KwangjuBusan

2.5G

Fukuoka

Korea

                                       

2.5G SONET

KORENTaegu

Daejon

10G

1G (10G)1G

1G

Seoul XP

Genkai XP

Kitakyushu

Kashima

1G (10G)

Fukuoka Japan

250km

1,000km10G

JGN II

9,000km

4,000km

Los Angeles

Chicago

Washington DC

MIT Haystack

10G

2.4G

APII/JGN II

Abilene

Koganei

1G(10G)

Indianapolis

100kmbwctl server

Performance Measurement Platformfor High-Performance Scientific Data Transfer

10G

Tokyo XP /JGN II I-NOC

*Performance Measurement Point Directory http://e2epi.internet2.edu/pipes/pmp/pmp-dir.html

perf server

e-vlbi server

JGNII

10G

GEANT

SWITCH

7,000km

TransPAC

Pittsburgh

U of Tokyo

Locate the problemInternational collaboration to support for science applications

U. Hawaii

15

Solutions by Advanced TCPs

• Loss-Based ► AQM (Advanced Queue Management)

Reno, Scalable, High-Speed, BIC, …

• Delay-BasedVegas, FAST

• Explicit Router NotificationECN, XCP, Quick Start, SIRENS, MaxNet

How can wee foresee collision (queue overflow)?

16

TCP Performance with Different Queue Sizes

17

* set to 100M for measurement

Measuring Bottleneck Queue Sizes

Switch / Router Queue Size Measurement Result

ReceiverSenderCapacit

y C

packet train lost packetmeasured packet

Queue Size = C x (Delaymax – Delaymin)

DeviceQueuing

Delay (µs)Capacity (Mbps)

Estimated Queue Size (1500B)

Switch A 6161 100* 50p/75KB

Switch B 22168 100* 180p/270KB

Switch C 20847 100* 169p/254KB

Switch D 738 1000 60p/90KB

Switch E 3662 1000 298p/447KB

Router F 148463 1000 12081p/18MB

Router G 188627 1000 15350p/23MB

cross traffic injectedfor measurement

18

RouterSwitch

1Gbps(10G)100Mbps

(1G)

b-1)

Typical Bottleneck Cases

RouterSwitch

a)

Queue~100 Queue

~1000

VLANs

Switch/Router

10G LAN-PHYEthernet Untag

b-2)

9.5G WAN-PHY802.1q Tag

19

e-VLBI Demonstration in JGN II Osaka (Jan. 2005)

e-VLBI data transfer achieved ~700Mbps from Haystack to Osaka ~900Mbps from Kashima to Osaka

Software Cross Correlation

~240Mbps per station

Dr. Koyama

4 Apple G5 Server machines (8 CPUs in Total)

Osaka

#7,#8

1G 1G

Raid Disks

1G x4 1G x4

Raid Disks

#5,#6

#1,#2

#3,#4

CPU x8

10G

Tokyo

NICTKashima

Abilene(10G)

MITHaystack

CHI

WAS

JGN II Int’l(10G)

1G/10G

1G/2.5G

*TCP parameters were tuned for the path.

20

                                       

VLBI Antenna Locations in North-East Asia

Shintotsukawa 3.8m

Tomakomai 11m, FTTH (100M)70km from Sapporo

Mizusawa 10m 20m118km from Sendai

Tsukuba 32m, OC48/ATMx2 SuperSINET

Kashima 34m, 1Gx2 JGN II, OC48/ATM Galaxy

Yamaguchi 32m1G, 75M SINET

Gifu 11m 3m, OC48/ATMx2 SuperSINET

Usuda 64m, OC48/ATM Galaxy

Nobeyama 45mOC48/ATM Galaxy

Nanshan (Urumqi) 25m70km from Urumqi

Koganei 34m, 1Gx2 JGN II, OC48/ATM Galaxy

Miyun (Beijing) 50m50km from Beijing

2Mbps

2Mbps

Yunnan (Kunming) 3m (40m)10km from Kunming

Sheshan (Shanghai) 25m30km from Shanghai

Observatory is on CSTNET at 100M

Jeju 20mTamna U

Seoul 20mYonsei U

Ulsan 20mU Ulsan

Daejon 14mTaeduk

Ishigaki 20m

Ogasawara 20mChichijima 10m

Iriki 20m Kagoshima 6mAira 10m

Legend

connectednot yet connectedantenna under construction

21

e-VLBI Data Transfer

Real-time e-VLBI – flat-rate live data transfer

InternetSynchronize

Correlation

Common e-VLBI – file transfer

Carry the disk to the nearest stationto put on-line

Correlate among many combinations concurrently to get more precise data (like a virtual huge antenna)

Future – e-VLBI infrastructuremulticast and automated

22

Summary

• High-performance scientific data transfer faces on network issues we need to work out.

• Big science applications like e-VLBI and High-Energy Physics need cooperation with network and Grid researchers.

• Deployment of performance measurement Infrastructure over research networks is on-going on world-wide basis.