EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE Recent eVLBI developments at JIVE Arpad...
-
Upload
kory-chase -
Category
Documents
-
view
215 -
download
2
Transcript of EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE Recent eVLBI developments at JIVE Arpad...
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Recent eVLBI developments at JIVE
Arpad Szomoru
Joint Institute for VLBI in Europe
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Outline• Introduction• Protocols, tuning issues• Tests and results
• ftp-based• Dual buffered• Single buffered• Real-time
• Jumbo frames• Network monitoring• Centralised control
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Introduction
• Move to disk-based operations nearly complete
•More reliable, data quality•Cheaper to maintain•Ease of use
•Direct access compared•Unattended operations
• eVLBI: the future?•No consumables•High bandwidth•Fast turn-around
Disk based recording
eVLBI using fiber
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
• for the EVN and JIVE
• Feasibility of eVLBI:- Costs, timescales, logistics.• Standards:- Protocols, parameter tuning, procedures at
telescope and correlator.• New Capabilities:- Higher data rates, improved reliability,
quicker response.
• for GÉANT and the NRENs• To see significant network usage with multiple Gbit streams
converging on JIVE.• Minimum is three telescopes (not including Westerbork)• Must be seen to enable new science and not just solve an
existing data-transport problem• Real-time operation is seen as the ultimate aim, buffered
operation accepted as a development stage.
POC targets:
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Fibre pair from Amsterdam
Optical Splitters
GE lines to correlator
LX Optics converters
Cisco ONS 15252
Correlator Interface
Current status:
•JIVE: 6 lambdas via Netherlight, each capable of 1 Gbps
•1 Gbps connections to Westerbork, Torun, Onsala, (Metsahovi?), 2.5 to Jodrell
•155 Mbps to Arecibo
•Connection of Medicina at 1 Gbps planned later this year
Current status:
•JIVE: 6 lambdas via Netherlight, each capable of 1 Gbps
•1 Gbps connections to Westerbork, Torun, Onsala, (Metsahovi?), 2.5 to Jodrell
•155 Mbps to Arecibo
•Connection of Medicina at 1 Gbps planned later this year
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Protocols
• Use of existing protocols, available hardware
• TCP: maximal reliability, very sensitive to congestion
• UDP: connectionless, fast(er) but not reliable (enough?)
• Different protocols will be tested in the near future (tsunami?)
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Tuning issues
• Dramatic improvement of TCP performance possible by adjusting default settings:
• congestion window: product of roundtrip time and bandwidth
• size of queues between kernel and NIC• SACK (Selective Acknowledgment) implementation• MTU size (jumbo frames)• Interrupt moderation: cpu may be bottleneck
http://sravot.home.cern.ch/sravot
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Tuning issues
• Dramatic improvement of TCP performance possible by adjusting default settings:
• congestion window: product of roundtrip time and bandwidth
• size of queues between kernel and NIC• SACK (Selective Acknowledgment) implementation• MTU size (jumbo frames)• Interrupt moderation: cpu may be bottleneck
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Mem-mem transfer via patch
200
400
600
800
1000M
bps
Dell-SMJumboSM-Dell
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Disk2net2disk via patch, jumbo
300
400
500
600
0,00E+00 2,00E+06 4,00E+06
workbuf (bytes)
Mb
ps
26832020971524194304
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Disk2net-net2disk Dwingeloo-Bologna
0
100
200
300
400
Dw-BoBo-DwDw-Bo*Bo-Dw*
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Network stress test (iperf)
0
200
400
600
800
Dw-BoBo-Dw
Dw-TrTr-Dw
Dw-OnOn-Dw
Mbps
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Iperf (TCP) 30s intervals
0
100
200
300
400
Mbps
Tr-Dw
Dw-Tr
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
• Intel(R) SE7501BR2• Intel(R) 7501 chip set• Dual Intel(R) Xeon (single,
2.8G, 512K L2 cash installed)
• Dual channel DDR266 registered SDRAM (512MB)
• Dual peer PCIX 64/100, 2+2 slots32/33 PCI, 2 slots
• On-board TX1000 and T10/100 NICs
• On-board Ultra320 SCSI• On-board PATA
Mark 5 Motherboard Upgrade
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Memory-memory Disk2net-net2disk In2net-net2disk
In2net-net2out
UDP TCP UDP TCP TCP TCP
Bench via patch 930 250 256
Idem, jumbo frames 960 544 512
Bench via Amsterdam
500 360 256
Idem, jumbo frames 341 456
Westerbork-JIVE 867 680 256 256
Idem, jumbo frames 249 378
Bologna-JIVE 670 128 307
Jodrell-JIVE 50 70 64 256
Arecibo-JIVE 88 32
Torun-JIVE 800 260 256
Onsala-JIVE 177 256 256
JIVE-Haystack 612 71
Test results
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Memory-memory Disk2net-net2disk In2net-net2disk
In2net-net2out
UDP TCP UDP TCP TCP TCP
Bench via patch 930 250 256
Idem, jumbo frames 960 544 512
Bench via Amsterdam
500 360 256
Idem, jumbo frames 341 456
Westerbork-JIVE 867 680 256 256
Idem, jumbo frames 249 378
Bologna-JIVE 670 128 307
Jodrell-JIVE 50 70 64 256
Arecibo-JIVE 88 32
Torun-JIVE 800 260 256
Onsala-JIVE 177 256 256
JIVE-Haystack 612 71
Test results
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Huygens descent tracking
• Data recorded at Mopra and Parkes, flown to Sydney, transported via dedicated 1Gbps lightpath to Jive
• Peak transfer rate ~400Mbps• Use of modified tsunami protocol
• “slow start” algorithm• Transfer freezes occasionally• Quite buggy (but not everybody thinks so..)
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
eVLBI Transfer modes
• ftp-based: bandwidth not critical, has greatly improved response to technical problems at telescopes
• Dual buffered: recorded on Mk5 diskpack, transferred through disk2net and net2disk, played back at correlator
• Single buffered: streamed directly from formatters to diskpacks at JIVE
• Real-time: directly from formatters to correlator without disk buffering
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
eVLBI Transfer modes
• ftp-based: bandwidth not critical, has greatly improved response to technical problems at telescopes
• Dual buffered: recorded on Mk5 diskpack, transferred through disk2net and net2disk, played back at correlator
• Single buffered: streamed directly from formatters to diskpacks at JIVE
• Real-time: directly from formatters to correlator without disk buffering
First real-time eVLBI Image
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
MERLIN, March 2002 eVLBI, September 2004
IRC+10420
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Jumbo frames
• Tested with:• Onsala @4470• Torun @8192• Westerbork @9000
• Results inconclusive, but…• Transfer between Mk5’s (bench)
• UDP: <500 Mbps• TCP: >500 Mbps, ONLY with jumbo frames
• Other hardware platform needed?
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Network monitoring
• BWCTL installed at most stations• More measuring points along the way needed• Script-based tools for inspection and
visualisation of state of network
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Centralised control
• New operating model for EVN; correlator as integral part of network
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Centralised control
• New operating model for EVN; correlator as integral part of network
• Monitoring tools at stations• Reliability of correlator, stability of Mark5 units• Choice of platform?
EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE
Conclusions
• Mark5 not very well suited for eVLBI• Hardware limitations (interrupt conflicts, outdated
motherboards, proprietary hardware)• Software limitation (stuck to kernel 2.4 because of
Jungo drivers, proprietary software)
• Next platform?• support for 4Gbps, PCI Xpres?• as much off-the-shelf as possible• as simple as possible (no disks!)
• Dedicated lightpaths…