IBOB Data Acquisition and Network Streaming Jan Wagner et al. TKK / Metsähovi Radio Observatory 7th...

iBOB Data Acquisition and Network StreamingJan Wagner et alTKK Metsaumlhovi Radio Observatory

7th International eVLBI WorkshopShanghai 2008

12 June 2008 iBOB Data Acquisition and Network Streaming 7th eVLBI Workshop Shanghai Slide 2

Current e-VLBI Data AcquisitionIntroduction

bull Telescope data generated by Data Acquisition (DAQ) hardwarebull e-VLBI telescope data is sent via networks to correlator

a) capture DAQ data locally to RAID disk array send afterwardsb) send data in real-time capture and store or real-time process remotely

bull Current DAQ situation in the EVNbull Receiver and RF front-end raquo512 MHz RF bandwidthbull VLBA rack downconverters VCs amp BBCs le8 MHz base-band chbull Mark4-style AD digitizers max 32 Msschannel 2-bit rarr 64

Mbs chbull max theoretical rate 16 x 2-ch BBCs rarr 2 Gbits (512 MHz

RF)bull max rate in practice formatting Mark5 diskampNIC rarr le 1

Gbits

=gt Stuck at 256512 MHz 1 Gbits


Towards More DAQ Bandwidthbull Todayrsquos networks and PC systems easily support over 1G rates

=gt just a matter of providing antenna data samples faster=gt need a DAQ with more bandwidth

bull More RF bandwidth is beneficial for astro VLBIbull Correlationsynthesis yields better

sensitivity noise improved imagesbull Observation time per object could be

reduced more objects per sessionbull More bandwidth from bit reso allows

AGC band equalizer RFI mitigation hellip

bull To improve on the current DAQ bull Input bandwidth off-the shelf gigasample AD converter(s) bull Processing keep as wideband for astro fine channelization for geo()bull Output interface copper Ethernet UDP packet streams

bull For Metsaumlhovi DAQ development and tests used iBOB platform

Introduction

C Phillips e-VLBI Scientific Benefits ATNF CSIRO memo 2004


iBOB DAQ Hardware Platformbull UC Berkeleyrsquos iBOB FPGA board

bull Virtex II Pro 2vp50 with PPC CPUsbull two copper 10GbE ports (CX4)bull two expansion ports (Z-DOK)

bull Berkeley iADC boardbull Atmel AT84AD001bull dual-channel 8-bit 1 Gssch=gt 1 GHz MHz BW 16 Gbits board

bull Nat LMX2531 or ADI ADF4360bull ~9501080 MHz PLL for ADC fs

bull Similar board ROACH (meerKAT)available ldquosoonrdquo has 4 x 10GbE iBOB designs should drop in

bull For iBOB created several DAQ ldquofirmwarerdquo test designs

iBOB DAQ Description


iBOB IF-to-Ethernet Design ExampleiBOB DAQ Description


iBOB DAQ and Protocol for 10GbEbull Real-time e-VLBI data integrity less important than timely send

bull data loss factors fle10 tolerable image degradeslt=gt packet loss and reorder ok payload integrity non-criticallt=gt donrsquot bother to retransmit lost packetslt=gt UDP checksums unnecessary frame CRC32-gt210-9BER

bull congestion control none - or client-specified sample mode changes

bull The only essential points ndash implemented in Test-DAQ UDPbull must know which RF bandchannel a sample comes from

bull sample series from same channel placed into same UDP packetbull for lt60k channels can just send channel X to IPv46 port Y

bull samples must be time stampedbull trigger acquisition at 1PPS give packets a 64-bit packet sequencebull assume 4kB UDP =gt ambiguity after 45e15 samples 245 years 10Tss

bull rest is ldquoconstantrdquo a priori info doesnrsquot belong into the stream (YMMV)

~ 1 f

UDP Packet Format


iBOB DAQ Testsbull Local 10G LAN tests

bull Analog signal sampled with iBOB streamed as UDP to 10G LANbull UDP stream recording with lsquoudp2raidrsquo lsquoraw2raidrsquo (~tcpdump gulp)bull udp2raidrsquoed to RAID-0 at up to 4 Gbps (current RAID ~43G XFS)bull Data spectrum checks with Matlab and swspectrometer

iBOB Streaming Tests



Mh-On StreamingNetwork Tests

bull First tests on 1415th Maybull Supplied Onsala with test firmware

100 MHz clock 1024 Mss samplingbull IQ VLBI IF 90540 MHz -10dBmbull On streamed to Mh RAID-PCbull Mh PC busy with other computationshellipbull Recorded 16 rest with slow-PC lossbull Checked for and found PCAL

bull Second test 11th June bull used rdquoadc2tsunamirdquo prototest designbull sent 2 channels (IQ) to On

IBOB regwrite dest_ip_ich 0x82F20A05 130242105 OnsalaIBOB regwrite dest_port_ich 46330 I-channel to port 46330IBOB regwrite dest_ip_qch 0x82F20A05 hellipIBOB regwrite dest_port_qch 46331 hellipIBOB regwrite teng_wordsperframe 512 512x8 4kB UDPIBOB regwrite teng_decfactor 0 sample rate reduction 1(1+n)

32G (64G) 16 Gbps 2 channelscopy NORDUnet (httpwwwnordunetstats)



Mh-On Streaming Test - Route

copy NORDUnet (httpwwwnordunetstats)

copy OptoSUNET (httpstatssunetse)FunetrarrNordunetrarrSunetMh On



Mh-On Streaming Test - Traffic

2G 4G 78G



Mh-On Streaming Test - Errors



Mh-On Streaming Test - Commentsbull Borje Josefsson (Sunet chief tech)

ldquoAFAIK the tests started at 10AM Swedish time why cant I ever go and have a beer at that time -)rdquo

bull BJ made Internet Land Speed record in 2004 43 Gbps over 42 hops of shared networks from commercial ISPhttpprojsunetseLSR3-s

bull Will upgrade Sunet-Nordunet to 40G this week

bull Roy Molini (FunetCSC)ldquoIt is definitely exciting to see data streaming from Finland at such incredible rates across infrastructures built specifically for meeting just those demands The idea of Research Networking is clearly shownrdquo


Conclusion

Conclusionsbull Basic Mh iBOB designs allow to

bull digitize antenna signal different ratesbull stream as UDP to local or remote IP

bull On-gtMh RAID recording was lossless ldquoonlyrdquo at 16 Gbpsbull try again when PC really free - local tests give 4G to disk

bull Mh-gtOn network streamingbull No problems with 0 ndash 78 Gbps trafficbull Credit goes to Nordunet Sunet Funet it is straightforward

and welcome to stream multi-Gbps over InternetRN

bull Future test or demo stream and record a maser sourcewith 2-bit 8-bit 2 x 512 MHz BW analyze in swspectrometer


Conclusion

QuestionsAnswersbull Contact information

Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi

bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi

bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642


Conclusion

Refsbull Comparison of two frequency synthesizers LMX2531 and ADF4360

httpwwwmetsahovifienvlbiibobComparing-ADI-NATpdf

bull Multiplier-Free Band-Selectable Digital FiltersArnaud Santraine Seacutebastien Leprince Fred TaylorICASSP 2004

bull Programmable multiplier free channelizerArnaud Santraine Seacutebastien Leprince

Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15


Current e-VLBI Data AcquisitionIntroduction

bull Telescope data generated by Data Acquisition (DAQ) hardwarebull e-VLBI telescope data is sent via networks to correlator

a) capture DAQ data locally to RAID disk array send afterwardsb) send data in real-time capture and store or real-time process remotely

bull Current DAQ situation in the EVNbull Receiver and RF front-end raquo512 MHz RF bandwidthbull VLBA rack downconverters VCs amp BBCs le8 MHz base-band chbull Mark4-style AD digitizers max 32 Msschannel 2-bit rarr 64

Mbs chbull max theoretical rate 16 x 2-ch BBCs rarr 2 Gbits (512 MHz

RF)bull max rate in practice formatting Mark5 diskampNIC rarr le 1

Gbits

=gt Stuck at 256512 MHz 1 Gbits










Introduction




















~ 1 f

UDP Packet Format





















2G 4G 78G












Conclusion








Conclusion






Conclusion





Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15










Introduction




















~ 1 f

UDP Packet Format





















2G 4G 78G












Conclusion








Conclusion






Conclusion





Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15



















~ 1 f

UDP Packet Format





















2G 4G 78G












Conclusion








Conclusion






Conclusion





Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15





















2G 4G 78G












Conclusion








Conclusion






Conclusion





Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15












Conclusion








Conclusion






Conclusion





Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15


Conclusion






Conclusion





Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15


Conclusion





Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

IBOB Data Acquisition and Network Streaming Jan Wagner et al. TKK / Metsähovi Radio Observatory 7th...

Documents

Transcript of IBOB Data Acquisition and Network Streaming Jan Wagner et al. TKK / Metsähovi Radio Observatory 7th...