ASKAP Beamformer
description
Transcript of ASKAP Beamformer
![Page 1: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/1.jpg)
ASKAP Beamformer
John BuntonCasper Workshop, Cape Town28 September – 2 October, 2009
![Page 2: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/2.jpg)
The Task
• 36 antennas with 94 dual pol feeds + 4 calibration/RFI signals (192 total)
• 304Mhz from each feed (BW/antenna 60GHz = ATA 100MHz)• Beamform data for each antenna to generate up to 36 dual
pol beams• Generate 18.5kHz frequency channels across 304MHz• Full Stokes FX correlator• Compute requirements over 1015 op/s• Also
• Min 32 tied array beams for Pulsar• 4 beams at VLBI clock rate• 8MHz with 0.5kHz channels
• …• Implemented in FPGA• Objective – minimise cost and power (~$4W/year)
![Page 3: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/3.jpg)
Progress
• First antenna - acceptance testing• First installation Boolardy – November 2009
• Field testing of 40 element Phased Array Feed• Digital system Engineering System 1
• Virtex 5
• Hardware complete
• ADC, coarse filterbank, signal transport - final testing
• Digital systems Critical Design Review 20 October• About to start work on Engineering system 2
• Virtex 6
• Mid 2010 two complete antennas with PAFs, ES2 Beamformers and Correlator
![Page 4: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/4.jpg)
The Team
• John Bunton Project Leader• Grant Hampson Project Engineer • Ilana Feain Project Scientist• Alan Ng Project Manager• Paul Roberts ADC• Joseph Pathikulangara Hardware/Firmware• John Tuthill Hardware/Firmware• Andrew Brown Hardware/Firmware• Jayasri Joseph Firmware • Tim Bateman Firmware • Ludi de Souza Firmware• Evan Davis Control PC's and Networks• Ron Koenig Parts and Library manager• Matt Shield Hardware Assembly / Testing • Raji Chekkala Hardware Assembly• Troy Elton Mechanical
• Other projects Signal Transport, SKAMP/MWA, Parkes Testbed, Pulsar processor, CABB, ICT centre work for some
![Page 5: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/5.jpg)
Beamforming Task
• At the focus of dish illumination of phase array depends on look angle
• See examples
• Detect power from phased array and sum
• Approx. conjugate match
• Do this for up to 36 look angles (beams)
• Weights different from beam to beam and are frequency dependent
• Focal spot size proportional to λ
![Page 6: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/6.jpg)
Beamforming
• Filter data so weight approximately constant across a frequency channel
• Too fine a channel - too many weights • Also 16k filterbank uses lots of BRAM compared to DSP48
• Increase FPGA resources needed
• Coarse filterbank – beamform – fine filterbank
• Beamforming reduces data rate• 94 dual pol. inputs generate ~36 beams at highest frequency
• Normally two stage processing is computationally more expensive than single stage
• Beamformer data rate reduction offsets this so cost is similar to single stage 16k filterbank for ASKAP
![Page 7: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/7.jpg)
Memory crunch
• Each fine filterbank processes over 360 1MHz channel, • 5 frequency channels, 36 dual pol beams
• Not enough memory on FPGA
• Still have a memory problem
• Store all beam data in DRAM • Read out long sequence of single beam and channel
• FPGA process only two signals at a time (dual pol beam)
• Single DRAM ~50Gbit/s of IO• 360 1MHz signals at 16 +16bits R/W = 24Gb/s
Beamformer5 1MHz channels36 dual pol beams
Re-order
DRAM
Re-order using
addressing
Fine FilterbankOne dual pol beam
One channelat a time
![Page 8: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/8.jpg)
Data precision
• System designed as a linear system will not clip or add significant quantisation noise
• If ADC not clipping then no coarse filterbank channel or beam will be overloaded
• A/D 8/bit• Output of coarse filterbank 14-bit• Narrow band RFI will concentrate into a single 1MHz channel• Number of bits used sufficient to keep system linear
• Output of fine filterbank 13-bit • Limited by data path 640Gbs per beamformer• Can have more bits a lower frequencies ( <1.4GHz)• Possibility of clipping on 18.5kHz channels
• At worst ~10 channels eg 200kHz flat spectrum interference
• 13 bit Correlator • 48+48 bit accumulations
![Page 9: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/9.jpg)
System Modules
• A/D is at the antenna, the data is then processed by the coarse filterbank and the beamformer/fine filterbank
• Next data is sent to the correlator, which must be at the central site.
• But where to put the division between antenna and central site?
• Dram interposed between Beamformer and Fine filterbank
A/DAt
antenna
Coarse Filterbank
Beamformer and Fine Filterbank
CorrelatorAt Central
Site
![Page 10: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/10.jpg)
Beamformer at the antenna
• Cheapest option as beamformer reduces data rate on optical fibre to central site
• 10Gbs per fibre
• But beamformer/fine filterbank is a major processing system, and
• Space limited in pedestal
• Harder to cool, maintain and debug at antenna
A/DAt
antenna
Coarse Filterbank
Beamformer and Fine Filterbank
CorrelatorAt Central
Site
64
![Page 11: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/11.jpg)
A/D only at the antenna
• All digital processing at central site• Increases data rate to central site, 128 fibres• A/D at the antenna still has FPGA for interface to optical fibre. • Must build separate coarse filterbank system
• Extra hardware development
• Four data interfaces AD-FPGA-Coarse FB-Beamformer-Correlator
A/DAt
antenna
Coarse Filterbank
Beamformer and Fine Filterbank
CorrelatorAt Central
Site
Interface FPGA
128
![Page 12: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/12.jpg)
Coarse filterbank data central site
• Use FPGA with A/D to do coarse filterbank• Digital hardware complexity at antenna same as A/D only
• Eliminates separate coarse filterbank system• Only 3 interfaces –reduced development time• Cost of extra data from antenna, balanced by
elimination of one subsystem• Still have antenna feed data at central site
• Possibility of SUMPLE calibration of antenna elements
• Use this option
A/DAt
antenna
Coarse Filterbank
Beamformer and Fine Filterbank
CorrelatorAt Central
Site
192
![Page 13: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/13.jpg)
Overall ASKAP system
Beamfomer and Fine filterbank
Antenna 1
192 fibre connection
Beamfomer and Fine filterbank
Antenna 2
192 fibre connection
A/D and Coarse
FilterbankAntenna 36
Beamfomer and Fine filterbank
Antenna 36
192 Analogue
inputs
192 fibre connection
Correlator Band 1
Correlator Band 2
Correlator Band 16
Tied Array Post processing
(VLBI and Pulsars)
To Computing Centre
Central Site at MRO
Cross connect between
beamformers and correlator
Represents a single Advanced TCA
chassisAll Identical
To Computing Centre
To Computing Centre
To VLBI network
or pulsar processors
A/Ds located in Antenna Pedestal
A/D and Coarse
FilterbankAntenna 36
192 Analogue
inputs
A/D and Coarse
FilterbankAntenna 36
192 Analogue
inputs
• Each block ~$100k
• $4M at antennas• $6M at central site• 0.6T ops/s at
antennas• 1.2T ops/s at
central site• 1.9Tb/s per
antenna• 76Tb/s to central
site
![Page 14: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/14.jpg)
ADC Hardware
• 4 8-bit ADCs at 768MS/s• Virtex 5 SX95T – Four 768 point oversampling filterbanks• Output 4 SFP+ 10Gbs single mode optical (up to 10 km)
![Page 15: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/15.jpg)
ADC subsystem (4 per antenna)
• 12 ADC cards (48 inputs)• Control card• Power card• Backplane with clock/control
distribution
Control board
![Page 16: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/16.jpg)
Cross connection
• In Beamformer have problem that many inputs must be brought together for processing
• 192 10G optical outputs from antenna • Data uni-directional
• Cost ~$50k at antenna for SFP+ modules
• Cross connect using COTS switch • Switch eg Cisco Nexus 7000 (12/24 line card) $600k??
• Power dissipation – ~10 kilowatt - $40k/year
• 192 SFP+ single mode $50k (excludes connection to beamformer)
• 10 year cost ~$1M/antenna
• Custom solution using ATCA• Cross point switches + ATCA back plane ~$10k
• Total power ~0.4kW in optical RX and cross connect
![Page 17: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/17.jpg)
DSP board and RTM
• DSP board, test jig and Rear Transition Module RTM shown
• Full system 16 DSP in ATCA with 16 RTM for input
• Input 12 x 10G optical per RTM• Sandwich card for testing• Command and control FPGA• Cross point switch• Four fully connected SX95T• Four DDR3 SODIMM• Eight 10G out (uni directional)
![Page 18: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/18.jpg)
ATCA 2.5Gbs Links• Input to RTM 12 x 10G produces 48 x 2.5Gbs• Cross-point switch connects from RTM to
backplane• ATCA full mesh backplane connects each
slot to every other slot with 8 serial data pairs (4-Rx + 4-Tx)
XilinxSX95T
CrossPoint
Switch
XilinxSX95T
XilinxSX95T
XilinxSX95T
4
4x4=1648
15x4=60to fabricinterface
2x10G
2x10G
2x10G
2x10G
4
44
4
444
4
444
4
444
ATCA backplane supplied with shelf
(RTM)
Pairs of FPGAs are also interconnected with a 64 p LVDS bus
DSP board
![Page 19: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/19.jpg)
Firmware
• Previous systems MOPS, CABB, NTD beamformer, Parkes test bed, SKAMP/MWA – build on existing firmware
• Some odd requirements • 768 point polyphase filterbank
• First implementation did not route
• 4x2.56 Gbs to 10G Fibre Channel optical to 4x2.56 GbsUni direction, 1% free for header and timing
• Need to control memory usage in data routing • Could easily blow out
• Efficient use of resources – particularly BRAM – critical• Evolving requirements 30 to 36 beams etc
![Page 20: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/20.jpg)
Firmware
• Firmware • Generally Simulink for DSP otherwise VHDL
• So far Monitor/Control, digitiser, coarse filterbank, optical links and data routing either working or final stage of debugging
• Beamformer and Array covariance design phase• Fine Filterbank, data reordering and DRAM –
borrow/modify from SKAMP/MWA• Minor subsystems CABB, SKAMP/MWA, Parkes Testbed• Complete prototype beamformer firmware in ~1 year
• ES1 Hardware in ~9 months
• Possible because of great team and previous systems• Previous system not only develop good ideas but also teach how not
to do it
![Page 21: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/21.jpg)
Thank you
ICT CentreJohn BuntonSenior Principle Research ScientistProject Engineer – ASKAP
Phone: +61 2 9372 4420Email: [email protected]: http://www.atnf.csiro.au/projects/askap/index.html
Contact UsPhone: 1300 363 400 or +61 2 9545 2176
Email: [email protected] Web: www.csiro.au
![Page 22: ASKAP Beamformer](https://reader035.fdocuments.net/reader035/viewer/2022081504/5681466c550346895db39166/html5/thumbnails/22.jpg)