xNTD/SKAMP/LFD Correlator

xNTD/SKAMP/LFD Correlator

4th RadioNet Engineering Forum Workshop

Next Generation Correlators for Radio Astronomy and Geodesy

27-29 June 2006, Groningen, The Netherlands,

John Bunton

CSIRO ICT Centre, Sydney

www.ict.csiro.au

New Generation CorrelatorsGroningen 26-29 June 2006

Common Design

Developing a set of technologies that will be used in xNTD, SKAMP3 and LFD

Common correlator, VHDL code and hardware elements

This talk mainly the correlator and xNTD beamformer

Team (SKAMP3, xNTD and LFD) Uni Sydney/ ATNF - Ludi de Souza,

Uni Sydney Duncan Campbell-Wilson, John Russel, Chris Weimann, Adrian Blake

ATNF/ ICT Centre John Bunton, Jaysri Joseph

MIT Roger Cappallo, Brian Fanous

ANU Frank Briggs

and growing

www.ict.csiro.au


xNTD specification

Connected element telescope with 20-30 antennas within ~10km area

Each antenna generates 32 dual pol. beams giving a field of view of ~30 square degrees – frequency independent

Dynamic range ~50dB (8-bit A/D) aided by low RFI environment at Mileura WA, >60dB filterbanks

Frequency range below 1.7 GHz with 300 MHz BW

www.ict.csiro.au


xNTD Correlator

FX correlator, FPGA based using polyphase filterbank

Large design with 20-40 antennas x ~32 beams Input to beamformers 1.5 THz, to correlator 500 GHz (16bits/Hz)

6,000-15,000 baselines, each full Stokes

~1000 Tops/sec (mainly beamformer/filterbanks)

Output data rate 1-2 Gcorrelation/visability set

Full 300 MHz processed to full frequency resolution ~50,000 channel – 6 kHz resolution (no options)

BUT Low resolution and continuum by averaging channels

Daisy chain data for special processing

Later can trade bandwidth for higher resolution or time sampling

www.ict.csiro.au


Design

Beamformers and filterbank ~200 inputs generating 32 dual pol. beams

Signal transport from feeds optical analogue (TBD)

Beamformer location TBD

Design similar to SKAMP3 filterbank (Ludi’s talk)

Correlator based on correlation cell Highly configurable as to number inputs

If the beamformer is at the antenna crossconnect in fibre network and O/E interface

Needs data re-ordering (Ludi’s talk)

www.ict.csiro.au


xNTD Beamformer

Beamformer input 96 Dual polarisation inputs

300 MHz bandwidth

About 1 Terabit/sec input data rate per antenna

Each of 32 output beams is the sum of 20 to 96 inputs depending on frequency Would like to have data from all inputs to each unit that generates

a beam

Instead “Divide and Conquer”

Use the same topology as SKAMP3 filterbanks (Ludi’s talk)

www.ict.csiro.au


Filterbank to Beamformer

24 channelOS Polyphase

12x10 MHzcomplex

24 channelOS Polyphase 12x10 MHz

complex

~200 MS/s Real

~200 MS/s Real

192 inputs

192 feeds4MHz 800 channels

Dual 6.1 Gbits/s

12 sets

192 inputs10MHz band2k ChannelPolyphase

192 inputs10MHz band2k ChannelPolyphase

CrossConnectOutputs10 MHzfor all

signals toeach finefilterbank


Output 16x23MHzcomplex


Output 16x23MHzcomplex

~660 MS/s Real

~660 MS/s Real

Inputs from 96dual polarisation

feeds(192 signals)

Beamformer33x2 beam

23 MHz

Beamformer33x2 beam

23 MHz

32x2 beams309MHz309 MS/s

(4R,4I) complex163 Gbits/s

to centralcorrelator

20.6MHz4k ChannelPolyphase

20.6MHz4k ChannelPolyphase

CrossConnectOutputs23MHzfor all

signals toeach BF

SKAMP Filterbanks (one card cage)

xNTD Beamformer

www.ict.csiro.au


Correlator

The challenge was to develop a correlator concept the could be used in xNTD, SKAMP3 and LFD - 20-30, 192, and 512 antennas

Systolic array too inflexible xNTD 760 correlations/beam LFD 500,000 correlations

Approach developed – Correlation Cell Combination of multiply-accumulate and storage Each cell handles 256 correlations at a time 49,000 correlations per FPGA simultaneously 512 time sample short integration on chip

www.ict.csiro.au


Correlation Cell

Input 16+ pairs of data 4bit complex multiply in 18-bit

multiplier Accumulation to block RAM Calculate 256 correlation,

512 successive time samples Data reordering in filterbank

xNTD 4-7 cells for all correlations 30-70 MHz BW per FPGA

All baselines LFD 12, SKAMP3 1.5 FPGAs 1.2-1.5 MHz of bandwidth

Dual PortMemory

Complex multiplierResources one 18-bit multipliers

Dual 4+4 bit inputs - 9+9 bit output

Single Dual Port Memory - 2 x 256 36-bit words

+

Data loading

Two4+4 bit outputs

18+18 bit outputs

Dual PortMemory

MUX MUX

Latch Latch

8-bit shift register

Serial input data

36-bit shift re

gister

Serial output data

8-bit shift register

www.ict.csiro.au


22 antenna xNTD configuration

Store 22 or 22 pairs per correlation cells

Within group 231 correlation

Between groups 484 correlations Need two cells

CorrelationCell

Correlationswithin a

group of 22

CorrelationCell

Half thecorrelationsbetween twogroups of 22

CorrelationCell

Half thecorrelationsbetween twogroups of 22

CorrelationCell

Correlationswithin a

group of 22

First group of 22 inputs

Second group of 22 inputs

48 sets of 22-antenna correlators per FPGA (Virtex4 SX35)

Clock cycles 242 for 44 inputs = 5.5 clocks per input

Input data rate 48/5.5 = 8.7 bytes/clock

As systolic array 192 multipliers gives 13x13 array

Input data rate 3 times higher at 26 bytes/clock

www.ict.csiro.au


Board Manufacture Simplification (1)

Manufacture of correlator board a major task Examples SKAMP1, EVLA …

Correlation cell reduces input data rate into correlation chip

For xNTD correlation cell data rate 3 times less compared to systolic array

For individual correlation cell 2 sets of 16 inputs requires 256 clock cycles to process. Data rate reduction up to a factor of 16 This value approached for SKAMP3 and LFD

For xNTD need ~5 byte wide busses into each correlator FPGA. (input data rate higher than correlator clock)

www.ict.csiro.au


Board Manufacture Simplification (2)

Correlation cell also reduces data duplication

SKAMP1 4x4 systolic array, EVLA 8x8 systolic array Data duplication 8 in EVLA, higher in SKAMP1 due to array reuse

Each Correlation Cell process 256 correlations at once Can reduce size of systolic array by sqrt(256)=16

No data duplication on board for up to 150 antennas

Data duplication none for xNTD, 1.5 for SKAMP3, and 3.5 for LFD, LFD 12 FPGAs data input 1/sqrt(12) of total

Correlation cell leads to a large simplification of correlator board

www.ict.csiro.au


Putting it Together – The SKAMP protoype

Routing

AutocorrelationsLong Term Accumulations

CorrelationsCorrelator interface

www.ict.csiro.au


Input, Daisy Chain, Route, & Autocorrelate

Two FX20

Interconnection for high antenna number designs

Input 16 rocket I/O on unidirectional Infinband

Output 16 Rocket I/O unidirectional Infiniband

Can daisy chain modules for reuse of data in further processing modules or conform to standards, Infiniband …

Two FX20

www.ict.csiro.au


Compute Engine

Six SX35 FPGAs Input 24, Output 18 LVDS

per FPGA 1152 Correlation cells total Up to 294k correlations on

board at a time (256 per cell) Data re-ordering in filterbank

to achieve Process of 512 time values for

each frequency channel Then dump to LTA

www.ict.csiro.au


Long Term Accumulator

Number of Correlation require DRAM for storage

Data rate requires two DIMM modules for three SX35

Half of Correlation and LTA hardware

Three SX35 Two LX15 + DDR2 DIMM

www.ict.csiro.au


Estimated Performance

Correlator board clock rate 330MHz, 192 cells/FPGA, 6 FPGAs Board processing rate 400GCMACs/s = 2.8Tops/s

Power consumption ~100W Power efficiency 0.25W/GCMAC (4bit FX)

Filterbank board 3.2Gsample/s, two polyphase filterbanks ~32 operations per sample Board processing rate 100Gops/s (18 bit)

Power consumption ~60W Power efficiency 0.6W/Gop

www.ict.csiro.au


Conclusion

Common hardware, hardware modules and VHDL for xNTD, SKAMP3 and LFD

SKAMP3 in the lead with filterbank and correlator hardware well on the way

Initial Manufacture this year Correlator common to all using correlation cell to

gain required flexibility xNTD beamformer hardware starts July 06

LFD receiver will piggy back on this

Developing international project - distributed design team

xNTD/SKAMP/LFD Correlator

Documents

Transcript of xNTD/SKAMP/LFD Correlator