The Square Kilometre Array - Engineers Australia...CSP is lead by NRC Canada with MDA managing...
Transcript of The Square Kilometre Array - Engineers Australia...CSP is lead by NRC Canada with MDA managing...
The Square Kilometre Array
CSIRO ASTRONOMY AND SPACE SCIENCE
John Bunton | CSP System Engineer
6-8 March 2013
SKA1 Low antenna station (Australia)
Square Kilometre Array
SKA1 MID antennas (South Africa)
SKA1 Overview
SKA1-low stations include Station Beamformer
CSP includes Correlator, Tied Array beamformers, Pulsar Search Engine and Pulsar Timing Engine
I work here
My work
The Low Correlator and Beamformer (CSP_Low.CBF) consortium are part of the larger Central Signal Processing (CSP) Consortium
CSP is lead by NRC Canada with MDA managing
CSP_Low.CBF is lead by CSIRO (Australia)
With ASTRON (Netherlands) and
AUT (New Zealand) as collaborator
SKA Science
Square Kilometre Array
Square Kilometre Array
Rainer Beck
Radio Telescope Sensitivity
Square Kilometre Array
Ideal Antenna and Receiver
Output power proportional to Temperature
27C =300K
-270C =3K
Power=1 Power=100
Square Kilometre Array
A real system
A real receiver has internal noise
Model as a resistor with an equivalent temperature Tr
Total power Tsky + Tr
Want to minimise Tr
A good system Tr=20K
Tr
Tsky
Signal = Tsky + Tr
Square Kilometre Array
Power in a resistor
Power = kTB k Boltzmann’s Constant = 1.38 x10-23 Joules/Kelvin
T Temperature of the resistor (20 Kelvin)
B Bandwidth
For 1Hz bandwidth, 20K system temperature Power at receiver input approx. 2.8 x 10-22 W
For a 1 MHz bandwidth its 2.8 x 10-16 W
A 1W transmitter at 100 km is ~10-11 W/m2
Square Kilometre Array
The Jansky
15 m antenna, area 177 m2, and after normalising for bandwidth
Equivalent incident power = 1.6 x 10-24 W/m2Hz
This is an inconveniently small quantity
So Astronomers invented the Jansky
1 Jansky = 10-26 W/m2Hz
Our antenna needs a 160 Jansky source for the power to double at the output.
Or a 1 nanoW transmitter at 100 km with at a 1MHz bandwidth
Square Kilometre Array
Antenna Signal to Noise
A strong source (radio star) is 10 Jansky Signal to Noise Ratio 0.06, -12dB (note our signal is less than unwanted noise)
In the 80s I worked for the Fleurs Radio Telescope just outside Sydney It could detect sources of ~1 mJansky
SNR 0.000 006 = -52dB
ASKAP can reach ~10 μJansky, SNR = -72dB (10-7.2)
SKA less than 1 μJansky, SNR less than -82dB (~1:1billion)
Square Kilometre Array
Interferometry
How the SKA work
Square Kilometre Array
The Interferometer
The SKA uses interferometers
Signals from pairs of antennas are Correlated
A correlations is a multiply and accumulate. We average to improve signal to
noise ratio
The conjugate * cancels phase offset
Signals from star are in-phase
u (wavelengths)
q
X
++
*
Square Kilometre Array
Drunken Walk
Consider a drunk who is trying to get home.
He cab stagger from one lamp post to the next.
But when he gets there he can’t remember which way to go. So each time he sets of in a random direction
After N lamp posts how far, on average, has he gone Answer √N
After 100 lamp post he has gone about 10
Square Kilometre Array
The Correlator
The signal from each antenna has two component The sky signal (wanted)
Antenna (System) noise (unwanted)
The system noise is randomly in-phase and out-of-phase. The signal adds in the same manner as a drunken walk - grows as √N
The sky signal is maintained in phase – grows proportional to N
Improvement ~√N
X
++
*
Square Kilometre Array
How good does it get
For the SKA bandwidths as much as 5 GHz 5 x 109 samples per second
Observations can be more than 24 hours 105 seconds
Number of sample = Bτ B Bandwidth, τ Time = 5 x 1014
SNR improvement √Bτ ~ 2 x 107 (73dB) For a single pair of antennas.
SKA has 20,000 (Mid) and 256,000 (Low) pairs of antennas
Square Kilometre Array
Warning: Complex Arithmetic coming up
Phase and phase difference are important
Product of two real sinusoids
cos(ωt) cos(θ ) = (cos(ωt + θ) + cos(ωt - θ))/2 Yuk!
Complex sinusoids
ejωt = cos(ωt) + j sin(ωt) j= √-1
think real in-phase, imaginary (j term) quadrature phase
Now product of two complex sinusoids
ejωt ejθ = ej (ωt + θ) Easy
Square Kilometre Array
Star brightness B at position l = cos(q)
(direction cosine)Distance between two antennas u
in wavelengts
Extra path length to one antenna of
D= u cos(q) = ul
Adds a phase term of ej2πul
Interferometry 101
Square Kilometre Array
uljBeoutput 2
u co
s(q)
u (wavelengths)
q
Complex
Correlator
Consider real part of output
= Bcos(2πul)
Cosine variation as θ varies l = cos(q) and change in θ small
Interferometer is sensitive to a “spatial” frequency A component of the sky image with a
sinusoidal variation across the sky
Interferometry 101
Square Kilometre Array
u (wavelengths)
q
X
++ Dump Sum once a second
q Period proportional to1/u
*
uljBeoutput 2
Summing over all l we find
If we measure at all separation we measure all “spatial” frequencies
We have measured the spectrum of the image
Measure different “frequencies” by having antennas at different spacings
Fourier transform “spectrum” to get image
Interferometry 101
Square Kilometre Array
uljBeoutput 2
dlelBuoutput ulj
2)()(
u (wavelengths)
q
X
++ Dump Sum once a second
q Period proportional to1/u
*
Correlation between pairs of antennas generate spatial frequency (Fourier) components of the image
N antennas gives N(N-1)/2 spacings (usually incomplete)
Transform to give (dirty) image (further processing needed)
Synthesis Mapping
Square Kilometre Array
0
20
40
0
20
400
0.5
1
300 400 500 600 700
200
300
400
500
Antenna location Fourier Components Image of Point Source
-200 0 200 400
-200
0
200
400
Musical Analogy
Square Kilometre Array
Fourier Components
-200 0 200 400
-200
0
200
400
Music score defines the notes (frequencies) and their position (time)
Fourier components defines spatial frequency and also a position (in this case and angle)
Need and orchestra to process the score into music
Need a complex computer program to convert Fourier component to an image
Spectral Response
Many radio astronomy signal are due to atomic transition – Neutral Hydrogen at 1.42 GHz, OH masers at 1.665 GHz…
Want spectrum of signal – Colour image, not just black and white
EITHER Measure correlation with different delays– Cross Correlation function and Fourier Transform – XF correlator
OR Split signal into subbands and measure correlation in each– Cross Power Spectrum - FX correlator
X
A
D
X
A
D
X
A
D
X
A
D
X
A
D
X
A
D
X
A
nD
Antenna 1 Antenna 2 Antenna 1
Frequency
Transform
Shift R
egis
ter
Shift R
egis
ter
Accumulator
Shift Register +x
Note: no change in
sample rate
Antenna 2
Frequency
Transform
Shift R
egis
ter
Shift R
egis
ter
Square Kilometre Array
Which to Use for the SKA
XF – complex multiplies per input sample = No. Channels ~104
FX – complex multiplies per input sample = 1 Plus overhead for frequency transform ~12 per complex antenna per sample
But number of correlations is N(N-1)/2 where N is number of antennas
FX computation more efficient when No. Channels > 1 + 24/N
X
A
D
X
A
D
X
A
D
X
A
D
X
A
D
X
A
D
X
A
nD
Antenna 1 Antenna 2 Antenna 1
Frequency
Transform
Shift R
egis
ter
Shift R
egis
ter
Accumulator
Shift Register +x
Note: no change in
sample rate
Antenna 2
Frequency
Transform
Shift R
egis
ter
Shift R
egis
ter
XF FX
Square Kilometre Array
Each antenna generates signal for two polarisations– X,Y linear or left and right circular
Must form all combination X1 X2*, Y1 Y2
*, X1 Y2*, Y1 X2
*,– Stokes parameters
Compute load for a single pair of antennas 4 multiplies and 4 adds per complex multiply
4 complex multiplies per data sample
1 data sample per Hz (complex data)
32 Gflops per GHz of bandwidth
For correlation (excluding image processing, beamforming etc)
• SKA Low 1.26 Pflops
• SKA Mid 2.95 Pflops
Polarisation and compute load
Square Kilometre Array
Filterbanks
From 1GHz to 1kHz
Square Kilometre Array
Generalised Filterbank
I/Q mixer on each channel, selects band centre
Filter to required bandwidth (impulse response ho(n), ho(t) analogue)
Down sample to required sample rate
Square Kilometre Array
Down convert
or ADC if
mixer and
filter
analogue
IQ Mixer
Low Pass filter
Cos(ωt)
Sin(ωt)
Single output from filterbank
Multiplication by complex sinusoid and impulse response can be interchanged.
Let w=e-jω
Square Kilometre Array
w0.n
Sum
w1.n
wm.n
x(n)
Xo
X1
XN-1
ho(n)
ho(n)
Sum
Sum
Single output, simplified
Need to do multiplication of input by ho(n) once
Take block of input data and value by value multiply x(n).ho(n)
On the right is a Fourier Transform
Choose w and m appropriately Have a Fast Fourier Transform
Square Kilometre Array
Fourier
Transform
Pre Multiply
Decimating the FFT
On 2GS/s data a 1MHz bandwidth filter and ho(n) ~20,000 samples long.
Therefore the channel spacing is 0.1MHz
We only want to keep every tenth output from the FFT. The rest can be discarded.
So why calculate them at all? Have
Want ever rth output:
Square Kilometre Array
1
0
)/2().().()(
N
n
nkNjenxnhkX
1
0
')/2().().()'(
N
n
nkNrjenxnhkX
Apologies for the Maths
If the length of the original impulse response and FFT is rM then
Interchanging the order of summation gives (ALMA memo 447)
Square Kilometre Array
1
0
')/2(1
0
1
0
'..2')/2(1
0
1
0
'))(/2(1
0
).().(
).().(
).().()'(
M
n
nkMrjr
m
M
n
kmjnkNrjr
m
M
n
kmMnNrjr
m
emMnxmMnh
eemMnxmMnh
emMnxmMnhkX
')/2(1
0
1
0
1
0
')/2(1
0
.)().(
).().()'(
nkMjM
n
r
m
M
n
nkMjr
m
emMnxmMnh
emMnxmMnhkX
Reduced size (one rth)
Fourier transform
Example r = 2
Consider an 8 point Fourier Transform and only every second output wanted
Delete every second row.
What is left is two 4-point Fourier Transforms
Sum the data before the transform and do one Transform
Square Kilometre Array
6
4
2
0
7
6
5
4
3
2
1
0
49423528
2460
35302520
4040
21147
246
15105
404
21181512
6420
7654
963
642
321
1
1
1
1
1111
1
1
1
1111
f
f
f
f
x
x
x
x
x
x
x
x
wwww
wwww
wwww
wwww
www
www
www
www
wwww
wwww
wwww
www
www
www Note shaded
blocks are the
same after
deletions
Square Kilometre Array
Graphical representation
Multiply input by prototype filter
Segment into r section of length M
Sum the r section to give output of length M
Fourier Transform
Generates a single output for each frequency channel
M-Point
FFT
M samples
Segment and sum
x
Filter h0(n)
Data
Overlapped
filter
segmentsFilter bank output
Square Kilometre Array
Critically sampled filterbank
Want to continuously process the data
Next output obtained by shifting the data) and applying the same process.
If the shift is by M samples then each input to the FFT is processing largely the same data. Gives a simple architecture
M-Point
FFT
M samples
Segment and sum
x
Filter h0(n)
Data
Overlapped
filter
segmentsFilter bank output
M-Point
FFT
po(m)
p1(m)
pM-1(m)
Pre-summation
Implementation of Critically Sampled Filterbank
SKA1 Low
to be built in Australia
Square Kilometre Array
Square Kilometre Array
Square Kilometre Array
SKA1 Low Location
Geraldton
Boolardy
Regulation of RFI
Geraldton
Low Population Density
gazetted towns: 0
population: “up to 160”
Geraldton
Australian Strengths
gazetted towns: 0
population: “up to 160”
Geraldton
Square Kilometre Array
SKA1 Low Station processing (Europe)
Bandwidth 300 MHz (Sky frequency 50-350 MHz)
512 antenna stations
Each station 256 dual polarisation antenna (log periodic)
Digitise 262k signal at 0.8 GS/s – total of 105 THz of bandwidth
Beamform to produce a 300MHz beam– Frequency domain processing.
– Apply filterbank 1024 channels (800Mhz to 781kHz)
– Beamforming in a single channels is just
– Multiply each input by a ejθ, adjust phase, and add
6 Petaflops
Total power ~1MW
SKA1 Low Correlator Base Requirements(CSIRO, ASTRON, AUT) Bandwidth 300 MHz (Sky frequency 50-350 MHz) From low station as 384 channels of 0.781kHz each
Full Stokes
512 antenna stations
Correlator compute load 1.25 Pflops equivalent About order of magnitude more than JVLA correlator (US biggest current)
About half of the proposed SKA1 Mid correlator
Basic operating mode is a “zoom mode” with at least 64k frequency channels –across 300MHz, ~172 per 781 kHz beamformer channel Resolution 4.07 KHz
Plus Zoom Modes
Four Independent Zoom Bands nominally either 4 MHz
8 MHz
16 MHz or
32 MHz each
– 256 possible combination
At least 16k channels in each zoom band
In addition “continuum” required at frequency within 300 MHz observing band, but not in a zoom band
Plus Subarraying
The telescope can operate as 1 to 16 INDEPENDENT subarrays
No antenna station can belong to the same subarray
Plus provision is made for a maintenance (17th) subarray
Up to 512 antennas when there is a single subarray
Each subarray operates independently
Last implies we cannot change firmware to change modes. FPGA based correlator
Additional outputs from SKA1_Low.CBF
16 voltage beams full bandwidth for pulsar timing
500 pulsar search beams at 128MHz bandwidth
Pulsar search can trade beams for bandwidth– 500 beams 128 MHz,
– 250 beams 2x128 MHz, or
– 133 beams 3x128 MHz
Again frequency domain beamforming.
Plus Multibeaming
Each subarray can apportion its 300MHz to 8 beams
Each beam is a contiguous section of bandwidth
Each beam pointing is independent of the other beams
Spectrum of beams can overlap Example 8 beams all covering 200-237.5 MHz on the sky
Large number of mode
“Normal” observing with zoom modes, subarraying, multibeaming, and independent scheduling blocks.
In the following will show how these are implemented in SKA1_Low Correlator and beamformer
Implementing Delay Correction
Must bring all signal to same wavefront before correlation in effect add delay of u.cos(θ) to second antenna
u co
s(q)
u (wavelengths)
q
Complex
Correlator
Step 1 Remove bulk delay by delay sampled data (~1us accuracy for LOW). Left with fractional delay error of up to 0.5us across ~1MHz
delay = -dθ/dω (dω.delay = 180 degrees)Phase changes by up to 180 deg across ~1MHz
Fine Delay Correction with Fractional Time Delay filters
FIR filter with values sampled from a continuous time filter.
Change the initial sampling point of continuous time filter (red + and blue dot are a half sample different in delay
Problems Residual amplitude and phase errors, possibly
delay errors
– Changes with delay value
– Improve by making filter longer
In practice filter length large – more compute intensive than a filterbank
Filter Impulse Response
Fine Delay Correction with Phase Slope
In frequency domain delay is equivalent of a phase slope
“Normal” frequency resolution of correlator requires ~256 channel filterbank. Phase slope across a fine channel (4 kHz) is less than 1 degree (±0.5deg), average error is zero
Apply correct phase, as a function of time, to each fine channel
Problem: Antennas with different velocities = different rate of change of delay = Doppler Shift, Different Doppler shifts on different antennas (less than 10Hz)
Fine channels select different part of the sky spectrum – correlation loss
Solution apply Doppler correction before fine filterbank
Implementing Spectral Resolution
Six spectral resolution required:– Continuum, “normal” (4kHz) and four zoom modes
Solution 1 Five Separate filterbanks– Implement a separate filterbank for each resolution – No filterbank for Continuum so fine delay must be fractional delay filter
Solution 2 Combine zoom filterbanks– Zoom filterbanks are power of 2. Build variable length filterbank – For data from stations Normal mode is resolution not a power of 2, separate
filterbank– Have two filterbanks and fractional delay filter
Solution 3 Frequency averaging– Implement filterbank at finest frequency resolution and average in frequency to
achieve other resolutions– 4096 channel filterbank -226Hz resolution,– Average 1, 2, 4, 8 channels gives 226, 452, 904Hz, 1.8kHz - all zoom modes– Average 18 channels give “normal” observing of 4kHz, average 3456 channels gives
continuum (781kHz)
SKA_Low solution
Use frequency averaging to implement all resolutions A single filterbank,
Uniform data flow into correlator (same for all resolutions)
Use phase slope method for fractional delay 4096 channel filterbank = 0.02degree maximum phase error across channel
Very small phase error
No added amplitude error
Small compute load
Correlation – six frequency resolutions
Correlator 1.25PFlops require 192 FPGAs
Data output from filterbank FPGAs on 8 x 25Gbps links
One link to group of 24 – cross connect with group of 24
Each FPGA process 1.56 MHz
Frequency averaging
for different frequency res.
24-way cross
connect
Correlator Frequency
Accumulation
Correlations
to SDPGearbox
Visibility Sharing &
Packetisation
Passive
Optical
Backplane
Passive
Optical
Backplane
From
Station
FPGAs
HMC
Memory
HMC
Memory
Multibeaming
For correlator difference in beam is simply a different delay polynomial (description of delay as a function of time)
Each antenna station is in a single subarray.
Eight beam per subarray require 8 delay polynomials for each antenna.
For each coarse channel apply the appropriate delay polynomial.
All extra complexity is with the filterbanks
Correlator independent
Subarraying
If there is NO SUBARRAYING correlation between all pairs of antennas is performed.
In this design all correlation for single frequency channel occurs in a single FPGA and are stored to DRAM
A subarray selects a subset of all possible correlation
Arrange data in DRAM so that a block read are for correlation where one the of antenna station inputs is common
SUBARRAYING becomes For each antenna station in a subarray read blocks for that antenna station
Select the subset of correlation for the subarray and transmit for data processing
Hardware Proposal
CSIRO-ASTRON Colaboration
Square Kilometre Array
Hardware implementationGemini Processing board
CSIRO ASTRON(Netherlands) collaboration
Optical Transceiver and FPGA SERDES number now allow
FULL OPTICAL DATA CONNECTIONS
Proposed processing board is a simple single board unit with
• A single FPGA
• Parallel Optical transceivers for I/O
(48 fibres total)
• Four HMC for data storage
• SFP+ 10GE for Monitor&Control
Square Kilometre Array
Perentie Rack Unit
Packaging
• Four Gemini processing boards
• Liquid cooling
• COTS Power
• Optical to Front Panel
• In a standard 1U rack unit
• 32Tflop equivalent
• 48 Rack units for correlation
Square Kilometre Array
Interconnecting Perentie Rack Units
If Front Panel had single fibres simply route each fibre to destination
But to keep density down have multi-fibre ribbons (up to 24/ri)
COTS Passive Optical Backplanes connect a single fibre from one ribbon to another. User writes specifications and has it built. Similar to acquiring circuit boards.
Square Kilometre Array
Putting it all together
Square Kilometre Array
Thank youCASSJohn BuntonSKA1 CSP System Engineer
t +61 2 9372 4420e [email protected] www.atnf.csiro.au/projects/askap
PO BOX 76 EPPING, 1710, AUSTRALIA
Example ASKAP
Major Cross connects for SKA1_Survey Filterbank to Beamformer ~2Tb/s per antenna
Beamformer to Correlator ~100Tb/s
Square Kilometre Array