PELICAN Imaging Framework
1
PELICAN Imaging Framework Imaging on short timescales leads to very large correlator output data rates. In order to cope with these rates and produce updated calibration coefficients it is necessary to process the output data stream in real time. PELICAN b developed by the Oxford e-Research Center (OeRC) is a efficient and modular framework to process real time data streams. Data is split into parallel streams processed on CPU/GPUs to form images of the transiting sky and differential images for transient detection. Design of a 96 Element FX Correlator for the LOFAR-UK Station G. Foster 1,2 and K. Zarb Adami 1 1 University of Oxford 2 [email protected] The LOFAR-UK Station The international LOFAR station at Chilbolton Observatory consists of a 96 element low band array (LBA) and a 96 element high band array (HBA) connected to a single digital backend. The station was completed in September 2010 and has been commissioned for operation. The current backend is designed to create beamlets from the station antennas to be beamformed and correlated at the LOFAR correlator in Groningen, Netherlands. A Single Station Correlator Each LOFAR station has a limited calibration correlator which has been used for single station, widefield images throughout station commissioning as a diagnostic tool and for developing the imaging pipeline. This correlator cycles through the individual subbands to produce a single channel correlation on second timescales. A dedicated correlator is in development which can process a selectable portion of the band (7 MHz per module), provide further subband channelization, and output correlations on subsecond timescales. A key science goal for this instrument will be, among others, monitoring and imaging of short timescale transient events. In addition to the FPGA based correlator a CPU/GPU realtime imaging pipeline will be necessary to cope with the large output data rates. This instrument will interface with the current LOFAR RSP such that commensal observations can be performed while the station is being used during international LOFAR operations. Completed development of the correlator and imaging pipeline is expect in early 2012. RSP Interface The current station digital backend uses a XAUI loop for forming beams and correlator calibration using the 24 RSP boards. Approximately 25% of the total bandwidth is unused. Each XAUI contains four lines, three will continue to be part of the main loop and the remaining line will be connected to the station correlator. An RSP firmware modification will allow a selectable 7 MHz of the band to be output over a single XAUI line. This firmware modification will be completed by ASTRON and be used in the SuperTERP correlator for the AARTFAAC c project. ROACH II Hardware The next generation ROACH-II board designed by CASPER a /KAT is based on a Xilinx Virtex 6 FPGA. The CASPER design tools are built around reusable DSP blocks. Designs are built and simulated using Simulink and the Xilinx toolflow. Traditional HDL can also be incorporated into designs. Each board can process up to 60 Gbps using CX-4 adapted cards. A modular design will be used to compute correlations subsets of the band across multiple boards. Two ROACH-II boards will be required to compute the full Stokes correlation of all 96 elements for the 7 MHz band. AP0 AP1 AP2 AP3 BP LCU Inter board interface (IBI) Inter board interface (IBI) ring ring CEP serdes serdes RSP Board . . . RCU0 (Ant 0: pol X, pol Y) RCU95 (Ant 95: pol X, pol Y) (Ant 0…4: pol X, RSP23 (Ant 92…95: pol X, pol Y) . . . x24 XAUI0 XAUI1 XAUI2 XAUI3 XAUI4 XAUI5 XAU I XAU I XAU I Subband Splitter XAU I XAU I Antenna Reorder Corner Turn Windowed X Engine 0 (48 Dual pol taps, 9 subbanbs) Quantizati on Equalizati on (4 bits) Windowed X Engine 1 (48 Dual pol taps, 9 subbanbs) Vector Accumulato r 0 Vector Accumulato r 1 10 GbE 10 GbE XAU I XAU I XAU I Subband Splitter XAU I XAU I Antenna Reorder Corner Turn Windowed X Engine 2 (48 Dual pol taps, 9 subbanbs) Quantizati on Equalizati on (4 bits) Windowed X Engine 3 (48 Dual pol taps, 9 subbanbs) Vector Accumulato r 2 Vector Accumulato r 3 10 GbE 10 GbE Subband Chunker Subband Chunker RFI Flagging Calibratio n Local Sky Model FFT Imager CLEAN Image Database Differenti al Image Threshold Detection Stack Image … … … XAUI Line ROACH-II 1 ROACH-II 0 Correlator Specs Ant-pols 192 Baselines 18528 (Auto + Cross) Bandwidth 7 MHz / XAUI RSP Boards 24 (4 antennas w/ 2 pol per board) Subband Width 200 kHz Data Format 16 bit complex Integration Time ~10-100 ms Integration Size 141 MB Input Data Rate 60 Gbps Windowed X Engine Taps 1 auto + 48 cross (1/2 the number of antennas for a dual pol array) CMACs / Tap 4 (16 4x4 multipliers and 4 accumulators) Multiplier s / X Engine 784 Xilinx FPGA Device Virtex 6 SX475T Clock ~300 MHz Logic Slices 74400 DSP48e* 2016 GTX IO 36 (6.6 Gbps Max) *DSP48e contains a 25x18 multiplier and accumulator ROACH II IO/Memory FPGA Virtex 6 SX475T QDR 4 x 36 bit x 2M QDR II+ DRAM 144 bit DDR3 DRAM Interface 10 GbE (3 CX-4 or 4 SFP+) x 2 PELICAN Framework To interface the correlator with the current LOFAR digital backend a modification to the RSP firmware is required along with a passive XAUI combiner board. The current correlator design is implemented on 2 ROACH-II boards, each board has 3 CX-4 interfaces for input from the RSP and 4 SFP+ interfaces for interboard communications and 10 GbE output. The input data streams need to be reordered such that half of the subbands go into a single ROACH-II. A CASPER windowed X Engine is used to optimize resource utilization. The PELICAN pipeline will be used to form sky images along with providing a calibration routine which will be able to update the correlator phase and amplitude coefficients in real time. A local sky model will provide the initial calibration. A GPU based 2D FFT will be used to form the dirty image. For short integrations and low resolution, bright point sources will dominate the field. A short CLEAN loop can be used to isolate the sources based on the sky model. A differential comparison of images based on a number of time steps will be performed and a threshold detector will be used to find transient events. A slower stacking module will also be used to form a high dynamic range sky survey image. References a Collaboration for Astronomy Signal Processing and Electronics Research (http://casper.berkeley.edu) b Pipeline for Extensible, Lightweight Imaging and CAlibratioN (https://wiki.oerc.ox.ac.uk/svn/pelican/user/index.html, https://github.com/pelican/) c Amsterdam—ASTRON Radio Transients Facility and Analysis Centre (http://www.aartfaac.org/) XML Configurat ion DataBlob input data abstractio n PELICAN C++ Module Class Auxiliary DataBlob DataBlob output data abstractio n PELICAN Module Global Sky Model Phase/Amp Coefficien ts Subband Chunker Subband Chunker I m a g e b y J e a n - M a t h i s a G r a s s m e i e r , p r o d u c e d u s i n g d a t a f r o m L O F A R S t a t i o n F R 6 0 6 a t N a n c a y A S T R O N L O F A R - U K : C h i l b o t l o n O b s e r v a t o r y C A S P E R : R O A C H - I I
description
Design of a 96 Element FX Correlator for the LOFAR-UK Station. ring. AP0. serdes. Inter board interface (IBI). AP1. BP. LCU. CEP. AP2. Inter board interface (IBI). serdes. AP3. ring. G. Foster 1,2 and K. Zarb Adami 1. 1 University of Oxford 2 [email protected]. - PowerPoint PPT Presentation
Transcript of PELICAN Imaging Framework
PELICAN Imaging Framework
Imaging on short timescales leads to very large correlator output data rates. In order to cope with these rates and produce updated calibration coefficients it is necessary to process the output data stream in real time. PELICANb developed by the Oxford e-Research Center (OeRC) is a efficient and modular framework to process real time data streams. Data is split into parallel streams processed on CPU/GPUs to form images of the transiting sky and differential images for transient detection.
Design of a 96 Element FX Correlator for the LOFAR-UK StationG. Foster1,2 and K. Zarb Adami11University of [email protected]
The LOFAR-UK Station
The international LOFAR station at Chilbolton Observatory consists of a 96 element low band array (LBA) and a 96 element high band array (HBA) connected to a single digital backend. The station was completed in September 2010 and has been commissioned for operation. The current backend is designed to create beamlets from the station antennas to be beamformed and correlated at the LOFAR correlator in Groningen, Netherlands.
A Single Station Correlator
Each LOFAR station has a limited calibration correlator which has been used for single station, widefield images throughout stationcommissioning as a diagnostic tool and for developingthe imaging pipeline. This correlator cycles throughthe individual subbands to produce a singlechannel correlation on second timescales. Adedicated correlator is in developmentwhich can process a selectable portion ofthe band (7 MHz per module), providefurther subband channelization, andoutput correlations on subsecondtimescales. A key science goal for thisinstrument will be, among others,monitoring and imaging of shorttimescale transient events. Inaddition to the FPGA basedcorrelator a CPU/GPU realtimeimaging pipeline will be necessaryto cope with the large output datarates. This instrument will interfacewith the current LOFAR RSP such thatcommensal observations can beperformed while the station is beingused during international LOFARoperations. Completed developmentof the correlator and imagingpipeline is expect in early 2012.
RSP Interface
The current station digital backend uses a XAUI loop for forming beams and correlator calibration using the 24 RSP boards. Approximately 25% of the total bandwidth is unused. Each XAUI contains four lines, three will continue to be part of the main loop and the remaining line will be connected to the station correlator. An RSP firmware modification will allow a selectable 7 MHz of the band to be output over a single XAUI line. This firmware modification will be completed by ASTRON and be used in the SuperTERP correlator for the AARTFAACc project.
ROACH II Hardware
The next generation ROACH-II board designed by CASPERa/KAT is based on a Xilinx Virtex 6 FPGA. The CASPER design tools are built around reusable DSP blocks. Designs are built and simulated using Simulink and the Xilinx toolflow. Traditional HDL can also be incorporated into designs. Each board can process up to 60 Gbps using CX-4 adapted cards. A modular design will be used to compute correlations subsets of the band across multiple boards. Two ROACH-II boards will be required to compute the full Stokes correlation of all 96 elements for the 7 MHz band.
AP0
AP1
AP2
AP3
BP LCU
Inter board interface (IBI)
Inter board interface (IBI)
ring
ring
CEP
serdes
serdes
RSP Board
.
.
.
RCU0(Ant 0: pol X, pol Y)
RCU95(Ant 95: pol X, pol Y)
RSP0(Ant 0…4: pol X, pol Y)
RSP23(Ant 92…95: pol X, pol Y)
.
.
.x24
XAUI0
XAUI1
XAUI2
XAUI3
XAUI4
XAUI5
XAUI
XAUI
XAUI
Subband Splitter
XAUI XAUI
Antenna Reorder
Corner Turn
Windowed X Engine 0(48 Dual pol taps, 9
subbanbs)
Quantization Equalization
(4 bits)
Windowed X Engine 1(48 Dual pol taps, 9
subbanbs)
Vector Accumulator
0
Vector Accumulator
1 10 GbE
10 GbE
XAUI
XAUI
XAUI
Subband Splitter
XAUI XAUI
Antenna Reorder
Corner Turn
Windowed X Engine 2(48 Dual pol taps, 9
subbanbs)
Quantization Equalization
(4 bits)
Windowed X Engine 3(48 Dual pol taps, 9
subbanbs)
Vector Accumulator
2
Vector Accumulator
3 10 GbE
10 GbE
Subband Chunker
Subband Chunker
RFI Flagging Calibration
Local Sky Model
FFT Imager CLEAN
Image Database
Differential Image
Threshold Detection
Stack Image
…
…
…
XAUI Line
ROACH-II 1
ROACH-II 0
Correlator Specs
Ant-pols 192
Baselines 18528 (Auto + Cross)
Bandwidth 7 MHz / XAUI
RSP Boards 24 (4 antennas w/ 2 pol per board)
Subband Width 200 kHz
Data Format 16 bit complex
Integration Time ~10-100 ms
Integration Size 141 MB
Input Data Rate 60 Gbps
Windowed X Engine
Taps 1 auto + 48 cross (1/2 the number of antennas for a dual pol array)
CMACs / Tap 4 (16 4x4 multipliers and 4 accumulators)
Multipliers / X Engine
784
Xilinx FPGA Device
Virtex 6 SX475T
Clock ~300 MHz
Logic Slices 74400
DSP48e* 2016
GTX IO 36 (6.6 Gbps Max)*DSP48e contains a 25x18 multiplier and accumulator
ROACH II IO/Memory
FPGA Virtex 6 SX475T
QDR 4 x 36 bit x 2M QDR II+
DRAM 144 bit DDR3 DRAM Interface
10 GbE (3 CX-4 or 4 SFP+) x 2
PELICAN Framework
To interface the correlator with the current LOFAR
digital backend a modification to the RSP
firmware is required along with a passive XAUI
combiner board.
The current correlator design is implemented on 2 ROACH-II boards, each board has 3 CX-4 interfaces for input from the RSP and 4 SFP+ interfaces for interboard communications and 10 GbE output. The input data streams need to be reordered such that half of the subbands go into a single ROACH-II. A CASPER windowed X Engine is used to optimize resource utilization.
The PELICAN pipeline will be used to form sky images along with providing a calibration routine which will be able to update the correlator phase and amplitude coefficients in real time. A local sky model will provide the initial calibration.
A GPU based 2D FFT will be used to form the dirty image. For short integrations and low resolution, bright point sources will dominate the field. A short CLEAN loop can be used to isolate the sources based on the sky model. A differential comparison of images based on a number of time steps will be performed and a threshold detector will be used to find transient events. A slower stacking module will also be used to form a high dynamic range sky survey image.
Referencesa Collaboration for Astronomy Signal Processing and Electronics Research (http://casper.berkeley.edu)b Pipeline for Extensible, Lightweight Imaging and CAlibratioN (https://wiki.oerc.ox.ac.uk/svn/pelican/user/index.html, https://github.com/pelican/)c Amsterdam—ASTRON Radio Transients Facility and Analysis Centre (http://www.aartfaac.org/)
XML Configuration
DataBlob input data abstraction
PELICAN C++ Module Class
Auxiliary DataBlob
DataBlob output data abstraction
PELICAN Module
Global Sky Model
Phase/Amp Coefficients
Subband Chunker
Subband Chunker
Image by Jean-M
athisa Grassmeier, produced
using data from LO
FAR Station FR606 at Nancay
ASTRON
LOFAR-U
K: Chilbotlon Observatory
CASPER: ROACH-II
