PELICAN Imaging Framework

Imaging on short timescales leads to very large correlator output data rates. In order to cope with these rates and produce updated calibration coefficients it is necessary to process the output data stream in real time. PELICANb developed by the Oxford e-Research Center (OeRC) is a efficient and modular framework to process real time data streams. Data is split into parallel streams processed on CPU/GPUs to form images of the transiting sky and differential images for transient detection.

Design of a 96 Element FX Correlator for the LOFAR-UK StationG. Foster1,2 and K. Zarb Adami11University of Oxford2griffin.foster@astro.ox.ac.uk

The LOFAR-UK Station

The international LOFAR station at Chilbolton Observatory consists of a 96 element low band array (LBA) and a 96 element high band array (HBA) connected to a single digital backend. The station was completed in September 2010 and has been commissioned for operation. The current backend is designed to create beamlets from the station antennas to be beamformed and correlated at the LOFAR correlator in Groningen, Netherlands.

A Single Station Correlator

Each LOFAR station has a limited calibration correlator which has been used for single station, widefield images throughout stationcommissioning as a diagnostic tool and for developingthe imaging pipeline. This correlator cycles throughthe individual subbands to produce a singlechannel correlation on second timescales. Adedicated correlator is in developmentwhich can process a selectable portion ofthe band (7 MHz per module), providefurther subband channelization, andoutput correlations on subsecondtimescales. A key science goal for thisinstrument will be, among others,monitoring and imaging of shorttimescale transient events. Inaddition to the FPGA basedcorrelator a CPU/GPU realtimeimaging pipeline will be necessaryto cope with the large output datarates. This instrument will interfacewith the current LOFAR RSP such thatcommensal observations can beperformed while the station is beingused during international LOFARoperations. Completed developmentof the correlator and imagingpipeline is expect in early 2012.

RSP Interface

The current station digital backend uses a XAUI loop for forming beams and correlator calibration using the 24 RSP boards. Approximately 25% of the total bandwidth is unused. Each XAUI contains four lines, three will continue to be part of the main loop and the remaining line will be connected to the station correlator. An RSP firmware modification will allow a selectable 7 MHz of the band to be output over a single XAUI line. This firmware modification will be completed by ASTRON and be used in the SuperTERP correlator for the AARTFAACc project.

ROACH II Hardware

The next generation ROACH-II board designed by CASPERa/KAT is based on a Xilinx Virtex 6 FPGA. The CASPER design tools are built around reusable DSP blocks. Designs are built and simulated using Simulink and the Xilinx toolflow. Traditional HDL can also be incorporated into designs. Each board can process up to 60 Gbps using CX-4 adapted cards. A modular design will be used to compute correlations subsets of the band across multiple boards. Two ROACH-II boards will be required to compute the full Stokes correlation of all 96 elements for the 7 MHz band.

AP0

AP1

AP2

AP3

BP LCU

Inter board interface (IBI)

Inter board interface (IBI)

ring

ring

CEP

serdes

serdes

RSP Board

.

.

.

RCU0(Ant 0: pol X, pol Y)

RCU95(Ant 95: pol X, pol Y)

RSP0(Ant 0…4: pol X, pol Y)

RSP23(Ant 92…95: pol X, pol Y)

.

.

.x24

XAUI0

XAUI1

XAUI2

XAUI3

XAUI4

XAUI5

XAUI

XAUI

XAUI

Subband Splitter

XAUI XAUI

Antenna Reorder

Corner Turn

Windowed X Engine 0(48 Dual pol taps, 9

subbanbs)

Quantization Equalization

(4 bits)

Windowed X Engine 1(48 Dual pol taps, 9

subbanbs)

Vector Accumulator

0

Vector Accumulator

1 10 GbE

10 GbE

XAUI

XAUI

XAUI

Subband Splitter

XAUI XAUI

Antenna Reorder

Corner Turn

Windowed X Engine 2(48 Dual pol taps, 9

subbanbs)

Quantization Equalization

(4 bits)

Windowed X Engine 3(48 Dual pol taps, 9

subbanbs)

Vector Accumulator

2

Vector Accumulator

3 10 GbE

10 GbE

Subband Chunker

Subband Chunker

RFI Flagging Calibration

Local Sky Model

FFT Imager CLEAN

Image Database

Differential Image

Threshold Detection

Stack Image

…

…

…

XAUI Line

ROACH-II 1

ROACH-II 0

Correlator Specs

Ant-pols 192

Baselines 18528 (Auto + Cross)

Bandwidth 7 MHz / XAUI

RSP Boards 24 (4 antennas w/ 2 pol per board)

Subband Width 200 kHz

Data Format 16 bit complex

Integration Time ~10-100 ms

Integration Size 141 MB

Input Data Rate 60 Gbps

Windowed X Engine

Taps 1 auto + 48 cross (1/2 the number of antennas for a dual pol array)

CMACs / Tap 4 (16 4x4 multipliers and 4 accumulators)

Multipliers / X Engine

784

Xilinx FPGA Device

Virtex 6 SX475T

Clock ~300 MHz

Logic Slices 74400

DSP48e* 2016

GTX IO 36 (6.6 Gbps Max)*DSP48e contains a 25x18 multiplier and accumulator

ROACH II IO/Memory

FPGA Virtex 6 SX475T

QDR 4 x 36 bit x 2M QDR II+

DRAM 144 bit DDR3 DRAM Interface

10 GbE (3 CX-4 or 4 SFP+) x 2

PELICAN Framework

To interface the correlator with the current LOFAR

digital backend a modification to the RSP

firmware is required along with a passive XAUI

combiner board.

The current correlator design is implemented on 2 ROACH-II boards, each board has 3 CX-4 interfaces for input from the RSP and 4 SFP+ interfaces for interboard communications and 10 GbE output. The input data streams need to be reordered such that half of the subbands go into a single ROACH-II. A CASPER windowed X Engine is used to optimize resource utilization.

The PELICAN pipeline will be used to form sky images along with providing a calibration routine which will be able to update the correlator phase and amplitude coefficients in real time. A local sky model will provide the initial calibration.

A GPU based 2D FFT will be used to form the dirty image. For short integrations and low resolution, bright point sources will dominate the field. A short CLEAN loop can be used to isolate the sources based on the sky model. A differential comparison of images based on a number of time steps will be performed and a threshold detector will be used to find transient events. A slower stacking module will also be used to form a high dynamic range sky survey image.

Referencesa Collaboration for Astronomy Signal Processing and Electronics Research (http://casper.berkeley.edu)b Pipeline for Extensible, Lightweight Imaging and CAlibratioN (https://wiki.oerc.ox.ac.uk/svn/pelican/user/index.html, https://github.com/pelican/)c Amsterdam—ASTRON Radio Transients Facility and Analysis Centre (http://www.aartfaac.org/)

XML Configuration

DataBlob input data abstraction

PELICAN C++ Module Class

Auxiliary DataBlob

DataBlob output data abstraction

PELICAN Module

Global Sky Model

Phase/Amp Coefficients

Subband Chunker

Subband Chunker

Image by Jean-M

athisa Grassmeier, produced

using data from LO

FAR Station FR606 at Nancay

ASTRON

LOFAR-U

K: Chilbotlon Observatory

CASPER: ROACH-II