1

New version of high performance Compute Nodefor PANDA Streaming DAQ system

Jingzhou ZHAO, Member, IEEE, Zhen-An LIU, Member, IEEE, Wenxuan GOND, Pengcheng CAO,Wolfgang Kuehn, Member, IEEE, Thomas Gessler, Bjoern Spruck

Abstract—PANDA is one of the major experiments currentlyunder construction at FAIR/Darmstadt. Its focus is physics withhigh intensity and high quality anti-proton beams with momentaup to 15 GeV/c. Event rates up to 20MHz, and a typical eventsize between 1.5 KB and 4.5 KB. lead to data rates as high as 200GB/s. A trigger-less streaming DAQ system is introduced in thispaper, featuring event filtering based on FPGAs and a CPU/GPUfarm. The Compute Node (CN) is the central board FPGAbased component in this system. A new version of the ATCAbased CN is presented. Its main features include high speed datatransmission, massive data buffering capabilities to support largelatency for complex decion algorithms, high performance dataprocessing and ethernet connectivity. First test results with aprototype are presented.

Index Terms—PANDA, DAQ, Compute Node, ATCA, MGT,DDR4, Ethernet Switch.

I. INTRODUCTION

PANDA is a next generation hadron physics detectorplanned to be operated at the future Facility for An-

tiproton and Ion Research (FAIR) at Darmstadt, Germany. Itwill use cooled antiproton beams with a momentum between1.5GeV/c and 15 GeV/c interacting in the high-energy storagering (HESR) with a hydrogen cluster jet or a high frequencyfrozen hydrogen pellet target reaching a peak luminosity of upto 2x1032cm−2s−1[1], corresponding to a data rate of up to200 GB/s. However, most of the interesting physics channelshave cross sections which are several orders of magnitudesmaller than the total inelastic cross section. Thus, eventfiltering is an integral part of the DAQ system.

Manuscript received June, 2018.This work was supported in part by National Natural Science Foundation

of China (11435013, 11405196), National Key Program for S&T Researchand Development(Grant No.2016YFA0400104)and BMBF (05P12RGFPF).

Jingzhou.ZHAO is with State Key Laboratory of Particle Detection andElectronics, Institute of High Energy Physics, CAS, Beijing 100049 CHINA(e-mail:zhaojz@ihep.ac.cn).

Zhen-An LIU is with State Key Laboratory of Particle Detection andElectronics, Institute of High Energy Physics, CAS, Beijing 100049 CHINA(e-mail:liuza@ihep.ac.cn).

Wenxuan GONG is with State Key Laboratory of Particle Detection andElectronics, Institute of High Energy Physics, CAS, Beijing 100049 CHINA(e-mail:gongwx@ihep.ac.cn).

Pengcheng CAO is with State Key Laboratory of Particle Detection andElectronics, Institute of High Energy Physics, CAS, Beijing 100049 CHINA(e-mail:caopc@ihep.ac.cn).

Wolfgang Kuehn is with II. Physikalisches Institut, Justus-Liebig-UniversityGiessen, 35392 Giessen, GERMANY(Wolfgang.Kuehn@exp2.physik.uni-giessen.de)

Thomas.Gessler is with II. Physikalisches Institut, Justus-Liebig-UniversityGiessen, 35392 Giessen, GERMANY(Thomas.Gessler@exp2.physik.uni-giessen.de)

Bjoern Spruck is with Institute of Nuclear Physics, Johannes GutenbergUniversity Mainz, 55128 Mainz, GERMANY(bspruck@uni-mainz.de)

II. NEW CONCPET DAQ FOR PANDAThe PANDA experiment adopts a streaming data acquisition

concept in order to allow as much flexibility as possible whichthe complex and diverse physics objectives of the experimentrequire, and also to fully exploit the high interaction rateof up to 2x107 events/s. Each sub-detector system runs au-tonomously in a self-triggering mode, yet synchronized witha high-precision time distribution system, SODANET. Zero-suppressed and physically relevant signals are concentratedand transmitted to a high-bandwidth computing network, fea-turing two layers. Layer 1 consists of FPGA based ComputeNodes (CN) designed in the ATCA standard. The modularand scalable layer features high-speed interconnects via opticallinks and ATCA full-mesh backplanes.

Furthermore, large local DDR4 memories attached to eachFPGA permit the implementation of algorithms with largelatency. The CN layer has two functions. As a first stage, burstbuilding and event building is performed. The data stream fromeach PANDA - subsystem for a single burst of antiprotons iscollected and stored in a single local DDR4 memory. Usinga high-precision timing system (SODANET), data originatingfrom individual projectile-target interactions are identified andassembled into events. In a second stage, course event filteringis performed. Here, high-level algorithms for feature extractionperform tasks such as tracking, EMC cluster recognition andparticle identification. The raw data is converted into a set of4-vectors and a first stage of event filtering is performed. Themain task of this stage is the rejection of events which are notof interest for a particular experiment.

The raw data of the remaining events is transferred to theFAIR Computing server farm, where a more refined recon-struction of the events will be performed in software runningon CPUs and GPUs. After further rejection of unwantedevents, the remaining data is transferred to mass storage forfurther offline analysis. The data acquisition system aims foran online data reduction of factor 100 – 1000, where theperformance and the partitioning of the rejection dependsstrongly on the properties of the PANDA detector and thephysics channels of interest.

Schematic views of the system are shown in figures 1 and2.

III. PANDA STREAMING DAQPANDA DAQ system is shown as Fig.2.[2] It is consisted

of FEE, Global timing distribution(SODANET), data concen-tratos and buffering, L1 network/Feature, high speed network,L2 network for final event reconstruction and filtering.

arX

iv:1

806.

0912

8v2

[ph

ysic

s.in

s-de

t] 2

Jul

201

8

2

Fig. 1. New concept of DAQ system.

Signals from detectors are self-triggered and sampled inFEE. The self-trigger in the FEE is based on the signalamplitude and time-stamped to allow assembly of events ata later stage. Pipelined readout mode is used to avoid deadtime.

SODANET, the global time distribution system providesabsolute timing stamping for the whole system as well as datapath for slow control functionality. Large data buffers are usedto provide enough latency for accurate event reconstructionand event filtering.

In the FPGA based L1 network, ,information on particlemomenta and particle identification are extracted. A first, morecourse reconstruction of events is performed and unwantedevents are rejected. A more refined event reconstruction in theL2 network using essentially offline-grade algorithms.

For the L1 network, ATCA architecture is used, providinghigh speed data transmission with full-mesh backplanes, aswell as scalability. The Compute Node plays a key role fea-turing high performance for data transmission and processing.

IV. COMPUTE NODE

Compute Node[3] is a high speed data transmission anddata processing module designed to be complaint with theATCA standard. It consists of an ATCA carrier board CNV3, four AMC cards xFP V4 and one RTM card[4]. CNV3 is designed based on Xilinx Virtex-4 FPGA, and features2GB DDR2 memory, 16 MGT channels to each other slot ofthe backplane, 1 Gigabit Ethernet port to RTM, JTAG Hubfor AMC cards, UART hub for AMCs and carrier FPGA,and ATCA compatible IPMI controller for IPMC(IntelligentPlatform Management Control). xFP V4 is designed based onXilinx Virtex-5 FX70T FPGA features 4GB DDR2, 8 MGTchannels( 2 channels to SPF+ port and 6 channels to AMCconnector), 1 Gigabit Ethernet port, UART port and AMCMMC(Module Management Control). Since the Vitex4 FPGA

Fig. 2. PANDA TDAQ system structure.

Fig. 3. Structure of Compute Node Version 4.0.

is is outdated, a new version of the Compute Node has beendesigned for the PANDA DAQ system, which is defined asCN V4.0. In the CN V4.0, the performance of CN is highlyimproved. The structure of CN V4.0 is shown in Fig.3. CNfeatures a Xilinx Ultrascale FPGA xcku060 chip and up to16 GBytes DDR4 memory. 4 single width AMC cards canbe installed on the CN V4.0 carrier board. 12 backplaneMGT ports are used for high speed data transmission withinthe shelf. The line rate of each port supports up to 16.3Gbps. A 10 Gigabit Ethernet link is provided for data outputtransmission. A Gigabit Ethernet Switch connects the Ethernetports of the AMC cards, the ATCA Ethernet Switch, and anRTM Ethernet port switching, which can be used as slowcontrol channel. IPMC/MMC controllers are compatible withthe xTCA specification[5]. Compute Node V4.0 is finished, asshown in Fig.4.

V. MGT TESTING RESULT

On first prototype version of CN V4.0, speed grade 1 FPGAare used in which the MGT rate is limited to 12.5Gbps[6].

3

Fig. 4. CN V4.0 board. Carrier board and RTM card are new designed. AMCcards are xFP V4.0. CNV4.0 AMC slots are pin compatible with xFP V4.0.

Fig. 5. Eye diagram for 12 Channel Backplane at 12.5Gbps.

MGT channels are tested on the backplane channel andthe AMC channels. The system comprises a Schroff Ad-vancedTCA 450/40 series shelf for the backplane MGT test.This shelf has two ATCA slots and its backplane and con-nectors support 25 Gbps per channel. 12 Backplane channelsare point-to-point connected between two slots. IBERT wasused for the MGT test which is the Xilinx MGT testing suite.24 hours continuous testing was done. No error was detectedon any channel. BER(Bit Error Rate) is less than 10−15. Eyediagrams of each channel are scanned by Vivado and shownin Fig.5.

CN AMC MGT channels are connections between CN car-rier FPGA MGT and AMC FPGA MGT. An AMC loopbackcard was designed for this testing, which is as shown in Fig.6.The AMC connector and the AMC B+ connector on CN carrierboard supports only up to 10Gbps. The testing method was thesame as for the CN backplane MGT test. 12 hours continuoustesting was performed and BER is less than 2x10−15. The eyediagram is shown as Fig.7.

VI. CONCLUSION

The concept of a trigger-less streaming DAQ system forthe PANDA experiment at FAIR/Darmstadt is presented. TheCompute Node is central FPGA nased board designed to becompatible with the ATCA standard. It features high speeddata transmission, massive data buffering, high performance

Fig. 6. CN AMC MGT channel testing setting up.

Fig. 7. Eye diagram for AMC MGT channel at 10Gbps.

data processing and Ethernet connectivity. A new version ofCN carrier board using Xilinx Kintex Ultrascale technologyhas been designed. All essential functions are tested success-fully.

REFERENCES

[1] PANDA collaboration, Technical Design Report for PANDA Electromag-netic Calorimeter (EMC), 2008

[2] Qiang WANG,Study of a Novel Trigger and Data Acquisition Systemfor the PANDA Electromagnetic Calorimeter Detector, A Thesis forthe Doctoral Degree of Nuclear Technology and Its Applications in theGraduate School of Chinese Academy of Sciences, April. 2011, pp. 15-25.

[3] Xu, H., Liu, Z. -a., Wang, Q., Zhao, J., Jin, D., Khn, W., Liu, M.(2012). An ATCA-based High Performance Compute Node for Trigger andData Acquisition in Large Experiments. Physics Procedia, 37, 18491854.https://doi.org/10.1016/j.phpro.2012.02.509

[4] Zhao, J., Liu, Z.-A., Xu, H., Gong, W., Wang, K., Lin, H., Deng, F.(2015). A general xTCA compliant and FPGA based data processingbuilding blocks for trigger and data acquisition system. In 2014 19thIEEE-NPSS Real Time Conference, RT 2014 - Conference Records.https://doi.org/10.1109/RTC.2014.7097528

[5] Z.-A. LIU, xTCA for Physics Initiatives for DAQ at IHEP Beijing, xTCAworkshop, Jun. 9-10, 2012.

[6] Kintex UltraScale FPGAs Data Sheet:DC and AC Switching Character-istics, DS892 (v1.12) April 1, 2016