DUNE DAQ Meeting

Felix+DPM

David Cussans30th July 2018

Introduction• Investigate ”FPGA-centric” approach for DUNE DAQ

• (TP/IDR baseline. Alternative is CPU/GPU centric )

• Provide information for TDR

• Need result Q3/Q4 2018

• Show plausibility of PCIe based ”FPGA-centric” approach.

• Filtering and hit-finding for SP-TPC done in processing FPGA

• Most firmware blocks agnostic to location of this FPGA ( in separate crate / incorporated with Felix )

• Technical Proposal (IDR) baseline solution has 10-second buffer in CPU memory. Test idea of buffer connected to FPGA.

• Solution must be suitable for all sub-systems: SP-TPC, DP-TPC, DP-PDS, SP-PDS

• Optional daughter-board? Different Felix versions?

David Cussans | DAQ meeting, 30th July 182

Generic Hardware Layout• 10 second buffer in DRAM• Collect enough data to decide if SN• Look back before trigger fires

• Solid-state storage for > 100s of data if SN trigger• Tech. proposal assumes 4 x DPM per APA. Aim for 1 x DPM per APA• Initial firmware results show compression , filtering can be fitted into less FPGA resource

ToBack-end

DataSelection

Dia

gram

–Ba

bak

Abi

TriggerCommand

David Cussans | DAQ meeting, 30th July 18

Sync Cmd

Time/Sync Cmd

(1/4 APA)

DPM-Felix Links• Data reception, timing reception, clocking on

Felix.• FIFO-like links DPMßàFelix?

David Cussans | DAQ Firmware meeting, RAL 31st May 18

DPM-Felix Links – WIB Data• Max density probably ~ 1 Felix/DPM per APA

• Limited by SSDs

• à 10 x 9.6 Gbit/s links into Felix, 10 x 9.6Gbit/s to DPM

David Cussans | DAQ Firmware meeting, RAL 31st May 18

DPM-Felix – Trigger Primitives• Assume dominated by 39Ar (107 Bq/10kT) , 16-

byte primitives• 67k primitives/s , 8.5 Mbit/s

David Cussans | DAQ Firmware meeting, RAL 31st May 18

DPM-Felix – Triggered Data• 10-sec buffer in FPGA

• Read out variable length, depending on commands from data selection.

• Messages to buffer controller ~ 67 kHz ( assume we select all 39Ar )• Each block of data ~ 2.4 kBits ( 100us, 12 bit , decompressed before transmission )• à 160 Mbit/s

David Cussans | DAQ Firmware meeting, RAL 31st May 18

DPM-Felix – SN Data• Send capture/readback commands over same

links as triggered data• Add one additional link

David Cussans | DAQ Firmware meeting, RAL 31st May 18

DPM-Felix – Setup/Control• DPM will need setup, control and monitoring

• Best as register access (?)• Protocol?• One additional bidirectional link

David Cussans | DAQ Firmware meeting, RAL 31st May 18

DPM-Felix – IPMI• DPM from xTCA ecosystem.

• Needs IPMI control• Physical layer = I2C

• Firmware/software• Not onerous (probably) for group with previous experience• May be possible to disable IPMI. ( Need to confirm )

David Cussans | DAQ Firmware meeting, RAL 31st May 18

DPM-Felix – Timing• Do we need a link to the timing/synchronization

system?• 250MBit/s signal in Single-Phase timing system

compatible with timing system hardware in Felix

David Cussans | DAQ Firmware meeting, RAL 31st May 18

DPM-Felix – Summary of Links• WIB Data à DPM: 10 links• Triggered data à Felix: < 10 links • Trigger primitives: 1 link• Supernova data: 1 link• Setup/control/monitoring: 1 link• Total: 13 bidirectional links, <= 10Gbit/s

• Links will need protocol

• IPMI: I2C ( 2 wires )

David Cussans | DAQ Firmware meeting, RAL 31st May 18

Firmware Blocks

David Cussans | DAQ meeting, 30th July 1813

• Pre hit-finding filtering:• RAL: Kostas Manoulopolous . See

https://indico.bnl.gov/event/4854/contributions/22958/attachments/19346/25193/NoiseFiltering.pdf

• Hit-finding:• Bristol: Kumar Kunal, Jim Brooke. See

https://indico.bnl.gov/event/4854/contributions/22892/attachments/19347/25194/JimBrooke-FPGAHitFinder-BNLWorkshop.pdf

• Buffer management:• UCL: Erdem Motuk, Ryan Nichols. See

https://indico.bnl.gov/event/4854/contributions/22957/attachments/19387/25252/buffer_management_dune20072018.pdf

• Framework for evaluation of firmware blocks in hardware:

• Bristol, RAL: Dave Newbold, Kostas Monoulopolous. See https://indico.bnl.gov/event/4854/contributions/22956/attachments/19340/25184/DAQ_FPGA_180720.pdf

Firmware Blocks

David Cussans | DAQ meeting, 30th July 1814

• Collaboration welcome in all areas • Especially welcome in:

• Compression –• Use lighter-weight algorithm than APE compression

implemented for protoDUNE• E.g. fixed encoding of the difference of samples• Loss in compression w.r.t. APE, but much lower resource usage

and quicker to decompress.• Storage to SSD

• Current assumption: Handled by Zync ARM core. Needs to be implemented and tested.

• Could buy IP to write from programmable logic ( get moregates per $/£/CHF/Euro for Kintex etc. than Zync ). Would need to be integrated, tested.

DUNE Specific Firmware for Felix• Transmit data to DPM

• Decoding WIB data done in Felix.• What format to send to DPM? Just a copy?

• Interfaces that could all look the same ( AXI4-Stream ?)

• Transmit trigger primitives• Receive trigger commands (including SN capture)• Transmit data in trigger buffer requested by trigger

command• Transmit (at low bandwidth ) SN data on request

David Cussans | DAQ Firmware meeting, RAL 31st May 18

Felix + DPM hardware plans• BNL will develop a variant of the Felix that can host a

DPM• Will also contain some other features ( e.g. new FPGA family,

removing need for external PCIe bridge)• Ready ~ October/November.

• DPM scheduled to be ready before modified Felix• Design a carrier board that can host a DPM + miniPod optics (

UCL ).• Based on SLAC carrier board(?)• Use with existing, unmodified, Felix.

• Need urgent work on establishing a way of exchanging design information about DPM, Felix.• FNAL Redmine not open to all collaborators. Pay for private

Github, Bitbucket repo?• Before DPM arrives – test with Eval board coupled to

existing Felix.• E.g. ZCU102 with two dual QSFP FMCs. ( 16 bidirectional links

total )

David Cussans | DAQ Firmware meeting, RAL 31st May 18

Hardware for TDR – Eval Boards• Eval boards• UnmodifiedFelix

Hardware for TDR – DPM + Carrier• DPM on simple carrier• Unmodified Carrier

Summary• Plan in place to construct hardware test-platform of Felix+DPM

on time-scale of TDR:

• Eval board ßà unmodified Felix at first

• Then DPM + carrier ß à unmodified Felix

• Then DPM + modified Felix

• This option ready in time for TDR, most data will be gathered with previous hardware.

• Plan in place to develop algorithm firmware

• Compression, storage to SSD still open

• Need to develop plan for DUNE specific Felix firmware.

David Cussans | DAQ Firmware meeting, RAL 31st May 1819