DABC
ROC-based DAQ: latest developments and perspectives
Jörn Adamczewski-Musch, Hans G. Essel, Sergey LinevGSI, Experiment Electronics: Data Processing group
• Optics• ROClib version 2• Preparation for beamtime in June 2010• BNet demonstrator• Throttling
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 2
DABCROClib - status end 2009
• Only nXYTER frontend was existing and supported• Connection between ROC and PC via Ethernet• ROC GUI was disabled• rocutil was the only tool for configuration of the ROC• DABC DAQ application for readout of several ROCs• go4monitor for simple online/offline analysis of data, taken during beamtime
• Worked reliable during beamtime 2009
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 3
DABCOptic developments
• Three kind of traffic in one media:– data– control– DLM
• Required extra hardware:– AVNet board
• Required extra software (included in ROClib):– PCI driver (root account)– mprace library– abbdaemon application
• Lot of tests last several month:– stable work in fixed configuration, bis 250 MB/s datarate– problems with simultaneous usage of both ports– already supported in ROClib, some improvements required
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 4
DABCNew functionality in ROClib v2.0
• Goal for v 2.0 – optics as mainstream, Ethernet as fallback solution
• New functionality in ROC FPGA code:– support of new frontend (FEET)– programmable commands lists– new message format (little endian)– reorganization of control address space (breaks compatibility)
• On the PC side:– support of the new FPGA features– device/transport classes for optic– new roc::Message format – support both old and new formats– roc::Iterator class for access data from different data sources– rocGui for ROC/nXYTER/FEET configurations– go4monitor can be connected directly to the ROC (without DAQ app)
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 5
DABCNew roc::Message and roc::Iterator classes
• roc::Message is replacement for old nxyter::Data class• Supports both old (v1.x) and new (v2.x) message formats
• roc::Iterator class allows to get data:– from ROC via Ethernet (via roc::Board class)– from ROC via Optic (also via roc::Board class)– from LMD files (replacement for fileapi)
roc::Board* brd = roc::Board::Connect(“optic://abb0”);brd->startDaq();roc::Iterator iter(brd);int cnt = 0;while (iter.next() && (cnt++<1000)) { iter.msg().printData();}brd->stopDat();roc::Board::Close(brd);
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 6
DABCrocGui
• Qt4-based
• Only for configurations– DAQ with go4monitor
• Can configure:– ROC itself– nXYTER frontend– FEET frontend
• Works with optic and Ethernet
• Able to produce scriptsfor rocutil
• Modular design, easy to extend
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 7
DABCMigration from v1.x to 2.0
• Version 2.0 is not backward compatible with 1.x
• Old LMD files are supported in v2.0
• Final version 2.0 will be announced together with exact instruction for upgrade
• Exact sequence to upgrade firmware/software on the ROC:– checkout new ROClib in separate location;– with old rocupload program update first firmware (not forget jumpers):
• rocupload cbmtest01 –fw ~/roclib2/firmware/Release/FEET02000004_ETHERNET02000002.bit
– with old rocupload program update PowerPC software:• rocupload cbmtest01 –sw ~/roclib2/sw-host/release/image.bin
– reboot ROC– compile new ROClib and try to use it with the ROC
• rocGui cbmtest01
– For more info see README.txt and cbm-wiki.gsi.de
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 8
DABCBeamtime - plans end 2008
Datacombiner
board
SFPSFP
SFP
SFP
SFP
SFP
FEB4nx SiCBM01B2
ROC
EthA
BSYNC-S
AUX
FEB4nx SiCBM01B2
ROC
EthA
BSYNC-S
AUX
FEB4nx SiCBM01B2
ROC
SFPA
B AUX
FEB1nxGen
BEAMAUX...
ROC
SFPA
B AUX
ROC
SFPA
BSYNC-S
AUX
FEB4nxSiDem1
ROC
SFPA
BSYNC-S
AUX
FEB4nxSiDem1
ROC
SFPA
BSYNC-S
AUX
FEB4nxSiDem1
ROC
SFPA
B AUX
FEB4nxSiDem1CBM
STSDemo 1
ROC
SFPA
BSYNC-S
AUX
FEB4nxSiDem1
ROC
SFPA
BSYNC-S
AUX
FEB4nxSiDem1
ROC
SFPA
BSYNC-S
AUX
FEB4nxSiDem1
ROC
SFPA
B AUX
FEB4nxSiDem1CBM
STSDemo 1
Datacombiner
board
SFPSFP
SFP
SFP
SFP
SFP
Datacombiner
board
SFPSFP
SFP
SFP
SFP
SFP
Eth
MBS
DABC node1
SFP/PCIe
ReadoutEvent building
FilteringFile store
IB/PCIe
DABC node2
SFP/PCIe
ReadoutEvent building
FilteringFile store
IB/PCIe
DABC node3
SFP/PCIe
ReadoutEvent building
FilteringFile store
IB/PCIe
DABC node4
Eth
ReadoutEvent building
FilteringFile store
IB/PCIe
Infin
iBan
d
swit
ch
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 9
DABCBeamtime in June 2010 – realistic plan
FEB4nx SiCBM01B2
ROC
EthA
BSYNC-S
AUX
FEB4nx SiCBM01B2
ROC
EthA
BSYNC-S
AUX
FEB4nx SiCBM01B2
ROC
SFPA
BSyncS
AUX
FEB1nxGen
BEAMAUX...
ROC
SFPA
BSyncM
AUX
ROC
SFPA
BSYNC-S
AUX
ROC
SFPA
BSyncS
AUX
ROC
SFPA
BSYNC-S
AUX
ROC
SFPA
BSyncS
AUX
DABC node1
SFP/PCIe
ReadoutEvent building
FilteringFile store
IB/PCIe
DABC node2
SFP/PCIe
ReadoutEvent building
FilteringFile store
IB/PCIe
DABC node3
SFP/PCIe
ReadoutEvent building
FilteringFile store
IB/PCIe
Infin
iBan
d
swit
ch
DABC node4
SFP/PCIe
ReadoutEvent building
FilteringFile store
IB/PCIe
Any other frontend
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 10
DABCBeamtime in June 2010
• Necessary components:– Up to 8 ROCs, 4 ABBs– SyncM/SyncS cabling– 4x Opteron-IB DAQ cluster with 4 TB storage– DABC with ROC-ABB and ROC-BNet plugin– Several PCs for monitoring
• Still to do:– reliable work of ABB with both SFPs– modifications in ROC-ABB plugin– Intensive ROC-BNet tests – can be done in next two months
• Fallback solution (if optic fails)– Same electronics, but readout via Ethernet– Reduced datarates– With or without BNet
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 11
DABCCBM DAQ perspectives - dataflow
FEE
Readout
Buffering
BNet
FLES
SLES, storage
Switch ~1 TB/s of raw data over networkProvides combined time slices to the FLES
Throttle data in case of buffers/networks overflow
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 12
DABCBNet simulations with SystemC (2005)
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 13
DABCInfiniBand performance tests (2006-2007)
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 14
DABCDABC development (since 2007)
• General purpose DAQ framework
• Data-flow core– threads, device & transports– user modules, application
• Configuration, control, monitoring, GUI
• Provides components for custom event building network - BNet
• Real data sources – MBS, ROC/UDP, ROC/Optic, generic PCIe board driver
• Used as DAQ for CBM 2008/2009 beamtimes
• Mid- and short-term plans:– Improved command interface (deadlocks recognition)– Support of other kinds of control system (EPICS)– Connection to cluster/job management system
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 15
DABCBNet demonstrator
• Since 2007:– better understanding of traffic/datarates– more realistic trigger algorithms were proposed– DABC was established
• One can launch activity for new BNet demonstrator:– with more realistic traffic generators– with ~100-200 nodes InfiniBand cluster (~20% of final CBM)– use DABC-based infrastructure– emulation of FLES activity (CPU consumption)
• Aim of demonstrator:– where InfiniBand technology now, that is near perspectives?– that kind of traffic shaping we need, if any?– is time-scheduled transport gives us significant benefits?– reliability and errors recovery issues?
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 16
DABCCBM DAQ perspectives - throttling
FEE
Readout
Buffering
BNet
FLES
SLES, storage
No backpressure until BNet.
How to throttle data when data rates higher than output link performance?
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 17
DABCThrottling algorithm
• While all messages are time-stamped, one can provide very simple throttle algorithm, based on absolute epoch number. For instance:
– drop every 8-th epoch, if not enough– drop every 4-th epoch, if not enough– drop every 2-nd epoch, if not enough– drop everything, but first epoch
• Any other dropping pattern can be implemented
• Algorithm required memory for at least last 8 epoch– no problems in readout buffer or BNet
• What to do in FEE, where buffer capacity is limited?
E4E3E2E1 E8E7E6E5
13.04.2010 S.Linev ROC-based DAQ: status and perspectives 18
DABCThrottling in FEE
• Doing same as in readout buffers and Bnet – requires large buffer space and complex arbitration logic, implemented in hardware => too complicated
• As proposed by W. Müller on last DAQ meeting, set busy bit (flip-flop) if FIFO full and reject any new data until next epoch. Efficiency of such algorithm is highly depends from relation between FIFO buffer space, epoch length (E) and output link performance (RATE):
– can reasonably work only when FIFO << E * RATE, but potentially high data lost – if FIFO ≈ E * RATE, it is about 50% probability if data of even or odd epoch will be
excluded by the FEE chip => completely unusable– if FIFO >> E * RATE, id of dropped epochs will vary from chip to chip, no any real
correlation, also unusable
• The only way to drop data in the FEE – introduce message counter per epoch. If amount of data in current epoch exceeds limit, all consequent messages in current epoch will be dropped. Guarantees, that at least in the epochs begin all data are preserved. Open question – per-channel or per-chip logic?