Online monitoringfor ProtoDUNE-SPJingbo Wang, Antonino Sergi, Marco Adinolfi

ProtoDUNE-SPDAQCommissioningMeeting

January 8, 2018, CERN

Online monitoring

§ Duringtherunitwillbeessentialtomonitortheperformances ofthedetectorsandmake sure everythingisfunctioningasexpected.

§ Theonlinemonitoringrunswith the DAQ duringdatatakingandupdatesatregularintervals to indicateanypossibleproblems.

§ Themonitoringmodulesarewritteninartandwillunpack therawdataandproducethemonitoring plots during the run.

§ Atthemoment,thebasiconlinemonitoringframeworkhasbeenbuiltandisbeing tested.

Slide2

Candidate Monitoring Metrics

§ Generalmonitoring– Numberofsubdetectorswith data– Eventsizevs“time”– Triggerratevs“time”

§ Event monitoring– Raweventdisplayinthreeviews for each APA– Mean/RMSoftheADCsvs channel number– FFTofwaveforms vs channel number– Stuckbitsfraction– Trigger type

§ Plotswithbasicreconstruction (dependingonthecomputationalcapability)– NumberofhitsperviewintheTPC– Numberofhits/averageamplitudeperviewvstime– Distributionof clusterlengthandnumberofhitspercluster

§ Hardwarediagnosticinformation(forelectronicsexperts)– Seekingsuggestions

Slide3

Architecture of OM software

Slide4

§ DataLogger sendseventstooneormoredispatcherprocesses.

§ Dispatcherrouteseventstotheonlinemonitoringprocessesthroughsharedmemory.

§ Rawdecodersunpacktherawdatafragments(Timing,RCE,FELIX,SSP)usingtheoverlayclasses,andperformlow-levelanalysis.

§ Artanalyzersperformhigh-levelanalysisoftheunpackedevents(offlinecompatible)

§ HistogramsaresavedlocallyintheROOTfiles.

§ Webdisplaytool(Monet/LHCb)copiesthehistogramstothewebserver

Installation of the online monitor

§ Installation of the online monitor (JohnFreeman):– https://cdcvs.fnal.gov/redmine/projects/dunetpc/wiki/Guide_for_protoDUNE_online_monitoring_d

evelopers

§ Installation of the web-display tool Monet (Marco Adinolfi):– https://twiki.cern.ch/twiki/bin/view/CENF/PdspMonet

§ Theonlinemonitor is installedin the CERN computer np04-srv-014:/nfs/sw/om– Using a dunetpc feature branch “feature/online_monitoring_artdaq” (depends on both

lbne-raw-dataanddune-raw-data) with the raw decoders– Using the develop branch of dune_raw_data that provides the overlay classes and the

channel mapping service.

§ Note: the RCE overlay class depends on an external library that is installed in theProto-DUNE computer at CERN. At the moment, the online monitoring software isonly functional on the CERN machines. Fermilab (Tom Junk) is working to mergethis library to dunetpc for offline use.

Slide5

Running the online monitor

§ Basic operation: https://twiki.cern.ch/twiki/bin/view/CENF/PdspOMOp (to beupdated)

§ Run from the DAQ interface– The configuration files (fcl files) are stored in np04-srv-014:/nfs/sw/om/fcl– RawDecoder.fcl defines the parameters for all the decoding and analyzing modules– RunOnlineMonitor_ShMem.fcl is called by the DAQ interface to analyze the live data

during a run– StartOM_lite.sh starts a single OM process and saves the process ID to a txt file.– StopOM_lite.sh reads the process ID and kills the OM process– The output root file and log message file are saved to a local path: /scatch/OMoutput to

be accessed by Monet.

§ Run from the terminal– source/nfs/sw/om/ProtoDUNE_OM/setupONLINEMONITORING– Online with live data: art –c RunOnlineMonitor_ShMem.fcl– Offline with an existing file: art –c RunRawDecoder.fcl RawRootFile.root

§ Multi-process script was prepared and will be tested soon. Slide6

Example fcl file

Slide7

np04-srv-014:/nfs/sw/om/fcl/RunOnlineMonitorShMem.fcl

Unpack the rawdata and write tooffline format

Analyze the event withoffline format

Data transferred throughshared memory

Channel mappingservice

Online monitoring specificservice allows for file switchingduring the run (per 100 events)

Producers and analyzers are ranin sequence and no intermediatefile needs to be saved to the disk

RCE/FELIX Channel Map

Slide8

Onlinechannel

RCE,FELIXID

RCE,FELIXchannel Regulator

Regulatorpin

Asic Asicchannel APA

APAplane

Offlinechannel

§ Modified the 35-ton channelmap service and the map file

§ RCE/FELIX fragment ID andinternal channel ID are knownparameters obtained from theDAQ

§ Mapping the RCE/FELIXparameters to oneline/offlinechannel numbers

§ Regulator and asicinformation are not used atthis moment, but will beuseful for coherent noiseremoval

§ APA and plane columns maybe not needed because theycan be obtained from theoffline channel number withthe geometry service

RCE and FELIX Monitoring

§ Raw decoder (art producer module)– Number of ticks– All ADC values– Online and offline channel IDs– Expect more electronics diagnostic input: asic ID,

asic channel?§ Monitoring analyzers (art analyzer modules)

– Raw event display in three views– ADC Mean/RMS VS channel ID for each view in each

APA (2D histogram)– Profiled histogram of the ADC Mean/RMS– FFT spectrum VS channel ID (2D histogram)

Slide9

ü FELIX is assigned to the last APA.

ü In the analyzer module, rawdigits from RCE and FELIX aremerged to a single “TPC” vector

ü Analyzer modules work foroffline analysis with simulatedfile as well

ü Channelmapstillneedstobesettled

For more available offline modules, seehttps://twiki.cern.ch/twiki/bin/view/CENF/DUNEProtSPOM

Test with simulation file

Slide10

SSP Monitoring

Slide11

§ Raw decoder (art producer module)– AllADCvalues– Numberofpacketsperevent– Peak(frommetadata)spectrumperchannel– Area(frommetadata)spectrumperchannel– Area(frommetadata)spectrumperchannel– One waveformperchannelperfile– FFTperchannel– Triggertypehistogram(internal,external)– Persistentwaveformsovermanyevents

§ Monitoring analyzers (art analyzer modules)– ADC Mean/RMS VS channel ID(2Dhistogram)– Profiled histogram of the ADC Mean/RMS– More suggestions?

ü Channel map needs to be settledü High-levelanalysis(FFT)canbe

movedtotheanalyzer

Plots added by Paolo Franchini

Timing Board Monitoring

Slide12

§ Raw decoder (art producer modules)– Timestampofthetimingfragment– Timedifferencebetweentwoconsecutive

fragments– Triggertype

§ Monitoring analyzers (art analyzer modules)– Suggestion?

PlotsaddedbyPhilipRodrigues

Web Display Tool: Monet

Slide13

§ Weballowsforeasymonitoringfromeverywherewithoutconnectingtoremotemachines

§ Proto-DUNEisusinga modifiedversion of Monetthat wasoriginallywrittenbyLHCb.

§ Monetis written in python and hasseveralusercases. The PromptDQM case ishacked for Proto-DUNE, asthatistheonethatreadsROOTfiles (done by MarcoAdinolfi).

ü Folder/page/histogramstructure allows to organizeplots for differentcategoriesofusers:shifter,expert,specialstudies...

ü /nfs/sw/om/Monet/Dune_page_file defines the pages

ü /nfs/sw/om/Monet/Dune_histo_file defines which histogramto present for each page.

CanbeaccessedwithinCERNat:http://np04-srv-014:8123/prompt_dq/

Preliminary plots in Monet: TPC

Slide14

ADCMean/RMSvsChannel#

Preliminary plots in Monet: TPC

Slide15

ADCMean/RMSvsChannel#(profiled)

Preliminary plots in Monet: TPC

Slide16

FFTvsChannel#

Preliminary plots in Monet: SSP

Slide17

Persistentwaveform

Preliminary plots in Monet: SSP

Slide18

Eventnumber,ADCs,Npackets,frequency,Triggertype,FFT

Preliminary plots in Monet: Timing

Slide19

Multi-process running

Slide20

§ Themodulemustbeveryefficientsincethemonitoringneedstoallowsupervisionoftherun

§ Multi-process capability allows faster running and scalability to larger detectors§ Multi-process running was tested with the old set of monitoring modules. We’ll

update the scripts soon.§ Art doesn’t support multi-process within a module, so we chose to implement the

multi-process feature in the terminal– ./Makefclconfig.sh creates a number of fcl files to be processed– ./StartOM.sh launches all the monitoring processes and saves the PIDs.– ./StopOM.sh reads the PIDs and kill all the processes.– Lastly, the root files of different process have to be merged into a single file.

The Prescaler module allows to assign events to differentprocesses.

Things to do

§ Will clean up the existing code and add all the produced histograms toMonet.

§ Willkeepaddingmorehistograms astheyarerequestedorrequired.§ The channel maps for the RCEs/FELEX and SSPs have to be settled.§ Monet is slow, especially when presenting 2-D histograms. Need to select

appropriate bin size to increase the uploading speed. Bugs in Monet?(PhilipRodriguesfixedabugandmadeMonetfaster)

§ Need to update the multi-process scripts and add the feature to the DAQinterface

§ The shared memory plugin doesn’t allow to transfer data across nodes.Need to switch to multi-cast plugin (John Freeman)

§ The online monitoring modules can be directly used by offline analysiswith very little modification (Fermilab).

Slide21

Conclusion

§ We haveamonitoringframeworksetupforreadingtheraw dataandproducingdataqualityplots during the run.

§ We are developingand improving themodules. We’ll add more plots asthey are required.

§ The web display tool is under development and will be the next focus.

§ The online monitor is currently being tested alongside the DAQ. Potentialissues could be found. PleasereportbugsbyE-mailtotheonlinegroup.

§ Any suggestions are very welcome!

Slide22