Juno proposal for PMT readout – GCU - INFN Sezione di Padova · INFN and University of Padova -...
10
INFN and University of Padova - GCU Proposal 1 Juno proposal for PMT readout – GCU M. Bellato 1 , R. Brugnera 2 , F. Dal Corso 1 , S. Dusini 1 , A. Garfagnini 2 , R. Isocrate 1 , I. Lippi 1 , G. Meng 1 , D. Pedretti 3 1 INFN – Sezione di Padova 2 Universita’ di Padova – Dipartimento di Fisica G. Galilei 3 INFN – Lab. Nazionali di Legnaro 1. Introduction The scope of this document is to describe a possible architecture for photo-multiplier (PMT) readout, following the indication of a baseline structure agreed at the Padua meeting of October 2015. Essentially, the baseline structure states that each PMT should embed both the High Voltage and the Readout electronics in a standalone manner, with sufficient I/O for PMT signal processing, high voltage interface, triggering support and data readout. 2. Architecture The Juno collaboration recommends a basic structure in which PMT readout, trigger primitive generation, fragment buffering, selective data readout and HV interface takes place on a per PMT basis. One possible solution is to embed these complex tasks on the PMT itself, by augmenting the PMT physical volume with a water-tight box housing HV and digital electronics and communicating with the external world by means of copper cables with an estimated length of ~ 100 meters. Given the high (~20.000) number of PMTs involved, the overall cost of the digital board and the number of cables are to be assumed as parameters to be minimized. The conceptual scheme for trigger and readout electronics can be summarized as in figure 1. Each PMT signal gets continuously digitized by a custom made ASIC and buffered in a local memory while looking on the fly for meaningful data with threshold comparison: in such a case a trigger request is generated and routed to a Global Trigger Processor via a dedicated link.
Transcript of Juno proposal for PMT readout – GCU - INFN Sezione di Padova · INFN and University of Padova -...
INFNandUniversityofPadova-GCUProposal
1
JunoproposalforPMTreadout–GCU
M.Bellato1,R.Brugnera2,F.DalCorso1,S.Dusini1,A.Garfagnini2,R.Isocrate1,I.Lippi1,G.Meng1,D.Pedretti3
1INFN–SezionediPadova2Universita’diPadova–DipartimentodiFisicaG.Galilei3INFN–Lab.NazionalidiLegnaro
1.Introduction
Thescopeofthisdocumentistodescribeapossiblearchitectureforphoto-multiplier(PMT)readout, following the indication of a baseline structure agreed at the Paduameeting ofOctober2015.Essentially,thebaselinestructurestatesthateachPMTshouldembedboththeHighVoltageand the Readout electronics in a standalone manner, with sufficient I/O for PMT signalprocessing,highvoltageinterface,triggeringsupportanddatareadout.2.Architecture
TheJunocollaborationrecommendsabasicstructureinwhichPMTreadout,triggerprimitivegeneration,fragmentbuffering,selectivedatareadoutandHVinterfacetakesplaceonaperPMT basis. One possible solution is to embed these complex tasks on the PMT itself, byaugmenting the PMT physical volume with a water-tight box housing HV and digitalelectronicsandcommunicatingwiththeexternalworldbymeansofcoppercableswithanestimatedlengthof~100meters.Giventhehigh(~20.000)numberofPMTsinvolved,theoverallcostofthedigitalboardandthenumberofcablesaretobeassumedasparameterstobeminimized.Theconceptualschemefortriggerandreadoutelectronicscanbesummarizedasinfigure1.EachPMTsignalgetscontinuouslydigitizedbyacustommadeASICandbufferedinalocalmemorywhilelookingontheflyformeaningfuldatawiththresholdcomparison:insuchacaseatriggerrequestisgeneratedandroutedtoaGlobalTriggerProcessorviaadedicatedlink.
INFNandUniversityofPadova-GCUProposal
2
Figure1DataarestoredwaitingforatriggeracceptthatmaycomeviaanEthernetinterface:insuchacase,aprocessingelementwillinspectthedatastoragelookingforaneventfragment(orabunchofevent fragments)matching the requestedone(s) inapreset timewindow;uponpositivematch,thecurrenttimestampandacentrallyassignedeventtagareassociatedwiththematchedfragmentandsenttotheDataAcquisitionSystemthroughtheEthernetlink.This conceptual scheme is widely deployed in most modern physics experiments and iseffectiveinreducingthebandwidthrequirementsofthedatalink.Theworstcasescenariointermsofbandwidthrequirementcomesfromtriggersduetodarkcurrent,whoseratemayreach ~50KHz; assuming 30ns of sampling period at 1 Gsample/s with 16 bit words, themaximumdatarateisintheorderof24Mb/s,wellintherangeofFastEthernet.HencethecostofthemediumcouldbegrantedbyaninexpensiveCAT5eEthernetcable.AcandidateschemeforlocalreadoutisillustratedinFig.2.
INFNandUniversityofPadova-GCUProposal
3
Figure2.Accordingtofig.2thecorecomponentsoftheGCUare:
• ADU–acustombuiltASICwith1Gs/sADCs,analogpreamplifieranddigitalI/FforPMTsignalprocessing;thedigitalI/FisLVDSdoubledatarate(DDR)at500MHz
• Storage:2GBytehighbandwidth,lowcostRAMchipsstoringupto1sofeventfragments
• EthernetPHY:a100Mb/sEthernetphysicallayerchipfordatareadoutandcontrol• PoE+transformer:a48V,30WIEEE802.3atcompliantchip,receivingpowersupply
throughEthernetcableinaccordancewithAlternativeA• ADC/DCconverterfeedingGCUwithappropriatevoltagerails• Linedriverandreceiver:suitablebuffersforclockreceptionandtriggertransmission
via~100mcoppercablepairs.• FPGA:gluelogic,highperformancestatemachines,communication,ASICreadout,
HVslowcontrol• Flashmemory:longtermstorageforFPGAconfigurationbit-stream;redundantfor
disasterrecoveryTo reduce the number of copper pairs reaching the GCU, the IEEE 802.3at Alternative Astandard for remotepower supplyof (usually) telecomhardwaremayhelp. The standardforeseestheinjectionofa~48Vbiasonthecentraltapsoftheisolationtransformersforthetransmissionandreceptionpairsof10/100Mb/sEthernetasshowninfig.3
INFNandUniversityofPadova-GCUProposal
4
Figure3.The remotepowereddeviceextracts the commonmodevoltageand feedsa localDC/DCconverterforappropriateuse,sinkingamaximumpowerof30.5W.Low cost commercial hardware is readily available to act as Ethernet Switch and PowerSourcingEquipmentforasmanyas48ports[3].ThesamepowersourcecanbeusedtosupplypowertotheHighVoltagemodule,althoughsomeefforthastobeenvisagedtoshieldtheGCUfrompossibleHVdischarges.TheEthernetmediumrequiresprotocolstobeofanypracticaluse,andJunoisnoexception.InthecontextoftheCMScollaboration[2],anefforthasbeenmadetodevelopanddeployanUDPbasedIP(IntellectualProperty)corethatabstractsanhardwarebusontopoftheInternetProtocolandiscalled“IPbus”[1].IPbusbringstheUDPprotocolattheFPGAleveland leveragestheubiquityof IP fordirectcommunicationwithdistributedFPGAsthroughEthernet. The IPbus ecosystem is comprised also of software APIs for a further layer ofabstraction,thatmakesdataacquisitionandslowcontrolforahighnumberofpeerdeviceseasiertomanage.IPbusisdeployedbyseveralprojectsandexperiments:CMS,ATLAS,LHCb,Compass,CBM,amongothersandneutrinoexperimentslikeSOLIDandDUNE.3.SynchronizationThe problem of GCU synchronization in Juno is related to the distributed nature of datareadout.EachGCUcollectsafragmentofaneventwhichhastobejoinedwithpotentiallymanyotherfragmentsofthesameeventbeforeprocessing.Abarrelsorterapproachthroughcommercialtelecom switches is a proven and efficient method to accomplish parallel eventreconstruction in an event building farm, as schematically shown in fig. 4. Most physicsexperimentsdataacquisitionsystemsadoptthistechniquebecauseofscalability,excellent
INFNandUniversityofPadova-GCUProposal
5
performanceatmoderatecosts,duetoreuseofcommoditywidespreadhardware.InJunoitmakesevenmoresenseifoneadoptsEthernetreadoutattheveryfront-end.
Figure4.ACentralTriggerProcessor(CTP)collectstriggerprimitivesgeneratedatCGUlevelwithfixedlatency and applies suitable algorithms (e.g. coincidence, multiplicity, windowing, etc.)according to physics constraints. Upon reception of a suitable number of primitives thatqualify a potentially interesting event, the CTP generates a tag related to a Global TimeCounter(GTC)thatmustbesharedamongCTP,LCUsandGCUs.ThetimetagwillthenbeforwardedtoallGCUsviaslowcontrolordedicatedlinkforreducedlatency.Uponreceptionofatime-taggedeventvalidation,GCUswilllookforamatchintheirstorageofbufferedeventfragments:thematchmayoccurinapresettimewindow,e.g.incloseproximityofthetimespecifiedbythetag. Incaseofamatch,GCUswillretrievethefragmentsfromtheirmemoriesandforwardthemtothedataacquisitionsystem(DAQ)viatheIPbusdedicatedlink.
INFNandUniversityofPadova-GCUProposal
6
Thisreadoutmechanismfeaturesvirtuallynodeadtimebutreliesdeeplyontheavailabilityof a global time at the level ofGCUs. In several experiments [4], [5], [6], the global timeavailabilityisbasedonthedistributionofatimecounterwithvaryingresolutionsbymeansofhighspeed(usuallyoptical)seriallinksthatembedalsootherusefulservices(clock,resets,etc.).InJuno,thereferenceimplementationisbasedaroundareducednumberoflonglength(~100m)copperlinks,thusavoidingthedeployofmultigigabitseriallinks.In thiscase,adifferentdistributionschemeforglobal timemustbeenvisaged.GCU’swillreceiveaglobalclockviaadedicateddifferentialpairandtheywillcountlocallythisclockthatwillactasalocalcopyoftheGlobalTimeCounter(GTC).ButeveryGCUwillexperienceanoffsetintheircopyofGTC.Thisoffsethastwocontributions:
• thestartof thecounting isnotsynchronizedamongGCUs.DuetodifferentpowercyclesanddifferenttimeofarrivalofresetcommandsinGCUs,theirGTCswillstartatdifferenttimesandwillkeepthesamepace.
• ThecableslinkingLCUswithGCUsarepresumablyofdifferentlengths:thevelocityfactorincat5cablesvaryfrom0.4cto0.7c,e.g.asignalpropagationspeedof~5ns/m.With an external clock of 62.5MHz, ~3m of length mismatch in clock cables aresufficientforaclockpulseslidingamongGCUs.
INFNandUniversityofPadova-GCUProposal
7
Figure5ThefigureshowsthemechanismofglobaltimesynchronizationbetweenoneLCUanditssubordinatedGCUsanditcanbesummarizedinthefollowinglist.Thesynchronizationprocedureisprecededbyameasureofroundtriptime(RTT)fromLCUsandtheirGCUsinordertorecovercablelengthsmismatches.
1. DCSwillputallGCUsinsynchronizationmode,byprogrammingoneofitsregistersviaIPBUS.
2. Fromthatmomenton,GCUswillignoredataandtriggerscomingfromADUs3. GCUswillmonitortheirlocalcopyofGTC,waitingforclockcycleinwhichtheirGTC
willmatchapresetvalue4. AtthatexacttimeallGCUswillissueatechnicaltriggeronthededicatedpairtotheir
LCUs5. LCUsrecordthetriggertimeoffsetsamongalltheirsubordinatedGCUs:those
offsetsreflecttheoffsetsofGCUslocalGTCs.6. AssumingthatLCUsshareacorrectGTC,theycaninformtheDCSofthemeasured
counteroffsetsinGCUs7. DCSwilladjustGTCineachGCUbyaddingthecorrectoffsetviaIPBUS8. DCSwillinstructGCUstoissueanothertechnicaltriggerwhentheirGTCsmatcha
newpresetvalueinordertoverifythecorrectnessofthesynchronizationprocedure9. DCSwillinstructGCUstoexitsynchronizationmode.
Thesynchronizationprocedure(excepttheRTTsmeasurements)canberepeatedperiodically(e.g.onceperhour)inordertoverifytheconsistenceofglobaltime.ItisworthnotingthatthedeadtimegeneratedbytheprocedurecanbekeptverylowifthepresetvalueatwhichGCUsissueatriggerisclosetotheactualGTC.ThehighestcontributiontodeadtimewilllikelycomefromDCSoperationsviaIPBUS,butweexpectthatthefullproceduremaylastintheorderofhundredsofms.4.FPGAreprogrammingFPGA reprogramming is an important feature for bug fixing, feature enhancements, faultfixingandmaintenance.Dueto inaccessibilityofGCUsafter installation,thetask iscriticalbecause the firmwaremaygetcorruptedduring the reprogrammingphase itself.A safetymechanismmustbeputinplacefordisasterrecovery.Awatchdogmechanism,periodicallytoggled,wouldallowtochoosebetweentwofirmwarerevisions(aproductionversionanda“golden”version)storedinaflashmemory.Inpresenceof external toggling, the FPGA would be presented and programmed with a productionfirmware at power-up, while in absence of external toggle, the firmware selected forprogrammingwouldbethe“golden”version.ThetogglemechanismcouldbesoftwaretriggeredviaIPbusorhardwaresupported,basedonthepresenceofanexternalsignal,e.g.theexternallysuppliedclock,asshowninfig.6.Inthe latter case,whenanexternal clock isnot fed toGCU, theprogrammingmemorygets
INFNandUniversityofPadova-GCUProposal
8
selectedwiththe“golden”firmware,theFPGArunsonaninternalclockand,afterapowercycle,theFPGAwillbeconfiguredwithaknown-to-be-workingfirmware.
Figure6.In case of bug fixes, firmware enhancements, tests, or GCU board qualification, the“production”firmwarecanbeoverwrittentakingcontroloftheflashmemorydatabusontheFPGA. With suitable support of IPbus mapped registers and FIFOs, the programmingprocedurecanbeperformedwithsimplesoftwarescripts.5.I/OTheInput/outputsignalsoftheGCUcanthereforebesummarizedasinthetablebelow.ApartfromHighVoltagebiasandcontrolsignalswhoserunisshort(e.g.,fewcm),allothersignalsaretobeconsidereddifferentialpairsfornoiseimmunityoverlongruns.
INFNandUniversityofPadova-GCUProposal
9
Name Type DirectionIPbustx Ethernet OutputIPbusrx Ethernet InputClock Cablereceiver InputTrigger Cabledriver OutputHV_i2c_SDA i2copto-isolateddata BidirectionalHV_i2c_SCL I2copto-isolatedclock BidirectionalHV_power_bias 48Vbias SharedHV_power_ref 48Vreference Shared6.ControlsTheadoptionofIPbuswillallowtransparentmanipulationofFPGAsregistersacrossEthernet,thuseasingthetaskofdistributedcontrol.TheinterfacetothePMThighvoltageboardcouldexploitsimpleserialbusses like I2CorSPI,conceivedforchiporperipheralcontrol.Thesebussesuseareducednumberofsignalsforcommunicationandalthoughdataexchangeisusuallysynchronoustoabusclock,pulsetimingcanextendintherangeofmsandbeyond,thus allowing direct software manipulation of bus signals instead of using dedicatedhardware.Thistechniqueisknownas“bitbanging”andcaneasilybeimplementedviaIPbus.7.References[1]-R.Frazier,G.Iles,D.Newbold,A.Rose:“SoftwareandfirmwareforcontrollingCMStriggerandreadouthardwareviagigabitEthernet”,TIPP2011.[2]–TheCMSCollaborationatLHC:http://cms.web.cern.ch[3]-http://www.dell.com/us/business/p/power-over-ethernet-switches[4]–P.Moreiraetal.“TheGBTproject”,TopicalWorkshoponElectronicsforParticlePhysics,Paris,France,21-25Sep2009,pp.342-346[5]–M.Bellatoetal,“GlobalTriggerandReadoutSystemfortheAGATAexperiment”,Real-TimeConference,200715thIEEE-NPSS[6]–TheIceCubecollaboration:“TheIceCubeDataAcquisitionSystem:SignalCapture,Digitization,andTimestamping,ARXIV,2008.
INFNandUniversityofPadova-GCUProposal
10
8.RevisionHistory
Date Version Revision11/12/2015 1.0 Initialrelease
