Upgrades for the CMS simulation
-
Upload
capricorn-rosas -
Category
Documents
-
view
30 -
download
0
description
Transcript of Upgrades for the CMS simulation
D. Lange, ACAT 2014 1
Upgrades for the CMS simulation
David J LangeLLNL
September 4, 2014
September 4, 2014
D. Lange, ACAT 2014 2
CMS simulation faces significant challenges for both today and tomorrow
The CMS Physics goal is to keep same performances as in Run 1 despite the increasing more harsh conditions.
We are busy preparing for 2015 to • The higher LHC beam energy means more
complex hard-scatter events• The higher LHC luminosity means larger
number of interactions per bunch crossing (higher pileup) and thus more time consuming to simulate and reconstruct
• Higher output rate of trigger (~1kHz) means demand for larger samples of simulated events
We have achieved significant progress on the resource needs of our simulation during LHC shutdown
September 4, 2014
D. Lange, ACAT 2014 3
CMS simulation faces significant challenges for both today and tomorrow
Preparing for CMS at the start of Phase 2 (HL-LHC):• The CMS detector configuration is still to be determined• Even higher output rate of trigger (potentially 10kHz)• Even higher luminosity and pileup (140+ interactions/crossing)
10351034
HL-LHC presents increased challenges for Triggering, Tracking and Calorimetry, in particular for low to medium PT objects
September 4, 2014
D. Lange, ACAT 2014 4
CMS Upgrade Strategy - Overview
LS1 LS2 LS3
Upgrades 2013/14 ongoing:• Completes muon coverage (ME4)• Improve muon trigger (ME1), DT electronics• Replace HCAL photo-detectors in forward and outer (HPD → SiPM)
Complete original detectorAddress operational issuesStart upgrade for high PU
Phase 2 Upgrades: 2023-2025 (Technical Proposal in preparation)• Further Trigger/DAQ upgrade• Barrel ECAL Electronics upgrade• Tracker replacement/ Track Trigger• End-Cap Calorimeter replacement
Maintain/Improve performance at extreme PU. Sustain rates and radiation doses
Phase 1 Upgrades 2018/19 (TDRs):• New Pixels, HCAL SiPMs and electronics, L1-Trigger• Preparatory work during LS1:
• new beam pipe• test slices of new systems (Pixel cooling, HCAL,
L1-trigger)
Maintain/Improve performance at high PU
September 4, 2014
D. Lange, ACAT 2014 5
Changes we are making to facilitate 2015 and PhaseI/PhaseII upgrade development
September 4, 2014
Happy users
Simulation of very high pileup within resource
constraints
Many different “CMS”
geometries to support
Faster performance
without sacrificing
physics
Easier configuration management
D. Lange, ACAT 2014 6
Flexible geometry infrastructure supports development of proposed systems
• Each “CMS” geometry is made up of XML components for each subdetector– XML development done mostly by
subdetector experts– Consistency checks and definition of
overall “CMS” geometry scenarios done by overall geometry expert
– Geometry is controlled via runtime job configuration to make it easy for users to get the geometry they want.
September 4, 2014
The initial CMS geometryinfrastructure design has worked very well for upgrade evaluation
D. Lange, ACAT 2014 7
Example configuration of Phase II geometries implemented in CMSSW geometry
September 4, 2014
D. Lange, ACAT 2014 8
Improving technical performance of the CMS detector simulation
We have evaluated and integrated a number of technical improvements1. Monte Carlo sampling techniques (e.g., Russian Roulette) 2. Deployment of Geant4.10 3. Improved library packaging
September 4, 2014
Together these improvements have gained a factor of 2 in simulation time/event in the last year of development
D. Lange, ACAT 2014 9
Russian Roulette: Sampling of low-energy particles in Geant4
• Method from neutron shielding calculations: Track only a small fraction of low-energy particles through the detector with no noticeable change in simulation results– We found that it was necessarily to have sampling factors and
thresholds that depend on both detector region and particle type.• Two parameters:
– RR factor (1/W): Fraction of particles to keep
– Upper energy limit (ERR)
• Hits from Particles below ERR that are tracked are given a weight W.September 4, 2014
D. Lange, ACAT 2014 10
Russian Roulette now used by default after long tuning and validation process
• RR factor of W=10 for neutrons and gammas found to give between 25% and 40% performance improvement with no observable effect on physics output– Energy and shower shape response in the high-resolution ECAL
barrel detector were the most sensitive to RR parameter tuning
September 4, 2014
D. Lange, ACAT 2014 11
10% performance gain from hidden visibility without playing with linker scripts
Repackage all shared libraries in CMSSW that depend on Geant4 into a single static library to “hide” Geant4 from the rest of CMSSW• Use single archive library for Geant4 itself• This allowed us to more aggressively optimize at link time:
adding “-flto -Wl,--exclude-libs,ALL” works best
Constraints this imposes:• Must control dependencies to use Geant4 only within this single library:
– This is “easy” for simulation (<2% of our libraries)– However, extending this idea to something effecting the full reconstruction is
difficult• Impact on simulation code developers minimized by keeping .so cached
in release. Static library rebuild is the only extra step if developer builds a package in this static library. September 4, 2014
D. Lange, ACAT 2014 12
Simulation approach: Hardscatter and pileup events simulated separately and “mixed” together
Physics Generators
Geometry/Material Description
Geant 4
Electronics Simulation
Noise ModelSimulated Raw Data
Particle 4-vectors
Simulated HitsSimulated Hits from Pileup Interactions
= “Digitization”September 4, 2014
D. Lange, ACAT 2014 13
Simulating Extreme Luminosities: The “old” way
• Model pileup byincluding G4hits fromMinBias events generatedseparately from the hard-scatter event
• The pileup interaction simulation
Simulated HitsSimulated Hits from Pileup Interactions
Electronics Simulation
From “hard-scatter” MC event
From individual minbias events
Simulated Hits from Pileup Interactions
Simulated Hits from all interactions in BX N
Simulated Hits from all interactions in BX N+1
Simulated Hits from all interactions in BX N+2
…
This loads all interactions in all beam crossings – all in memory simultaneously! ⇒ unsustainable at HL-LHC luminosities: ~140 interactions x 16 BXs
= 2240 events in memorySeptember 4, 2014
D. Lange, ACAT 2014 14
Modifications to allow very high pileup simulation within memory constraints• We re-factored the pileup simulation to process each interaction sequentially
– Required substantial rewrite of digitization code, and the re-organization of internal event processing
• The content of individual interaction events is dropped once they have been processed: – only 1 event in memory at any given time, so arbitrarily many pileup events can be
included in the digitization
Simulated hits from one interaction
“Accumulate”hits/Energy
Electronics Simulation
repeat until all interactions are processed, including “hard-scatter”
THEN
September 4, 2014
D. Lange, ACAT 2014 15
We are now solving the remaining I/O issues with high pileup simulations
• Even now that the MinBias events are read one-by-one, we still must read more than 2000 minbias event to produce a single event with appropriate pileup at HL-LHC luminosities
• Potential Solution: “Pre-Mixing”– Create library of events containing only pileup contributions,
following pre-determined luminosity profile to calculate how many interactions to include
Simulated hits from one interaction
“Accumulate”hits/Energy
Electronics Simulation
repeat until all minbias interactions are processed
THEN
Simulated Raw Data
September 4, 2014
D. Lange, ACAT 2014 16
• The hard-scatter sample is created and processed through the digitization step with no pileup, convert to our raw data format
• Then the two streams are merged
• Only 1 pileup event is needed for each “hard scatter” MC event– Much easier to process through computing infrastructure once the premixing
sample is created
The hard-scatter and pileup events are combined as “RAW” data format events
Physics Generators Geant 4 Electronics
SimulationSimulated Raw Data
Simulated Raw Data
Simulated Raw Data
Unpacking (RawToDigi)
Unpacking (RawToDigi)
Digi-Mixer (channels)
Reconstruction, etc.
September 4, 2014
D. Lange, ACAT 2014 17
Premixing deployment• Initial version implemented and deployed for part of our large
2015 preparation samples– Production tests went smoothly and demonstrated gains in CPU
efficiency at high pileup– We have done an extensive validation against our normal mixing
simulation approach. At this point, there are only a few known physics issues left to solve (saturation effects)
• Need to take care with the re-use of events.– Generating 1 pre-mixed event for each hard-scatter event (e.g., 10
billion) would significantly reduce the potential benefit• We are evaluating the statistical issues with reuse of pileup events
– We have always reused individual MinBias events in the pileup simulation, but with pre-mixing, we would reuse groups of MinBias events
• In any case, care is needed in the MC production system to avoid excess resuse of pre-mixed events
September 4, 2014
D. Lange, ACAT 2014 18
Managing CMS detector configurations
• We expect that a single “CMSSW” version will support simulation and reconstruction of past, current and future detector configurations– We are not there yet as we have a release for the 2010-2016
configurations and another for 2017-202X• Challenges
– The geometry and detailed channel-level information is part of our conditions database
– Limitations in detector numbering schemes: Most were written specifically to the Run1 CMS detector specification. These are the biggest issue for generalizing our software
– Application configuration needs to be flexible for both detector and LHC conditions differences
September 4, 2014
D. Lange, ACAT 2014 19
Not everything that is detector dependent is easily made a condition• CMS uses a PYTHON based configuration language for
SIM/RECO/ANALYSIS job configuration control. – Developers define “modules”– Sequences of modules define a “process” for our workflows
• Upgrade software developers often need to change the configuration from that of the current CMS detector. Examples:– Detector dependent algorithm parameters (detector segmentation)– Pileup dependent tracking configuration – Sometimes it is simply the most convenient way to do something
that could/should be a condition or derived from the geometry
September 4, 2014
D. Lange, ACAT 2014 20
We have so far centralized these configuration changes
• To keep control over these, we kept all configuration changes in one location per detector configuration (10s of lines of python)
def customize_reco(process):process.trkingIter1.ptMin=0.2process.particleFlow.endcapEcalSource=‘shashlik’
…… # 10-200 lines of changes typically needed
return process
September 4, 2014
cms2017Customize.py
D. Lange, ACAT 2014 21
“ERA”s concept will let us de-centralize configuration changes• Top level configuration defines “ERA” (eventually derived from the
geometry and conditions specified)– Handful of parameters that can be used by modules to change their
configuration appropriately
• Moves the development and maintenance of the upgrade detector configurations back to the developers (where the expertise is)
• Better integration with on-going developments for the 2015 startup– Can also be used for the small number of configuration differences needed for
2015 vs 2012 CMS detectors
trkingIter1=EDProducer(“IterativeTrackingModule”, ptMin=0.15)
if era.pileup > 80:trkingIter1.ptMin=0.2
September 4, 2014
D. Lange, ACAT 2014 22
Conclusions
• Recent development work in CMS means a big reduction in simulation resource needs for 2015 even in the face of higher event complexity and trigger rates.
• CMS detector upgrades push us to use today’s software/computing for tomorrow’s event complexity. – The detector upgrade developments have proven to be an
excellent platform for the quick deployment of new simulation features
September 4, 2014