Servicing HEP experiments with a complete set of ready integrated and configured common software...
-
Upload
clifford-carroll -
Category
Documents
-
view
222 -
download
0
Transcript of Servicing HEP experiments with a complete set of ready integrated and configured common software...
![Page 1: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/1.jpg)
Servicing HEP experiments with a complete set of ready integrated and configured common software components
Stefan Roiser 1, Ana Gaspar 1, Yves Perrin 1, Karol Kruzelecki 2
CERN PH/SFT1 & CERN PH/LBC2
![Page 2: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/2.jpg)
Outline
• LHC Software layers and their testing infrastructure
• How to use LCG software in a non LHC environment
26 March '09 Stefan Roiser - CHEP '09 2
![Page 3: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/3.jpg)
LCG Applications Area
• Develops and maintains basic software components shared by LHC experiments– E.g. Geant4, ROOT, POOL, COOL, CORAL, etc. – On multiple platforms / compilers / architectures– Released in “LCG Configurations” on demand
• Other fields– Research projects
• Virtualization, Multicore – Communication Tools
• Savannah, Hypernews
26 March '09 Stefan Roiser - CHEP '09 3
![Page 4: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/4.jpg)
Windows (XP)
Mac OSX (10.5)
LHC Software
LCG / AA external software
LCG / AA external software
Python
BoostQt
Xerces
GSLvalgrind
Grid … ~70packages
LCG / AA projectsLCG / AA projects ROOT
POOL COOL CORAL
RELAX
Linux (slc4, slc5)
LHC Experiment Software
LHC Experiment Software
AliRoot
CMSSW
Gaudi
Athena
Xgcc 4.0icc 11
gcc 3.4gcc 4.3 llvm 2.4
vc 7.1
vc 9
32 bit
64 bit
= ~ 20 different platforms
LCG
Con
figur
ation
26 March '09 Stefan Roiser - CHEP '09 4
Java
![Page 5: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/5.jpg)
LCG Releases
• LCG Configurations released on demand– Schedules discussed with experiments in
Architects Forum– Usually 2 major release series / year
• Targeting major changes e.g. new compilers / platforms• + bug fix releases on top when needed
26 March '09 Stefan Roiser - CHEP '09 5
2008 2009
LCG54
LCG 55
LCG 56
![Page 6: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/6.jpg)
LCG Project Software Testing
26 March '09 Stefan Roiser - CHEP '09 6
Every dayEvery day
Different platformsDifferent platforms
Build & TestBuild & Test
All LCG/AA projectsAll LCG/AA projects
Different ConfigurationsDifferent Configurations Test History
Test History
V. Diez; Geant4 testing integration into LCG nightly builds system
![Page 7: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/7.jpg)
LHC Software Testing Stack
26 March '09 Stefan Roiser - CHEP '09 7
K. Kruzelecki; The nightly build and test system for LCG AA and LHCb software
![Page 8: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/8.jpg)
LCG s/w in a non LHC environment
Neutrino
Astrophysics
Cosmic rays
Dark matter
Dark energy
Space
Fixed Target
Cosmology
Gravity
Infrared
Gamma ray
• Software built for LHC experiments
…
26 March '09 Stefan Roiser - CHEP '09 8
Many Physics experiments around (> 600) *
* http://www.slac.stanford.edu/spires/experiments/online_exp.shtml
• What about making LCG Configurations usable outside LHC?
![Page 9: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/9.jpg)
3 ways to get LCG ConfigurationsCERNVM Deploy LCG binaries Recompile sources
MotivationAutomatic deployment of LCG/AA software
FunctioningUsage of virtualization technologies
Software deploymentDone via network + special caching file system
PrerequisitesHypervisor + Cernvm image
MotivationRun native on LCG/AA provided platform
FunctioningBinary packages provided by LCG/AA infrastructure
Software deploymentDownload of packages via web interface
PrerequisitesLCG/AA compliant os
MotivationRun native on non LCG/AA provided platform
FunctioningRecompilation of all LCG/AA packages from source
Software deploymentLocal deployment by user
PrerequisitesLinux/Mac OSX (!windows)
11 22 33
26 March '09 Stefan Roiser - CHEP '09 9
![Page 10: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/10.jpg)
CERNVM1
26 March '09 Stefan Roiser - CHEP '09 10
P. Buncic; CernVM - a Virtual Software Appliance for LHC applications
• Virtual machine running with several hypervisors & host platforms
• CVMFS allows download & caching of only needed parts for execution of the application– Usually a fraction of the originally deployed stack
To be
deployed
![Page 11: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/11.jpg)
Deploying LCG/AA binaries
1 Go to http://lcgsoft.cern.ch
2 Choose your LCG Configuration
3 Choose needed packages and download tar file
2
26 March '09 Stefan Roiser - CHEP '09 11
![Page 12: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/12.jpg)
Build LCG/AA s/w from source
PythonPython
CMTCMT
LCGCMTLCGCMT
3
26 March '09 Stefan Roiser - CHEP '09 12
We are using the sameTools and procedure forall LCG/AA packages for • Nightly builds• AFS installations• User recompilation
![Page 13: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/13.jpg)
How to build an LCG/AA package
1 Bootstrap the 3 necessary pack-ages, i.e. down-load and install
1 Python2 CMT3 LCGCMT
3
26 March '09 Stefan Roiser - CHEP '09 13
![Page 14: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/14.jpg)
How to build an LCG/AA package
2 Override default values if needed + setup environment
3
26 March '09 Stefan Roiser - CHEP '09 14
macro LCG_builddir “/build”macro LCG_home “/opt/lcg”
tag x86_64-ubuntu810-gcc43-opt target-linux target-x86_64 target-gcc43 target-opt
macro LCG_builddir “/build”macro LCG_home “/opt/lcg”
tag x86_64-ubuntu810-gcc43-opt target-linux target-x86_64 target-gcc43 target-opt
lxplus:~> export CMTCONFIG=x86_64-ubuntu810-gcc43-optlxplus:~> . /opt/lcg/sw/CMT/v1r20p20081113/mgr/setup.sh lxplus:~> export CMTCONFIG=x86_64-ubuntu810-gcc43-optlxplus:~> . /opt/lcg/sw/CMT/v1r20p20081113/mgr/setup.sh
![Page 15: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/15.jpg)
How to build an LCG/AA package
3 Repeat standardized build procedure for the needed set of LCG/AA packages
3
26 March '09 Stefan Roiser - CHEP '09 15
lxplus:~>cd LCGCMT_56/LCG_Builders/ROOT/cmtlxplus:cmt> cmt pkg_getlxplus:cmt> cmt pkg_configlxplus:cmt> cmt pkg_makelxplus:cmt> cmt pkg_installlxplus:cmt> cmt pkg_testlxplus:cmt> ls …/ROOT/5.22.00a/x86_64-ub81-gcc43-opt/rootbin etc geom include LICENSE man testcint fonts icons lib macros README tmvalxplus:cmt> ls logs ROOT_x86_64-ubuntu810-gcc43-opt_config.logROOT_x86_64-ubuntu810-gcc43-opt_make.log … … …
lxplus:~>cd LCGCMT_56/LCG_Builders/ROOT/cmtlxplus:cmt> cmt pkg_getlxplus:cmt> cmt pkg_configlxplus:cmt> cmt pkg_makelxplus:cmt> cmt pkg_installlxplus:cmt> cmt pkg_testlxplus:cmt> ls …/ROOT/5.22.00a/x86_64-ub81-gcc43-opt/rootbin etc geom include LICENSE man testcint fonts icons lib macros README tmvalxplus:cmt> ls logs ROOT_x86_64-ubuntu810-gcc43-opt_config.logROOT_x86_64-ubuntu810-gcc43-opt_make.log … … …
![Page 16: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/16.jpg)
Real world example
• Software for large water cerenkov detectors eg. DUSEL (US), MEMPHY (EU)
• Packages rebuilt so far with LCG/AA procedure for ubuntu 8.04– GAUDI, ROOT, RELAX, GAUDI, Cmake , Python,
Clhep, Gccxml, XercesC, HepMC, HepPDT, GSL, Bjam, Boost, Mysql, QMTest, CppUnit
• New package build information was fed back to LCG/AA for OpenScientist
3
26 March '09 Stefan Roiser - CHEP '09 16
![Page 17: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/17.jpg)
Advantages of LCG Configurations• Self-consistent sets of basic software packages
– Use of recent packages / tools– HEP specific patches when needed
• Several deployment method possible– Virtual Machine, LCG/AA binaries or recompilation
• Multi platform / architecture / compiler– Continuous performance / unit / integration testing– Adding to overall software stability
• Flexible adaptation possible– Using e.g. CMT as configuration tool
26 March '09 Stefan Roiser - CHEP '09 17
![Page 18: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/18.jpg)
Future enhancements
• Software deployment– Integration with CERNVM– Package deployment system for binaries (rpm?)– Tool for bootstrap procedure when recompiling
• !!! Documentation !!!
26 March '09 Stefan Roiser - CHEP '09 18
![Page 19: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/19.jpg)
Summary
• LCG Applications Area builds and tests the basic software packages for LHC experiments
• Non LHC experiments may use and profit from – Consistent set of packages– Multi platform/architecture/compiler– Agile package version policy– Continuous integration, performance, unit testing
26 March '09 Stefan Roiser - CHEP '09 19
![Page 20: Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.](https://reader035.fdocuments.net/reader035/viewer/2022062305/5697bfea1a28abf838cb7322/html5/thumbnails/20.jpg)
Further Info• Announcements of new LCG Configurations
– [email protected]• Details about LCG Configurations (packages, versions, …)
– http://lcgsoft.cern.ch• Minutes of the Architects Forum (LCG/AA steering committee)
– http://lcgapp.cern.ch/project/mgmt/af.html• Software Process and Infrastructure project
– http://spi.cern.ch• LCG/AA nightly builds overview page
– http://lcgapp.cern.ch/spi/cgi-bin/nightlies.py• CMT configuration and management tool
– http://www.cmtsite.org
26 March '09 Stefan Roiser - CHEP '09 20