Presentation at CHEP 2006
-
Upload
softwarecentral -
Category
Documents
-
view
280 -
download
2
description
Transcript of Presentation at CHEP 2006
The LCG SPI project in LCG Phase II
CHEP’06, Mumbai, India
Feb. 14, 2006
Andreas Pfeiffer -- for the SPI team
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 2
Overview
• Introduction and Motivation• Services of SPI
• Savannah service• External software and distribution services• Configuration and Build Infrastructure• Software testing and QA services• Documentation service
• Outlook and future plans• Summary
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 3
The SPI project in the LCG Applications Area
• Goal of the project • Provide to the development projects of the LCG Application Area
- Basic environment for physics SW development
- General scientific libraries and class libraries
- Software development tools
- Documentation tools and document templates
- Compiler expertise
• Have consistency and homogeneity in the development of the different packages of the LCG Application Area
- Configuration management
- Central software repositories
- Common policies (naming, building, etc.)
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 4
Software Process and Infrastructure
• Work guided and prioritized by Architects Forum• Providing services to the projects (and experiments)• Channelling the requests from projects (and experiments)
• Providing the following services:• Savannah service
- issue and task tracker
• Software services - central installations of external software, software distribution
• Software development service - configuration mgt, pre- and post-build service, tools for testing
and QA
• Web and Documentation
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 5
Savannah Service
• Web portal for LCG software projects• Single entry point for all projects
- Uniform “look and feel” for all projects
• Totally web based
• Used for various functionalities• Bug/issue tracking• Task management• Download area
• Customized from GNU savannah project • In collaboration with main developer
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 6
0
200
400
600
800
1000
1200
1400
1600
aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'06
Adoption by projects
1575 users
0
50
100
150
200
aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'06
176 projects
0
2000
4000
6000
8000
10000
12000
14000
aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'060
500
1000
1500
2000
2500
3000
aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'06
2770 tasks13500 bugs
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 7
Functional enhancements
• Project specific 'cookbooks' • based on the tracker engine• Inherits the tracker functionality while providing a friendly
book-like item display.
• Export facility • Helps extracting data to insert into reports (new releases,
statistics, project monitoring, etc)
• Voting system • Gives registered users the possibility to prioritize the way
items are handled
• Additional browsing criteria• Selection of items modified/unmodified since a certain date
• Extended transition mechanism• Automatic update of multiple fields on specific value
transitions
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 8
External Software Services
• Install software needed by LCG projects and experiments• Open source and public domain software
- Libraries and tools
• Standard structure of directories (central AFS repository)• <packageName>/<version>/<platform>/(lib, src, bin, …)
• Build and dependency information now stored in XML files with scripts to build (semi-)automatically • Very easy to build a new version or new platform• Easy to convert to formats for other tools (e.g. pacman)
• Experiments and projects decide what is installed• Through “Architects Forum”
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 9
External software service -- example
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 10
Software Distribution Service
• Existing solution since 2002• Script to download and install tar-balls of binaries for
software of LCG AA projects with all dependencies• Limited to binaries of platforms build at CERN• Dependency information from configuration management
and build tool (scram)
• Effort started to download source tarballs and (re-) build as needed• Simplifies porting to new platforms (or development work)• Re-using build and dependency information from XML files• Will be extended to download/install binaries
• Work ongoing to fully automate this service
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 11
Configuration and Build Infrastructure
• LCG Librarian• Maintaining configuration information centrally for build
systems in use- XML files + converters to scram and cmt
• Coordinating releases and pre-releases
• Librarians integration group• Discuss with experiments’ librarians/integrators additional
needs on external packages• Bi-weekly meeting (LIM)
• Developed simple tools to automate builds of LCG AA software• Three phases: pre-build, build (on all platforms in parallel),
post-build
• Work ongoing to improve automation• Provide tools for release managers in projects to do this
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 12
Software Testing Service
• Providing Test frameworks, support, policies and documentation• CppUnit, Oval• Qmtest• valgrind• On different compilers/platforms
• Running the tests is an integral part of the development in the projects• Unit tests, integration tests and regression tests
• Also available for experiments• Initial setup• User support
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 13
Quality Assurance Service
• Central QA checklist on each release of LCG AA projects• Code coverage• Providing statistics on tests and from savannah as a
coherent view
• Automated procedure to generate reports• At every release• Published on the SPI web site
• Evaluation and usage of external tools• E.g. RuleChecker, Logiscope
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 14
QA example (ROOT project)
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 15
QA example (POOL project)
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 16
Documentation Service
• Document LCG Policies• Homogenize the projects “look and feel”• CVS and release directory structures• Software testing policies
• Code documentation• Through SPI web site and wiki pages• Tools used: doxygen and LXR• Run for every release of LCG AA projects plus some
external packages (CLHEP, ROOT, HepMC)- Procedure is now semi-automated
• Internal procedures documented• “explicit knowledge” rather than “reading scripts”
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 17
Web and Documentation
• Maintain and improve existing web pages• Update internal and external documentation (wiki pages)• SPI and LCG Workbooks
• Automate content wherever possible• Create more “active/dynamic” web pages
- E.g.: status of external tools, configuration info- New: pages to do pre- and post- build procedures
• Update and clarify existing policies• To be reviewed by AF
• Update web pages for external packages• Responsibles, status, … for each package
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 18
Challenges for LCG phase II
• Servicing LHC experiments more than AA projects• Discussion with experiments to (re-)define scope
- What is needed, what can be improved
• Participation from people in experiments• “On-the-spot” help with requests from experiments• SPI provides a “frame” to do complex work
- Guidelines, AFS space, web space for docs, …
• Coping with strong (50%) reduction of resources • Optimization opportunities • Further optimization and automatization of procedures
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 19
Summary
• SPI provides a large set of stable services that fully use the current resources• Savannah, External Software, Testing Frameworks, Software
Distribution, Build and Release, QA Activities
• SPI work is guided by the Architects Forum• AF represents the users and steers our priorities
• In Phase II we will concentrate more on the experiments’ needs• Who see LCG AA project software as “external”
• Ongoing discussion with AF and experiments• To redefine the scope of the project
- Resources are significantly reduced
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 20
“Thank you!” to the SPI team(s)
• 2005• Alberto Aimar, Johanne Benard, Ian McLaren,
Yannick Patois, Yves Perrin, Andreas Pfeiffer, Eric Poinsignon
• 2006• Full time:
Ian McLaren, Yves Perrin, Andreas Pfeiffer• Part time:
Bertrand Bellenot, Manuel Gallas, Stefan Roiser
14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 21
Additional Information
• SPI project web:• http://spi.cern.ch
• SPI project wiki pages:• https://twiki.cern.ch/twiki/bin/view/SPI/WebHome
• LCG Applications Area web:• http://lcgapp.cern.ch
• LCG web:• http://cern.ch/lcg