EGEE is a project funded by the European Union under contract IST-2003-508833

Software Configuration Management

and Integration

Alberto Di Meglio

EGEE JRA1 Integration Manager

EGEE Kick-Off Meeting – Cork – 18-22/04/2004

www.eu-egee.org

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 2

Contents

• JRA1 Integration Mandate

• Team Objectives

• The SCM Plan: an Overview

• SCM and Integration Implementation

• Configuration and Version Control

• Build Systems

• Release Process

• Other Configuration and Change Control Procedures

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 3

JRA1 Integration Mandate

• EGEE JRA1 has a dedicated team for SCM and Integration activities

• The vision of the team is that SCM and Integration must start before a single line of code is written and must provide an integrated development and collaboration environment

• The mission is to provide a solid, reliable infrastructure to manage changes and release quality software based on industry-standard SCM procedures and tools throughout the entire duration of the project

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 4

Integration Team Objectives

• The Integration System must be simple, flexible and solid

• Excessive diversity in tools, languages, technologies must be avoided

• Centralization of processes and procedures is favoured

• Whenever possible industry-standard tools and methods with a large user base and reliable, official support must be used

• Making it work well on one platform is preferable to making it available on all platforms (at least initially)

• Extension to multiple platforms and usage scenarios will naturally follow for a well designed, developed and integrated software

• EGEE Middleware is a complex, multi-disciplinary, highly distributed project. This is part of the novelty and challenge of it and will require a novel approach to old and new problems and some courageous decisions

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 5

SCM Plan Overview

• A master Software Configuration Plan is being finalized now [1]

• It contains basic principles and rules about the various areas of SCM and Integration (version control, release management, build systems, bug tracking, etc)

• Compliant with internationally agreed standards (ISO 10007-2003 E [2], IEEE SCM Guidelines series [3])

• Most EGEE stakeholders have already been involved in the process to make sure everybody is aware of, contributes to and uses the plan

• An EGEE JRA1 Developers’s Guide will follow shortly in collaboration with JRA2 (Quality Assurance) based on the SCM Plan

• It is of paramount importance to deliver the plan and guide as early as possible in the project lifetime

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 6

SCM and Integration Implementation

• As soon as the SCM Plan is finalized, we will start the implementation phase (some steps already taken)

• All tools, servers, procedures and communication channels will be put in place

• Developer’s and User’s Guides will be produced with clear instructions about how to use and interact with the project and manage changes in the correct way

• The complete infrastructure will be in place at the end of Month 3 (30 June) (MJRA1.1)

• Goals for the first 12 months are to release incremental code baselines to test with a very rapid lifecycle (weekly) and finally release v. 1.0.0 (main milestone is MJRA1.5 at Month 10: integrated EGEE RC 1 released to testing)

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 7

Version and Configuration Control

• Based on CVS using CERN Central IT CVS service

• Fixed directory structure for each module

• Rules for tagging and branching (e.g. bug fix branches)

• Common naming conventions for baseline and release tags

• Configuration files to automate creation of workspaces Used to enforce build reproducibility Used to create private workspaces for developers

• Authorization based on commitavail and tagavail according to SCM roles (developers, release managers, integrators, etc.)

• Supported protocols: K4, SSH (pserver only for anonymous read-only). A CERN AFS account is required

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 8

CVS Structure

JRA1 EGEECVS

org.egee

org.egee.subsys

org.egee.subsys.comp1

org.egee.subsys.compN

project

src

test

doc

interface

examples

config

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 9

Workspaces

• A Workspace is one of the basic SCM Patterns [4]• It is a private area where a build can be produced under configuration

control• More independent workspaces can coexist on the same computer for

parallel development• The workspace is created using a Configuration Specification File

(CSF)• The CSFs contain all the information necessary to prepare the

workspace extracting modules and other dependencies from CVS or the central software repository

• There is a CSF file for the overall org.egee Middleware suite and one CSF for each subsystem

• CSFs are stored in the corresponding modules in CVS and versioned so that any version of the MW suite and all its subsystems can be reproduced at any time

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 10

Build System

• Common Build System A project-wide build system based on Ant to allow all developers,

integrators and testers to manage private builds on their own computers with the same tools and settings (compilers, flags, configurations, etc)

Based on the concept of Private Workspaces, more than one independent workspace can be created on the same computer

Uses a set of common build and properties files to define targets and properties for all project modules

External dependencies are stored and versioned either in CVS or a central repository maintained by CERN SPI. All modules must use the SAME versions of the external dependencies across the project

• Central Build System Located at CERN and maintained by the JRA1 Integration Team Based on the Common Build System, but automated by means of

CruiseControl and Maven to generate a continuous, automated integration process and the overall project web site (Apache style).

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 11

Build System (2)

• The Ant build system provides the common build orchestration for all modules, not only Java

• Modules can be built independently and use Ant or Make as specified by project requirements, independently from the module implementation language

• A set of common targets has been defined for all modules (environment checks, style checks, compilation, unit tests, distribution, etc)

• The entire MW suite, individual subsystems and individual modules can be built with the same set of targets

• Quality Assurance checks are built into the build process and are mandatory. Failure to pass a quality check (coding styles, unit testing, minimum test coverage, documentation, etc) fails the build

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 12

Build System (3)

• Five main types of builds: Continuous Incremental Builds: every few hours, mail messages are

sent to developers who break the build Nightly build: every night from clean snapshot Weekly builds: every week from clean project configurations Maintenance builds: used to fix software defects on existing

releases Private builds: whenever needed on the private development

computers

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 13

Build Tools

• Ant: used for the general build management and all Java modules. The advantages/disadvantages of using it also for C/C++ and other languages modules are under investigation

• Make + GNU Autotools: for C/C++ and other languages as necessary. Although this is the standard, familiar approach, it has a lot of drawbacks, especially for large, distributed projects (difficult to maintain and centralize, lots of external dependencies, poor internal dependency tracking, file based instead of task based, etc)

• CruiseControl: used to automate the integration builds• Maven: used to automate the project web site, reports and

documentation builds

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 14

Packaging and Configuration

• Installation packages for the EGEE Middleware will be produced by the integration team

• Packages will be provided for the entire middleware and for all subsystems and deployment modules

• Supported packaging formats are: Source tarballs with build scripts Native packages for supported platforms (e.g. RPMs for Red Hat Linux

Enterprise 3.0, MSIs for Windows XP, etc)

• Extracting and using modules directly from CVS is not recommended, since there is no guarantee the dependencies are satisfied outside the build system context

• Configuration issues are being investigated with SA1• The goal is to have a simple, common format for all configuration items,

usable by standard installation/configuration tools• Some type of “Middleware Configuration Front-End” may also be

provided in collaboration with SA1

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 15

For each module in the system

All system modules released

No

Yes

All functional and regression tests

passed No

Release Process

1. Developers works on code functionality and documentation and produce baseline candidates

2. Baseline candidates are continuously integrated and unit and smoke tested

3. A build is successful if all the functionality contained in the module is properly unit tested and documented. If a build is not successful, the baseline candidate is rejected and control returns to the developers

4. Baselines are automatically tagged if the build process is successful

5. When all baseline candidates in the application build list (Application CSF file) are successfully built, an application baseline is produced and passed to testing

6. If all test, certification and validation procedures are successful, the individual baselines in the build are considered eligible to be promoted to release if all required functionality is present

7. Development Release Managers promote eligible baselines to full releases

8. When all components have been released, functionality is frozen and the application is promoted to Release Candidate.

9. The Release Candidate is thoroughly tested, validated and certified, all critical and major bugs are fixed, most normal bugs are fixed. A number of Release Candidates are produced during a short number of iterations.

10. The final release is published

BaselineCandidate

Yes

code

Start state

Developer

Functionality finished and unit

tested No

Integrate

All checks and unit tests passed

All smoke tests passed

No

Yes

No

Integrator

Baseline

Yes

IntegrationRelease Manager

Modulerelease

ModuleRelease Manager

Yes

System release

ProjectRelease Manager

Yes

Release Candidates Iteration

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 16

Change Control Board

• All public changes must go through a formal approval process

• The CCB is tasked to collect and examine the change requests

• Changes must be tracked and communicated as efficiently as possible

• The role of the CCB (and SCM in general) is not to prevent change, but to organize it so it has only positive impacts on the project

• It will meet periodically and on demand

• Need to define composition of the board, very likely it will be composed of the JRA1 activity and cluster managers plus representatives of the architecture and design teams, deployment and applications

• The relationship with the EGEE Architecture Oversight Committee (AOC) is under discussion

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 17

Defect Tracking System

• Bugzilla-like tool provided as part of the Savannah project portal

• Provided and supported by SPI team at CERN

• Need to add tools for collecting QA metrics (current reporting in Savannah is currently not sufficient)

• Used also for change requests (for example API changes, external libraries version changes, etc). In this case, request are assigned to the Change Control Board for further evaluation

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 18

Defect Tracking Cycle

• Ten states: Open, Analyzed, Accepted, In Progress, Ready for Integration, Ready for Test, In Test, Approved, Ready for Review, Closed

• Transition between two states subject to specific checks (Is it a defect? Can it be fixed? Test passed, Review passed?)

• ITeam are first filter to assign bugs to people or categories if required

Open

Is it a defect?Can it be fixed?

AnalyzedNo

Yes

Integrator

Developer

Defect is submitted

Integrators filter the defects and assign them if necessary

to people and categories. Feature requests are reassigned to CCB

Accepted

Write resolutionjustification

Ready for Test

Integrator

Integrators create buildIncluding fix

In Test

Tester

Testers take new release and test it

Test passed?No

Yes

Approved

Ready for Review

Review passed?

No

Yes

Closed

In Progress

Ready for Integration

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 19

Software Quality Assurance

• Software Metrics are collected as part of the build process

• Failure to pass a quality check fails the build

• Additional checks are implemented in the version control system (coding style checks)

• Software Defect and QA Metrics are collected from the defect tracking system

• All reports are published every night on the project web site

• Reports should be used by project and development managers to take corrective actions

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 20

Collaboration Tools

• Project Portal: Savannah from CERN SPI

• Single entry point for the project, news, tools, documentation

• Links to other relevant sites (CVS web browsing, builds status, package repository, project metrics and other artefacts)

• Mailing lists from CERN or Terena

• News, RSS feeds, etc, as necessary

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 21

Contacts

• ITeam mailing listproject-eu-egee-middleware-integration@cern.chList of the ITeam members and other activities representatives (closed list)

• Integration supportproject-egee-middleware-integration-support@cern.chPublic support list for reporting integration and build problems

EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 22

References

• [1] EGEE Middleware SCM Plan (https://edms.cern.ch/document/446241/0.6)

• [2] ISO 10007:2003 - Quality management systems -- Guidelines for configuration management, ISO, 2003

• [3] http://standards.ieee.org/reading/ieee/std/se/828-1998.pdf (requires a subscription)

• [4] Software Configuration Management Patterns, S.P. Berczuk, Software Patterns Series, Addison-Wesley, 2002