Www.eu-etics.org INFSOM-RI-026753 ETICS: E-infrastructure for Testing, Integration and Configuration...

www.eu-etics.org

INFSOM-RI-026753

ETICS: E-infrastructure for ETICS: E-infrastructure for Testing, Integration and Testing, Integration and Configuration of SoftwareConfiguration of Software

Alberto Di MeglioAlberto Di Meglio

Project ManagerProject Manager

EGEE JRA1 All Hands Meeting- 23 March 2006 - CERN 2INFSOM-RI-026753

The ETICS Partners

Build system, software

configuration, service infrastructure,

dissemination, EGEE, gLite, project coord.

Software configuration, service infrastructure, dissemination

Web portals and tools, quality process, dissemination, DILIGENT

Test methods and metrics, unit testing tools, EBIT

The Condor batch system, distributed testing tools, service infrastructure, NMI


ETICS Organization

Project Manager(Alberto Di Meglio)

WP1Project

administrative andtechnical management

Alberto Di Meglio

WP2Infrastructure and

service managementMarian Zurek

Matteo Selmi

WP3Software configuration

tools andmethodologiesElisabetta Ronchieri

Enrico Ferro, Matteo Selmi, Guillermo Diez

WP4Testing tools andmethodologies

WP5Dissemination,

certificationand web toolsMarc-Elian Begin

ETICS

Experts BoardPMB


Systems status

• ETICS is born especially from the experience of gLiteETICS is born especially from the experience of gLite• The goal is to provide efficient automated tools for building and testingThe goal is to provide efficient automated tools for building and testing• It also must be as simple as possible to useIt also must be as simple as possible to use• Currently the gLite build system allows to:Currently the gLite build system allows to:

– Build the entire gLite project managing automatically internal and external dependenciesBuild the entire gLite project managing automatically internal and external dependencies– Produce packages like source and binary tarballs and binary RPMS for RHEL3/SLC3Produce packages like source and binary tarballs and binary RPMS for RHEL3/SLC3– Trigger coding conventions, unit test, documentation tools and collect and publish the Trigger coding conventions, unit test, documentation tools and collect and publish the

results (more or less automatically)results (more or less automatically)• But we know there are issues and missing functionality:But we know there are issues and missing functionality:

– Dependency mechanism is powerful, but not very flexible and it’s difficult to find Dependency mechanism is powerful, but not very flexible and it’s difficult to find informationinformation

– More package types are neededMore package types are needed– Much better support for multiplatform development is neededMuch better support for multiplatform development is needed– It must be possible to build everything from sourceIt must be possible to build everything from source– It must produce run-time configuration information from build informationIt must produce run-time configuration information from build information– Coupling with CVS and Ant is too tightCoupling with CVS and Ant is too tight– Better support for the release process must be provided (tagging, release notes, quick fix Better support for the release process must be provided (tagging, release notes, quick fix

tracking, etc)tracking, etc)– ……

• And the current gLite system doesn’t provide any automated testing functionalityAnd the current gLite system doesn’t provide any automated testing functionality


Service Overview

Build/TestArtefacts

Web Application

ReportDB

ProjectDB

NMI Scheduler

Clients

Web Service

NMI ClientWrapper

Via browser

Command-Line tools

WNs ETICS InfrastructureContinuous Builds


ETICS Web Interface

• Web Application layoutWeb Application layout


NMI Interface

• Preliminary integration of the client with NMIPreliminary integration of the client with NMI


Continuous builds interface


Command line tools

• ClientClient


Databases and repositories

Build/TestArtifacts

ReportDB

ProjectDB

Contains all project and Contains all project and configuration information, configuration information, currently implemented as a currently implemented as a MySQL DBMySQL DB

Contains the output of builds and Contains the output of builds and test runs. Implemented as a test runs. Implemented as a MySQL databaseMySQL database

Contains the artifacts produced Contains the artifacts produced by build and test runs (packages, by build and test runs (packages, text, files, complete distributions, text, files, complete distributions, etc). Implemented as a file systemetc). Implemented as a file system


User profiles

• DevelopersDevelopers: can do local builds of project, subsystems : can do local builds of project, subsystems and components.and components.

• IntegratorsIntegrators: can create and run integration : can create and run integration configurationsconfigurations

• TestersTesters: can register and submit tests attached to : can register and submit tests attached to configurationsconfigurations

• Release ManagersRelease Managers: can create and publish : can create and publish configurations, create release notes and other configurations, create release notes and other documentationdocumentation

• Project ManagersProject Managers: can create and manage projects, : can create and manage projects, assign permissions, set policies, etcassign permissions, set policies, etc


What will change for you?

DevelopersDevelopers• Impact: Impact: lowlow..

– A few A few new commandsnew commands for creating workspaces and building components for creating workspaces and building components– Syntax is very close to familiar CVS commandsSyntax is very close to familiar CVS commands– The general principle is the The general principle is the same as the gLitesame as the gLite build system with build system with workspacesworkspaces

containing everything that is needed for a build, the stage area, the repository containing everything that is needed for a build, the stage area, the repository area, etc. area, etc.

• Functionality changes: Functionality changes: highhigh– The The smallest unit of buildsmallest unit of build is a component. Dependencies can be set at the is a component. Dependencies can be set at the

component levelcomponent level. No need to download full subsystems anymore, only what’s . No need to download full subsystems anymore, only what’s really needed (if the build configuration is correctly defined).really needed (if the build configuration is correctly defined).

– Three Three build modesbuild modes are supported: are supported:– Source onlySource only, where everything (including external dependencies) is built from sources, where everything (including external dependencies) is built from sources– Binary onlyBinary only, where everything except the modules in the currently built configuration are , where everything except the modules in the currently built configuration are

taken from the repositorytaken from the repository– MixedMixed, where everything is taken from the repository if it exists, otherwise is built from , where everything is taken from the repository if it exists, otherwise is built from

sourcesource– General functionality is General functionality is improved improved (customization, platform detection, speed, etc)(customization, platform detection, speed, etc)– Automatic early validation on Automatic early validation on multiple platforms multiple platforms comes for free without any comes for free without any

additional effort by submitting builds to the ETICS serviceadditional effort by submitting builds to the ETICS service• Timescale: Timescale: From April 2006From April 2006



IntegratorsIntegrators• Impact: Impact: mediummedium

– The definition of The definition of external dependenciesexternal dependencies is now kept in the ETICS DB and the info in the is now kept in the ETICS DB and the info in the Ant XML file is not used by ETICS.Ant XML file is not used by ETICS.

– CVS tags (both for subsystems and components) are still required in the same way as CVS tags (both for subsystems and components) are still required in the same way as now, but they are not taken from any Ant XML file. Instead they are used to define now, but they are not taken from any Ant XML file. Instead they are used to define CONFIGURATIONSCONFIGURATIONS in the ETICS database. in the ETICS database.

– Some equivalent and Some equivalent and improved toolsimproved tools for tagging, branching, etc have to be provided with for tagging, branching, etc have to be provided with support for ETICS configurations managementsupport for ETICS configurations management

• Functionality changes: Functionality changes: highhigh– Much Much easier to navigateeasier to navigate the relationships, make modifications, clone existing the relationships, make modifications, clone existing

configurations, submit configuration to be built on multiple platformsconfigurations, submit configuration to be built on multiple platforms– Better categorizationBetter categorization of dependencies: build-time, run-time or both, per platform of dependencies: build-time, run-time or both, per platform– Automatic creationAutomatic creation of run-time configuration information (environment variables) of run-time configuration information (environment variables)– The The Continuous IntegrationContinuous Integration tool has been redeveloped from scratch in python and it’s tool has been redeveloped from scratch in python and it’s

faster and doesn’t have the CruiseControl memory issues.faster and doesn’t have the CruiseControl memory issues.– More package types are created. The first release of the ETICS packages will be able to More package types are created. The first release of the ETICS packages will be able to

produce:produce:

source and binary tarballssource and binary tarballssource, binary and debug RPMSsource, binary and debug RPMSdebsdebsMSIsMSIs

• Timescale: Timescale: From April 2006From April 2006



TestersTesters• Impact: Impact: highhigh

– It may require a It may require a different way of workingdifferent way of working. Tests can be run by testers on . Tests can be run by testers on multiple platforms and different conditions and test results analysed multiple platforms and different conditions and test results analysed asynchronouslyasynchronously

– Tests may have to follow Tests may have to follow some conventionssome conventions to fully exploit the ETICS to fully exploit the ETICS functionality (but it’s not mandatory)functionality (but it’s not mandatory)

• Functional changes: Functional changes: highhigh– Once a test is available, ETICS allows to Once a test is available, ETICS allows to registerregister it and it and submitsubmit it for it for

execution to any of the available platforms with custom schedulingexecution to any of the available platforms with custom scheduling– All results are All results are archivedarchived in the ETICS database and it is possible to do in the ETICS database and it is possible to do

functional, regression, stress and performance tests across functional, regression, stress and performance tests across different different configurationsconfigurations, builds and releases, builds and releases

– A number of common test output schemas are natively supported. A number of common test output schemas are natively supported. Some conventions may have to be defined/agreed by projects to get full Some conventions may have to be defined/agreed by projects to get full functionalityfunctionality

• Timescale: Timescale: May-June 2006May-June 2006



Release Managers/Project ManagersRelease Managers/Project Managers• Impact: Impact: lowlow

– No particular effort is requested from RM/PM, except to take the time to No particular effort is requested from RM/PM, except to take the time to see what benefits they get from using the new toolssee what benefits they get from using the new tools

• Functionality changes: Functionality changes: highhigh– ETICS supports several projects with access control and rolesETICS supports several projects with access control and roles– Once a configuration has been successfully built and tested, it can be Once a configuration has been successfully built and tested, it can be

frozen and published as a release/RC.frozen and published as a release/RC.– Reporting tools will allow to create release notes and other QA reports Reporting tools will allow to create release notes and other QA reports

directly from the ETICS web interface.directly from the ETICS web interface.

• Timescale: Timescale: Summer 2006Summer 2006


Conclusions

• ETICS provides a multi-platform environment for ETICS provides a multi-platform environment for building and testing middleware and applicationsbuilding and testing middleware and applications

• ETICS focus is on automation: of builds, of tests, of ETICS focus is on automation: of builds, of tests, of reporting, etc. The goal is to simplify life when reporting, etc. The goal is to simplify life when managing complex software management tasksmanaging complex software management tasks

• We want it to be as simple as possible to use. For this We want it to be as simple as possible to use. For this we need your feedback:we need your feedback:

[email protected]@cern.ch

[email protected]@cern.ch


Thanks

http://www.eu-etics.org

Www.eu-etics.org INFSOM-RI-026753 ETICS: E-infrastructure for Testing, Integration and Configuration...

Documents

Transcript of Www.eu-etics.org INFSOM-RI-026753 ETICS: E-infrastructure for Testing, Integration and Configuration...