www.eng.it

PROFES 2011 Torre Canne (BA) – Italy, June 21 2011

Proposing an ISO/IEC 15504 Compliant Method for Process Capability / Maturity Models Customization

Jean C.R. HauckHauck

Christiane Gresse von WangenheimGresse von Wangenheim

Fergal McCafferyMcCaffery

Luigi BuglioneBuglione

www.eng.it

UFSC At a glance

Federal University of Santa CatarinaFlorianópolis/Brazil [http://www.ufsc.br]

• 48 Master courses• 38 PhD courses• 62 Undergraduate courses• EUR 268 million annual budget • 25,737 Undergraduate students• 8,543 Graduate students• 34,280 Students

INCoD is to be a reference institute for excellence in research, validation and dissemination to support digital convergence. [http://www.incod.ufsc.br]

The Software Quality Group focuses on scientific research, development and transfer of SE models, methods and tools. [http://www.gqs.ufsc.br]

[http://www.youtube.com/watch?v=V6E1Z5DEuvk]

www.eng.it

DKIT At a glance

Dundalk Institute of Technology is a 90 acre campus situated between Dublin and Belfast (each approximately 50 miles away).

The Institute consists of 4 Schools:

1. Business & Humanities2. Informatics & Creative Arts3. Engineering4. Health & ScienceThe Regulated Software Research Group which is

part of LERO (the Irish Software Engineering Research Centre) is part of the School of Informatics & Creative Media

www.eng.it

Engineering At a glance

ERP ECMIT Security

Plant ManagementSystem

Broadband & MediaManaged Operations

System Int. & System Int. & ConsultancyConsultancy

OutsourcingOutsourcing

SoftwareSoftware

7070

1010

2020

8080

2020

5454

2727

1919

8080

1010

1010

FinanceFinance IndustryIndustry TELCOTELCO UtilitiesUtilities

%%

%%

%%

ResearchResearch and and DevelopmentDevelopment

3535

1919

4646

PA & HCPA & HC

_ The first Italian ICT player

_ more than 730 M/€ revenues_ 1000 clients_ 6,300 IT specialists

www.eng.it

5 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Goals of the presentation

G1. Discuss the daily need to tailor, applied to the SPI world G2. Analyze the state-of-the-art about SPCMMs customizing G3. Propose a 5-step process and related basic techniques from a SLR analysis

6 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Agenda

• Introduction– Customize in the ‘real’ world– …and in the SPI world

• Requirements for SPCMMs– Criteria and Characteristics

• Existing Methods for SPCMMs Customization– Methods for Developing SPCMMs– Development Process for Standards– Developing a SPCMM in Practic

• A Proposal for a Method for Customizing SPCMMs– Elements to consider– Main phases and Basic Techniques– First results & Discussion

• Conclusions & Prospects• Q & A

7 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Introduction ‘NaturalNatural’ Process Improvement

• StalactitesStalactites – 1 cm – 1 cm each 50-60 yearseach 50-60 years

• StalagmitesStalagmites – 1 cm – 1 cm each 100-120 yearseach 100-120 years

• …have our companies also a (mid-)long term vision for process improvement?

8 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Introduction Customize in the ‘real’ world...

9 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Introduction Customize in the SPI world...

• How to customize a SPCMM and which elements to take into account?

... ...

10 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Agenda

• Introduction– Customize in the ‘real’ world– …and in the SPI world

• Requirements for SPCMMs– Criteria and Characteristics

• Existing Methods for SPCMMs Customization– Methods for Developing SPCMMs– Development Process for Standards– Developing a SPCMM in Practic

• A Proposal for a Method for Customizing SPCMMs– Elements to consider– Main phases and Basic Techniques– First results & Discussion

• Conclusions & Prospects• Q & A

11 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Requirements for...

Criteria Characteristics

Dimensions 1D 2D Hierarchic

Representation Continuous Staged

Audience Unique Multiple

Assessment Approach Qualitative Quantitative

Criteria & Characteristics (1)

• Several classification for requirements expressing the capability/maturity Several classification for requirements expressing the capability/maturity of an organizationof an organization Becker et al. (2009) 7 criteria

o (i) comparison with existing models, (ii) iterative development, (iii) model; evaluation, (iv) multi-methodological procedure, (v) identification of the relevance of problem, (vi) problem definition, (vii) published results and (viii) scientific documentation;

Matook & Indulska (2009) 5 criteriao generality, flexibility, completeness, usability and comprehensibility

Lahrmann & Marx (2010) 4 criteria, as follows

12 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Requirements for... Criteria & Characteristics (2)

• ISO/IEC ISO/IEC 15504-215504-2 proposes specific criteria for PRM and PAM proposes specific criteria for PRM and PAM

• R1 - Declaration of the specific domain and community of interest, including aspects of consensus achievement;

• R2 - Description of processes including: unique title, purpose and outcomes;• R3 - Presentation of the existent relationships between processes;• PAM• R4 - Statement of scope and coverage of the model;• R5 - Indication of the capability levels selected from a measurement

framework for the processes, starting at level 1;• R6 - Mapping for the selected processes of the chosen PRM(s) ;• R7 - Details of performance indicators of the processes, mapped to the

purposes and outcomes of selected the processes of PRM(s);• R8 - Detailed process attributes of measurement framework;• R9 - Objective evidence that the requirements are fulfilled.

13 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Agenda

• Introduction– Customize in the ‘real’ world– …and in the SPI world

• Requirements for SPCMMs– Criteria and Characteristics

• Existing Methods for SPCMMs Customization– Methods for Developing SPCMMs– Development Process for Standards– Developing a SPCMM in Practic

• A Proposal for a Method for Customizing SPCMMs– Elements to consider– Main phases and Basic Techniques– First results & Discussion

• Conclusions & Prospects• Q & A

14 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Existing methods... Methods for developing SPCMMs

• Several methods proposed during the past years:Several methods proposed during the past years:

• de Bruin et al. (2005) – 6 steps sequence for developing a Maturity Assessment Model, not addressed in detail the customization issue

• Mettler et al. (2009) – 6 main steps from a model user perspective • Maier, Moultrie & Clarkson (2009) – guide for developing maturity grids

but not focusing on SPCMMs• Salviano et al. (2009) – PRO2PI framework, by a 7-step process, but not

details about how to customize SPCMMs• Matook & Indulska (2009) – QFD-based approach for keeping the Voice of

the Customer (VoC), with a quality measure as outcome• Becker et al. (2009) – Set of defined requirement for development but no

details about the evolution of such models

• NoteNote: none of them is specifically targeted to meet ISO/IEC 15504-2 : none of them is specifically targeted to meet ISO/IEC 15504-2 requirements!requirements!

15 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Existing methods... Development Process for Standards (1)

www.iso.org/iso/stage_codes.pdf

16 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Existing methods... Development Process for Standards (2)

www.iso.org/iso/stage_codes.pdf

17 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Existing methods... Developing a SPCMM in Practice

SourceSource: Gresse von Wangenheim C., Hauck J.C.R., Zoucas A. Salviano C.F., McCaffery F., Shull F., Creating Software Process Capability/Maturity Models. IEEE Software, vol. 27 no. 4, pages 92 -94, July/August 2010.

• 52 SPCMMs analyzed• Only 21% of those models (n=11) presented

detailed info about their development, 27% (n=14) superficial ones, 52% (n=27) none

18 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Agenda

• Introduction– Customize in the ‘real’ world– …and in the SPI world

• Requirements for SPCMMs– Criteria and Characteristics

• Existing Methods for SPCMMs Customization– Methods for Developing SPCMMs– Development Process for Standards– Developing a SPCMM in Practic

• A Proposal for a Method for Customizing SPCMMs– Elements to consider– Main phases and Basic Techniques– First results & Discussion

• Conclusions & Prospects• Q & A

19 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

A proposal for... Elements to take into account

1. Standards development procedures

2. Existing methods for developing SPCMM/grids

3. Current way to customize SPCMMs

4. KE techniques

20 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

A proposal for... Main Phases & Basic Techniques

1. Knowledge

Identification

2. Knowledge

Specification

3. Knowledge

Refinement

4. Knowledge

Usage

5. Knowledge

Evolution

Method Phase Basic Technique(s)1. Knowledge Identification Ontology Development; Glossary Development; Literature Review;

Systematic Literature Review; Goal Question Metric; Expert Selecting; Delphi; Focus Groups

2. Knowledge Specification Delphi; Perspective-Based Reading; Checklist-based Reading; Semantic Mapping; Domain quality requirements elicitation; Focus groups; Structured Interview; Nominal group; Software Process QFD; Process Selection

3. Knowledge Refinement Expert Selecting; Delphi; Guidelines of Modeling; Behavior Engineering; Interrater Agreement; Checklist-based Reading

4. Knowledge Usage Goal Question Metric; Practical Software and Systems Measurement

5. Knowledge Evolution Model change request management

21 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

A proposal for... First Results & Discussion

• Some application of those concepts, by Explanatory Case StudiesSome application of those concepts, by Explanatory Case Studies

SPCMM Comments/Notes1. SaaS domain Software solution offered as a service, developed using SOA

SaaS scenario requires specific Non-Functional Reqs (e.g. security, availability, service continuity…) needs a SPCMMs customization

SaaS SPCMM [74] developed at UFSC adopting steps #1-3, not yet performed steps #4-5

2. Medi SPICE – http://medispice.ning.com

customization of ISO/IEC 15504 for software development of medical devices

Particularities due to regulatories approvals within the region(s) in which the medical device will be marketed (e.g. AAMI/IEC 62304, FDA, etc...)

Performed steps #1-3 from Jan-Dec 2010

22 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Agenda

• Introduction– Customize in the ‘real’ world– …and in the SPI world

• Requirements for SPCMMs– Criteria and Characteristics

• Existing Methods for SPCMMs Customization– Methods for Developing SPCMMs– Development Process for Standards– Developing a SPCMM in Practic

• A Proposal for a Method for Customizing SPCMMs– Elements to consider– Main phases and Basic Techniques– First results & Discussion

• Conclusions & Prospects• Q & A

23 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Conclusions && Perspectives

• Tailoring, Customizing and SPI Continuous need to find for the ‘best fit’, both in the real world as well as in SPI ISO/IEC 15504 family of models is an example, contextualizing SPICE to several

domains• State-of-the-art and main requirements

Several studies for determining the typical requirements for developing a SPCMM... ...but few information about how to customize them

A proposal for customizing SPCMMs Four elements to take into account, jointly with a Knowledge Engineering (KE)

approach 5 steps (identification; specification; refinement; usage; evolution)

Some lessons learned It doesn’t exist an ‘absolute’ model, each model has a logical boundary as well as its

pros & cons to know for been selected and chosen for building a new model A combination of 2+ SPCMMs is better than running 2+ models in parallel (less effort

and costs) the reference point is your management system, not the SPCMM

If you are out to describe the truth, leave elegance to the tailor. (Albert Einstein)

24 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Q && A

Grazie per l’attenzioneGrazie per l’attenzione!!Thanks for your attentionThanks for your attention!!

25 PROFES 2011 – Torre Canne (BA), June 21, 2011– © 2011 L.Buglione

Customizing SPCMMs Contacts

Luigi Buglione

Engineering.IT/ETSluigi.buglione@eng.it

Jean Carlo R. HauckUFSC

jeanhauck@gmail.com

C. Gresse von Wangenheim

UFSCgresse@gmail.com

FergalMcCaffery

DKITfergal.mccaffery@dkit.ie