Risk management and Process Improvement of Off-The-Shelf Based Development

CBSE Seminar -4 Feb 2005- OSLOCBSE Seminar -4 Feb 2005- OSLO 11

Risk management and Process Risk management and Process Improvement of Off-The-Shelf Based Improvement of Off-The-Shelf Based

DevelopmentDevelopment

Jingyue Li Jingyue Li (jingyue(jingyue@@idi.ntnu.noidi.ntnu.no)), ,

Reidar Conradi, Odd Petter N. SlyngstadReidar Conradi, Odd Petter N. Slyngstad,,

Norwegian University of Science and TechnologyNorwegian University of Science and Technology

Marco Torchiano, Maurizio Morisio,Marco Torchiano, Maurizio Morisio, Dip.Automatica e Informatica, Politecnico di Torino Dip.Automatica e Informatica, Politecnico di Torino

Christian BunseChristian BunseFraunhofer IESEFraunhofer IESE


AgendaAgenda

Research designResearch design• BackgroundBackground• Research questionsResearch questions• Sample selectionSample selection

ResultsResults• Selected samplesSelected samples• Answers to research questionsAnswers to research questions

DiscussionsDiscussions Conclusions and future workConclusions and future work


Research design – MotivationResearch design – Motivation Pre-study backgroundPre-study background

• This study This study followings followings a pre-study with 16 structured a pre-study with 16 structured interviews in Norway, from Oct. 2003 to Feb. 2004.interviews in Norway, from Oct. 2003 to Feb. 2004.

• Focused on Focused on SPISPI in COTS-based development in COTS-based development• Respondents shared a lot of experiences on Respondents shared a lot of experiences on risk risk

managementmanagement in COTS-based development in COTS-based development• Limitations of the pre-study Limitations of the pre-study

Small sample sizeSmall sample size Sample selected Sample selected on convenienceon convenience

Motivation of this main studyMotivation of this main study• State-of-the-practice State-of-the-practice surveysurvey• RandomlyRandomly selected selected much larger samplesmuch larger samples to validate to validate

conclusions of the pre-studyconclusions of the pre-study• Also included Also included Open Source ComponentOpen Source Component


Research design – research Research design – research questions questions

RQ1RQ1 - How to - How to improve the development improve the development processprocess in projects using OTS components. in projects using OTS components.

RQ2RQ2 - How to - How to predict possible riskspredict possible risks (problems)(problems) in projects using OTS in projects using OTS components?components?

RQ3RQ3 - What are the effective methods to - What are the effective methods to mitigate risksmitigate risks in projects using OTS in projects using OTS components?components?

RQ4RQ4 - What are the similarities and - What are the similarities and differences between differences between projects using COTS projects using COTS and OSS componentsand OSS components??


Research design – sample Research design – sample selectionselection

NorwayNorway GermanyGermany ItalyItaly

(Sample selection reported in later (Sample selection reported in later presentation)presentation)


Research results – selected Research results – selected samplessamples

Current dataCurrent data• Total 86 projectsTotal 86 projects• Norway Norway

46 projects from 38 companies46 projects from 38 companies One company filled in 4, one filled in 3, and one filled in 2.One company filled in 4, one filled in 3, and one filled in 2. In other companies, we selected only one project each In other companies, we selected only one project each

companycompany• GermanyGermany

29 projects from 29 companies29 projects from 29 companies• ItalyItaly

11 projects from 11 companies11 projects from 11 companies Data collection is still on-going in Germany and Data collection is still on-going in Germany and

ItalyItaly


Company's main business

43 %

42 %

12 %

3 %

Software house

IT consulting

IT department of atraditional industry

Telecom. Industry

Research results – selected Research results – selected companiescompanies


Company size

29 %

41 %

30 %

Small

Medium

Large

Research results – selected Research results – selected companies (cont’)companies (cont’)

Small (0-19) Medium (20-99) Large (more than 100)


Application domain of the integrated system

26 %

19 %

12 %

20 %

23 % Traditional industry

Bank

Other privateservicesPublic sector

ICT sector

Research results – selected Research results – selected projectsprojects


Role of respondents

24 %

42 %

20 %

14 %IT Manager

Project manager

Software architect

Developer

Research results – selected Research results – selected respondentsrespondents


85% respondents have more than 3 85% respondents have more than 3 years experience on OTS-based years experience on OTS-based developmentdevelopment

Most respondents have the Bachelor Most respondents have the Bachelor degree in informatics, 10% have degree in informatics, 10% have Ph.D degree.Ph.D degree.

Research results – selected Research results – selected respondents (cont’)respondents (cont’)


Research question RQ1Research question RQ1

How to improve the development How to improve the development process in projects using OTS process in projects using OTS components?components? • Overall development processOverall development process

Do I need to change my main development Do I need to change my main development process dramatically in projects using OTS?process dramatically in projects using OTS?

What activities and roles should be added?What activities and roles should be added?

• OTS selection processOTS selection process Formal decision making process?Formal decision making process? Familiar with component process?Familiar with component process?


RQ1: Do I need to change my main RQ1: Do I need to change my main development process dramatically?development process dramatically? More than 80% projects members More than 80% projects members

decided their main development decided their main development process (Waterfall, incremental, etc.) process (Waterfall, incremental, etc.) beforebefore they started to think about they started to think about using OTS.using OTS.

It actually worked.It actually worked.


RQ1: What should be added?RQ1: What should be added?

ActivitiesActivities• ””Acquire” vs. ”build” decisionAcquire” vs. ”build” decision• OTS component selectionOTS component selection• Learning OTS componentLearning OTS component• Build glueware and/or addwareBuild glueware and/or addware

A A new rolenew role (OTS knowledge keeper) (OTS knowledge keeper)• Germany (100%)Germany (100%)• Norway (37%)Norway (37%)• Italy (9%)Italy (9%)


RQ1: What is the proper OTS RQ1: What is the proper OTS selection process?selection process?

Formal decision making process (Formal decision making process (by 15% usedby 15% used))• Selecting evaluation criteria (factors)Selecting evaluation criteria (factors)• Collecting and assigning values to these criteriaCollecting and assigning values to these criteria• Applying formal decision making algorithms such as Applying formal decision making algorithms such as

MAUT or MCDA etc.MAUT or MCDA etc. Familiar with component process (Familiar with component process (by 85% usedby 85% used))

• Search internetSearch internet• Limited to 2-3 componentsLimited to 2-3 components• Download demo version and try it, then decideDownload demo version and try it, then decideOr Or • Recommended from internal/external expertsRecommended from internal/external experts



How to predict possible risks in How to predict possible risks in projects using OTS components?projects using OTS components?• What were the What were the most frequentmost frequent risks risks

(problems) in practice?(problems) in practice?• Was there any relationship between Was there any relationship between

those those risks (problems) and the project risks (problems) and the project profileprofile??


RQ2: Typical risksRQ2: Typical risksPhase Risks

Project plan The project was delivered long after schedule

Effort to select OTS components was not satisfactorily estimated

Effort to integrate OTS components was not satisfactorily estimated

Requirement Requirements were changed a lot

OTS components could not be sufficiently adapted to changing requirements

It is not possible to (re) negotiate requirements with the customer, if OTS components could not satisfy all requirements

Component integration

OTS components negatively affected system reliability

OTS components negatively affected system security

OTS components negatively affected system performance

OTS components were not satisfactorily compatible with the production environment when the system was deployed


RQ2: Typical risks (cont’)RQ2: Typical risks (cont’)Phase Risks

Maintenance and evolution

It was difficult to identify whether defects were inside or outside the OTS components

It was difficult to plan system maintenance, e.g. because different OTS components had asynchronous release cycles

It was difficult to update the system with the last OTS component version

Provider Relationship

Provider did not provide enough technical support/ training

Information on the reputation and technical support ability of provider were inadequate


RQ2: Frequency of typical risks (problems) RQ2: Frequency of typical risks (problems) in OTS based developmentin OTS based development

747583798084788585786779838285N =

6

5

4

3

2

1

0


RQ2: Frequency of typical risks in OTS RQ2: Frequency of typical risks in OTS based development (cont’)based development (cont’)

Most frequentMost frequent risks risks• Effort to integrate OTS components was not Effort to integrate OTS components was not

satisfactorily estimatedsatisfactorily estimated• Keep up with requirements evolutionKeep up with requirements evolution• Identify defects inside or outside OTS Identify defects inside or outside OTS

componentcomponent Least frequentLeast frequent risks risks

• Negative reliability effectNegative reliability effect• Negative security effectNegative security effect• Negative performance effectNegative performance effect• Lack provider informationLack provider information


RQ2: Relationship between typical risks RQ2: Relationship between typical risks (problems) and project context(problems) and project context

The more The more different OTS-componentsdifferent OTS-components used in the project, the more used in the project, the more frequent the following risks:frequent the following risks:• Identify whether defects were inside or Identify whether defects were inside or

outside the OTS components outside the OTS components • It was difficult to update the system It was difficult to update the system

with the last version OTS componentswith the last version OTS components• Provider did not provide enough Provider did not provide enough

technical support/training technical support/training


RQ2: Relationship between typical risks RQ2: Relationship between typical risks (problems) and project context (cont’)(problems) and project context (cont’)

The higher the The higher the general experiencegeneral experience on on OTS-based development in projects, OTS-based development in projects, the less frequent the following risks:the less frequent the following risks:• Effort to integrate OTS c components Effort to integrate OTS c components

was not satisfactorily estimated was not satisfactorily estimated • It was difficult to identify whether It was difficult to identify whether

defects were inside or outside the OTS defects were inside or outside the OTS componentscomponents


RQ2: Relationship between typical risks RQ2: Relationship between typical risks (problems) and project context (cont’)(problems) and project context (cont’)

The project with an The project with an OTS knowledge OTS knowledge keeperkeeper had less frequency on the had less frequency on the following risks than project without following risks than project without OTS knowledge keeper:OTS knowledge keeper:• Difficult ot identify risks inside or outside Difficult ot identify risks inside or outside

OTS componentsOTS components• Lack the information of the vendors’ Lack the information of the vendors’

reputation and support abilityreputation and support ability



What are the effective methods to What are the effective methods to mitigate risksmitigate risks in projects using OTS in projects using OTS components?components?• Which strategies had been frequently Which strategies had been frequently

used in practice?used in practice?• What were the effective strategies?What were the effective strategies?


RQ3: Proposed risk management RQ3: Proposed risk management strategiesstrategies

Customer had been actively involved in “acquire” vs. “build” decision

Customer had been actively involved in OTS component selection

OTS components were selected mainly based on architecture and standards compliance, instead of expected functionality

OTS components qualities (reliability, security etc.) were seriously considered during selection

Effort in learning OTS component was seriously considered in effort estimation


RQ3: Proposed risk management strategies RQ3: Proposed risk management strategies (cont’)(cont’)

Effort in black-box testing of OTS components was seriously considered in effort estimation

Unfamiliar OTS components were integrated first Did integration testing incrementally (after each

OTS component was integrated) Local OTS-experts actively followed updates of

OTS components and possible consequences Maintained a continual watch on the market and

looked for possible substitute components Maintained a continual watch on provider support

ability and reputation


RQ3: Frequency of using proposed risk RQ3: Frequency of using proposed risk management strategies in practicemanagement strategies in practice

8085848276818383848485N =

6

5

4

3

2

1

0


RQ3: Frequency of using proposed risk RQ3: Frequency of using proposed risk management strategies in practice (cont’)management strategies in practice (cont’)

The The most frequentlymost frequently used risk management used risk management strategies:strategies:• OTS components qualities were seriously OTS components qualities were seriously

considered in the selection processconsidered in the selection process• Unfamiliar OTS components were integrated firstUnfamiliar OTS components were integrated first• Did integration testing incrementallyDid integration testing incrementally• Local OTS-experts actively followed updates of OTS

components and possible consequences The The least frequentlyleast frequently used risk management strategies: used risk management strategies:

• Involve customers in the Involve customers in the “acquire” vs. “build” decision

• Invove customers in OTS selectionInvove customers in OTS selection


RQ3: What were effective risk management RQ3: What were effective risk management strategies ?strategies ?

RisksRisks Effective risk management methodEffective risk management method

Estimate selection Estimate selection efforteffort

OTS components OTS components qualities qualities (reliability, security etc.) (reliability, security etc.) were seriously consideredwere seriously considered inin the selection process the selection process

Estimate integration Estimate integration efforteffort

OTS components OTS components qualities qualities (reliability, security etc.) (reliability, security etc.) were seriously consideredwere seriously considered inin the selection process the selection process

Follow requirement Follow requirement changeschanges

Maintained a continualMaintained a continual watch watch on the on the marketmarket and and looked for possible substitute componentslooked for possible substitute components

Plan maintenance Plan maintenance OTS components OTS components qualities qualities (reliability, security etc.) (reliability, security etc.) were seriously consideredwere seriously considered inin the selection process the selection process

Lack provider supportLack provider support Maintained a continual Maintained a continual watch watch on on provider provider support support ability and reputationability and reputation


RQ3: Risk management RQ3: Risk management recommendations in OTS-based recommendations in OTS-based

projectsprojects Avoid riskAvoid risk

• Do not use too many different OTS components Do not use too many different OTS components in one projectin one project

Manage riskManage risk• Manage the knowledge of OTS properly Manage the knowledge of OTS properly (Have (Have

a OTS expert and share OTS experience a OTS expert and share OTS experience regularly)regularly)

• Spend enough time on OTS quality evaluation. Spend enough time on OTS quality evaluation. Hand-on trial is necessaryHand-on trial is necessary

• Do not marry specific OTS. Be ready for Do not marry specific OTS. Be ready for possible replacementpossible replacement

• Maintain a continual watch on provider support Maintain a continual watch on provider support ability and reputationability and reputation



What are the similarities and What are the similarities and differences between differences between projects using projects using COTS and OSS components?COTS and OSS components?• Are there any similarities and Are there any similarities and

differences in:differences in: Company, project, system profile ?Company, project, system profile ? Motivation of using them ?Motivation of using them ? Frequency of risks (problems) ?Frequency of risks (problems) ?


RQ4: Selected samples – COTS RQ4: Selected samples – COTS projects vs. OSS projectsprojects vs. OSS projects

56 projects used only COTS56 projects used only COTS 25 projects used only OSS25 projects used only OSS 5 projects used both COTS and OSS 5 projects used both COTS and OSS

(not considered in data analysis)(not considered in data analysis)


RQ4: Are there any similarities and RQ4: Are there any similarities and differences in company profile ?differences in company profile ?

Company size

0

0,05

0,1

0,15

0,2

0,25

0,3

0,35

0,4

0,45

0,5

Small Medium Large

OSS

COTS


RQ4: Are there any similarities and RQ4: Are there any similarities and differences in company profile ? differences in company profile ?

(cont’)(cont’)Company's main business

0

0,1

0,2

0,3

0,4

0,5

0,6

Softwarehouse

IT consulting IT departmentof a traditional

industry

Telecom.Industry

OSS

COTS


RQ4: Are there any similarities and RQ4: Are there any similarities and differences in project profile ?differences in project profile ?

System application domain

0

0,05

0,1

0,15

0,2

0,25

0,3

0,35

Traditional industry Bank Other private services Public sector ICT sector

OSS

COTS


5254555654555554N =

COTS projects

6

5

4

3

2

1

02324252524252525N =

OSS projects

6

5

4

3

2

1

0

RQ4: Are there any similarities and RQ4: Are there any similarities and differences in system profile ?differences in system profile ?


Our conclusionOur conclusion• There is no difference in company, There is no difference in company,

project and system profile between project and system profile between projects using COTS and OSS. projects using COTS and OSS.

RQ4: Are there any similarities and RQ4: Are there any similarities and differences in company, project, differences in company, project,

and system profile ?and system profile ?


RQ4: Are there any similarities and RQ4: Are there any similarities and differences indifferences in

motivation of using COTS vs. OSS ?motivation of using COTS vs. OSS ? CommonalitiesCommonalities

• Shorter time-to-marketShorter time-to-market• Less development and maintenance effortLess development and maintenance effort• Higher reliabilityHigher reliability

DifferencesDifferences• COTSCOTS

Follow the market trendFollow the market trend Paid software will give good reliabilityPaid software will give good reliability Good supportGood support

• OSSOSS New technologyNew technology Free source code Free source code Avoid the risk in OSS evolutionAvoid the risk in OSS evolution


RQ4: Are there any similarities and RQ4: Are there any similarities and differences in frequency of risks differences in frequency of risks

(problems) ?(problems) ? CommonalitiesCommonalities

• Requirement changed a lot and it was Requirement changed a lot and it was difficult to keep up with these changesdifficult to keep up with these changes

DifferencesDifferences• COTS: higher risk on following evolution COTS: higher risk on following evolution

of both requirements and COTS of both requirements and COTS componentcomponent

• OSS: higher risk on getting good supportOSS: higher risk on getting good support


Questions ?Questions ?

Risk management and Process Improvement of Off-The-Shelf Based Development

Documents

Transcript of Risk management and Process Improvement of Off-The-Shelf Based Development