University of Groningen

Large scale continuous integration and deliveryStahl, Daniel

IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.

Document VersionPublisher's PDF, also known as Version of record

Publication date:2017

Link to publication in University of Groningen/UMCG research database

Citation for published version (APA):Stahl, D. (2017). Large scale continuous integration and delivery: Making great software better and faster[Groningen]: University of Groningen

CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).

Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.

Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.

Download date: 11-02-2018

Large Scale Continuous Integrationand Delivery

Making Great Software Better and Faster

PhD thesis

to obtain the degree of PhD at theUniversity of Groningenon the authority of the

Rector Magnificus Prof. E. Sterkenand in accordance with

the decision by the College of Deans.

This thesis will be defended in public on

Friday 10 March 2017 at 14.30 hours

by

Daniel Bertil Ståhl

born on 7 March 1982in N Solberga, Sweden

SupervisorProf. J. Bosch

Assessment CommitteeProf. P. AvgeriouProf. I. CrnkovicProf. P. Lago

Large ScaleContinuous Integration and Delivery

Making Great Software Better and Faster

Daniel Ståhl

Abstract

Since the inception of continuous integration, and later continuous delivery, the methods ofproducing software in the industry have changed dramatically over the last two decades.Automated, rapid and frequent compilation, integration, testing, analysis, packaging and delivery ofnew software versions have become commonplace. This change has had significant impact not onlyon software engineering practice, but on the way we as consumers and indeed as a society relate tosoftware. We have grown used to, and indeed come to expect, continuous deployment of newsoftware to everything from computers and mobile devices to vehicles and infrastructure. Becauseof this, continuous integration and delivery systems play a pivotal role in modern society: as we livein an increasingly software-intensive and software-dependent world, the quality and reliability ofthe systems we use to build, test and deliver that software is a crucial concern. At the same time, itis repeatedly shown that the successful and effective implementation of continuous engineeringpractices is far from trivial, particularly in a large scale context.

This thesis approaches the software engineering practices of continuous integration and deliveryfrom multiple points of view, and is split into three parts, accordingly.

Part I focuses on understanding the nature of continuous integration and differences in itsinterpretation and implementation. It is found that experiences of continuous integration varygreatly from case to case, and also that there is a significant degree of divergence inimplementations of the practice. Furthermore, several types of impediments to successful adoptionof continuous practices are discussed, including challenges related to software-intensive embeddedsystems and large scale development efforts.

In order to address this divergence and provide practitioners and researchers alike with better andless ambiguous methods for describing and designing continuous integration and delivery systems,Part II applies the paradigm of system modeling to continuous integration and delivery. Throughmultiple iterations of design, industry application and evaluation, these modeling techniques arerefined and ultimately evolved into the Cinders architecture framework.

Meanwhile, Part III addresses the problem of traceability. It is established that traceability is farfrom a unique concern to continuous integration and delivery, but due to their emphasis on highspeed and frequency these practices pose unique challenges which call for novel solutions andimproved infrastructure. One such solution – the open source framework Eiffel – is evaluated, andits effects on traceability practice as well as on the mindset of engineers are considered. Finally, theopportunities for further improvements to software engineering efficacy presented by thetraceability capabilities of such a framework are discussed.

The key contributions of this thesis are:• Overview and classification of the many points of divergence in continuous integration

practice.• Analysis and discussion of impediments to effective implementation of continuous

integration, particularly in a large scale context.• Cinders, an architectural framework for the design and description of continuous integration

and delivery systems, is presented.• Eiffel, an industry developed open source framework for continuous integration and

delivery, placing particular emphasis on scalability, technology agnosticism and traceability,is presented and evaluated.

i

Acknowledgments

Doing a PhD is an amazing journey. Not only have I met people I would not otherwise have metand visited places I would not otherwise have visited – and had food poisoning in the process – butmy mind has explored ideas and concepts I would not otherwise have imagined. It is, however, alsoa journey one does not enter upon – and much less conclude – alone. This thesis had not come to behad it not been for the support, encouragement and assistance of many; too many to name here.

That being said, first I would like to thank my promoter and mentor, Jan Bosch. With hisstraightforward honesty, unwavering support and unrivaled knack for pushing people to be all thatthey can be I owe him much – more than just this thesis. His pithy questions and constructivecriticism has not only led me to rewrite my papers more times than I care to count, but helped me tosharpen my arguments and thereby my mind. Thank you, Jan.

I owe great thanks to the many colleagues and managers at Ericsson who supported my ambitionand my work. I have been privileged to pursue my research in parallel with my work at Ericsson –designing, developing, architecting and maintaining the very type of continuous integration anddelivery systems I have been researching. It is a sign of a healthy company culture and exemplaryleadership to give one's engineers the leeway to take a trip into the unknown, to improvethemselves, and not least to challenge their own organizations.

I also wish to thank the talented individuals I am honored to call co-authors; in no particularorder, Kristofer Hallén, Kristian Sandahl, Tor Skude, Lina Lagerberg, Pär Emanuelsson and TorvaldMårtensson.

Warm thanks to the skilled researchers and engineers who make up the Software Center. Inspiringconversations, cases to study, colleagues to bounce ideas off of and critics to help improve my workhave never been more than a mail or a phone call away. In particular, I wish to thank the manyengineers at the Software Center companies of Ericsson, Saab, Volvo Cars, Volvo Trucks, Jeppesen,Axis Communications and Sony for their openness, insights and patience with all my questions.

Last but certainly not least, I wish to thank my family. You are the reason it is all worth it; any ofit. It is through your eyes that I look upon myself and my work, it is through you that I find myplace in the world. I love you. A special thank you to my parents: you were always my greatest fans;your unshakable belief in my ability has instilled that same belief in me. I firmly believe that to beone of the greatest gifts one can give to a child, and one I can only hope that I myself will be able topass on.

ii

Contents

Abstract................................................................................................................................................i

Acknowledgments...............................................................................................................................ii

List of Figures....................................................................................................................................vi

List of Tables.......................................................................................................................................x

Chapter 1. Introduction.....................................................................................................................11.1 The Rise and Prominence of Continuous Integration and Delivery..........................................11.2 Problem Statement.....................................................................................................................31.3 Research Objectives...................................................................................................................41.4 Research Methodology..............................................................................................................51.5 Research Questions....................................................................................................................81.6 Thesis Outline and Article Overview.......................................................................................101.7 Applicability of the Research Results......................................................................................111.8 Detailed Overview of Research Processes and Methods.........................................................121.9 Related Publications................................................................................................................171.10 Contributions to the Articles..................................................................................................17

Part I: Continuous Integration and Delivery Practice Impediments and Divergence...............20

Chapter 2. Experienced Benefits of Continuous Integration in Industry Software ProductDevelopment......................................................................................................................................21

2.1 Introduction..............................................................................................................................212.2 Research Method.....................................................................................................................222.3 Hypotheses...............................................................................................................................232.4 Data Collection........................................................................................................................242.5 Hypotheses and Data Examination..........................................................................................272.6 Conclusion...............................................................................................................................33

Chapter 3. The Impact of Agile Principles and Practices on Large-Scale SoftwareDevelopment Projects.......................................................................................................................35

3.1 Introduction..............................................................................................................................353.2 Agile Principles and Practices..................................................................................................363.3 Research Design......................................................................................................................373.4 Results......................................................................................................................................423.5 Related Work............................................................................................................................473.6 Discussion................................................................................................................................483.7 Conclusions..............................................................................................................................50

Chapter 4. Continuous Integration Applied to Software-Intensive Embedded Systems –Problems and Experiences...............................................................................................................51

4.1 Introduction..............................................................................................................................514.2 Case Study Companies............................................................................................................524.3 Problems and Experiences.......................................................................................................534.4 Analysis....................................................................................................................................564.5 Conclusion...............................................................................................................................58

iii

Chapter 5. The Continuity of Continuous Integration: Correlations and Consequences.........605.1 Introduction..............................................................................................................................605.2 Related Work............................................................................................................................625.3 Research Method.....................................................................................................................635.4 Metrics Identification...............................................................................................................645.5 Results......................................................................................................................................685.6 Analysis....................................................................................................................................705.7 Validation.................................................................................................................................775.8 Threats to Validity....................................................................................................................825.9 Conclusion...............................................................................................................................83

Chapter 6. Modeling Continuous Integration Practice Differences in Industry SoftwareDevelopment......................................................................................................................................87

6.1 Introduction..............................................................................................................................876.2 Research Method.....................................................................................................................886.3 Statement Clusters...................................................................................................................916.4 A Descriptive Model..............................................................................................................1006.5 Illustrative Case Study...........................................................................................................1056.6 Conclusion.............................................................................................................................107

Part II: Continuous Integration and Delivery Modeling and Architecture...............................111

Chapter 7. Automated Software Integration Flows in Industry: A Multiple-Case Study........1127.1 Introduction............................................................................................................................1127.2 Background............................................................................................................................1137.3 Research Method....................................................................................................................1147.4 Case Study..............................................................................................................................1167.5 Guidelines..............................................................................................................................1307.6 Conclusions............................................................................................................................132

Chapter 8. Industry Application of Continuous Integration Modeling: A Multiple-Case Study..........................................................................................................................................................134

8.1 Introduction............................................................................................................................1348.2 Background............................................................................................................................1358.3 Research Method...................................................................................................................1408.4 Results....................................................................................................................................1418.5 Related Work..........................................................................................................................1488.6 Threats to Validity..................................................................................................................1498.7 Conclusion.............................................................................................................................150

Chapter 9. Cinders: The Continuous Integration and Delivery Architecture Framework.....1519.1 Introduction............................................................................................................................1519.2 Problem Statement.................................................................................................................1529.3 Background............................................................................................................................1549.4 Research Method...................................................................................................................1589.5 Results....................................................................................................................................1619.6 Evaluation..............................................................................................................................1759.7 Threats to Validity..................................................................................................................1799.8 Conclusion.............................................................................................................................181

Part III: Continuous Integration and Delivery Traceability......................................................183

Chapter 10. Achieving Traceability in Large Scale Continuous Integration and Delivery.....184

iv

10.1 Introduction..........................................................................................................................18410.2 Research Method.................................................................................................................18610.3 Eiffel....................................................................................................................................18810.4 Identification of Traceability Needs....................................................................................19310.5 Validation.............................................................................................................................20110.6 Threats to Validity................................................................................................................20810.7 Conclusions and Further Work............................................................................................209

Chapter 11. Dynamic Test Case Selection in Continuous Integration: Test Result AnalysisUsing the Eiffel Framework...........................................................................................................212

11.1 Introduction..........................................................................................................................21211.2 The Eiffel Framework..........................................................................................................21311.3 Test Case Selection Strategies..............................................................................................21511.4 Automated vs. Manual Tests................................................................................................21511.5 Test Case Selection Based on Eiffel Data............................................................................21611.6 Test Case Atomicity.............................................................................................................21711.7 Conclusion...........................................................................................................................218

Chapter 12. Conclusion..................................................................................................................21912.1 Research Questions..............................................................................................................21912.2 Threats to Validity................................................................................................................22412.3 Generalization......................................................................................................................22812.4 Key Contributions................................................................................................................22912.5 Further Research..................................................................................................................230

Bibliography....................................................................................................................................233

Appendix A......................................................................................................................................246

Appendix B......................................................................................................................................256

Curriculum Vitae............................................................................................................................257

v

List of Figures

Figure 1: Problem domain breakdown.................................................................................................4Figure 2: Iterative research process......................................................................................................7Figure 3: The four quadrants of software development projects represented in the case study.........24Figure 4: Average scores assigned by interviewees in response to questions 1 and 2........................28Figure 5: Average scores assigned by interviewees in response to questions 3 and 4........................29Figure 6: Average scores assigned by interviewees in response to questions 5, 6, 7 and 8................30Figure 7: Average scores assigned by interviewees in response to questions 9, 10 and 11................32Figure 8: Relations between corner stones and impediments.............................................................59Figure 9: Module dependencies and direct change impact.................................................................61Figure 10: Plot of number of committers versus average commit size..............................................71Figure 11: Plot of number of organizational size versus average commit size..................................72Figure 12: Split of cases according to way of working......................................................................73Figure 13: Plot of number of organizational size versus average commit size, excluding commits byexternal consultants............................................................................................................................74Figure 14: Plot of number of organizational size versus number of builds per commit....................75Figure 15: Plot of total headcount over number of developers versus number of commits...............76Figure 16: Candlestick chart of interviewee responses to questions IQ3-8.......................................79Figure 17: Overview of the organizational size of the projects of the interviewees..........................80Figure 18: The meta-model of the descriptive model.......................................................................101Figure 19: A description model........................................................................................................105Figure 20: The Integration Flow Anatomy of Project A...................................................................117Figure 22: A compressed model of the integration flow of case A...................................................118

vi

vii

viii

Figure 46: A scatter plot of the functional and non-functional themes representing needs..............196Figure 47: The thematic network resulting from the semi-structured interviews in case A.............200Figure 48: A simple example of an Eiffel event graph.....................................................................214Figure 49: Eiffel events required for selecting tests that tend to fail when certain parts of the sourcecode are modified.............................................................................................................................217Figure 50: Mapping of research questions to problem domain breakdown.....................................224

ix

List of Tables

Table 1: Thesis research questions.......................................................................................................9Table 2: Focus and context of research studies..................................................................................12Table 3: Characteristics of Project A and Project B............................................................................38Table 4: Adoption of agile practices in Project A and Project B........................................................39Table 5: The most strongly contributing studies included in the theoretical framework...................41Table 6: Survey results on the amount of internal software documentation.......................................43Table 7: Survey results on knowledge sharing...................................................................................44Table 8: Survey results on project visibility.......................................................................................45Table 9: Survey results on pressure and stress...................................................................................45Table 10: Survey results on coordination...........................................................................................46Table 11: Our findings in relation to related work (Part I).................................................................47Table 12: Our findings in relation to related work (Part II)................................................................48Table 13: Duvall's seven corner stones of continuous integration......................................................52Table 14: The seven topics discussed in Section 4.3..........................................................................53Table 15: Overview of Pearson's test of linear correlation results for size and continuity metrics....70Table 16: Validation interview questions............................................................................................78Table 17: Inclusion and exclusion criteria of the literature review.....................................................89Table 18: Clusters of descriptive statements extracted from literature...............................................90Table 19: Proposed guidelines..........................................................................................................133Table 20: Interview questions...........................................................................................................141Table 21: Inclusion and exclusion criteria of the literature review...................................................148Table 22: Inclusion and exclusion criteria of the literature review...................................................159Table 23: Summary of viewpoints and information layers of the architecture framework..............172Table 24: Interview questions used during the semi-structured interviews......................................174Table 25: Interview questions used during the workshop and the specialists interview..................177Table 26: Single interview question posed to interviewees in case A..............................................195Table 27: Overview of coded interviewee statements......................................................................197Table 28: Interview questions posed to interviewees in cases B and C............................................199Table 29: Interview questions posed to engineers in case A............................................................201Table 30: Interview questions posed to engineers in cases B and C................................................203Table 31: A comparison of the data collection procedures in 2011 and 2015..................................204Table 32: Inclusion and exclusion criteria of the systematic literature review.................................206

x

xi