PRECEDENT FOR DOING THINGS DIFFERENTLYCase Studies of Agile Software Development

in Large, Distributed Outsourcing ProjectsBy Egbert R. G. van der VeenA Project Submitted to the Graduate Faculty of Management and Governance, in partial fulfillment of the requirements for the degree of MASTER OF SCIENCEMajor subject: BUSINESS INFORMATION TECHNOLOGY

University of TwenteEnschede, the Netherlands

For Graduation December 2011

Scientific supervisors:dr. Chintan Amritdr. Maya Daneva

dr. Klaas Sikkeldr. Smita Ghaisas

Executive Summary

Agile software development methodologies have proven their value in practice, but little is

known of their application, or indeed applicability, in large or distributed projects. Yet,

organizations delivering such projects are interested in adopting Agile approaches and want

to know what Agile practices are applicable to their project contexts. To investigate the

phenomenon of Agile in the large, we carried out an embedded case study of requirements

prioritization in projects that were both large and had distributed components. Specifically,

our research project focused on Agile requirements engineering and its key component;

requirements prioritization. Our overall goal was to gain a deeper understanding of how

large and distributed teams handle requirements in Agile projects. The research was carried

out within a large Asian provider of outsourcing solutions. We carried out a total of 16

in-depth, semi-structured interviews with practitioners in various roles in three different

projects. Grounded Theory was used to analyze the resulting data and construct our theory.

Although Agile software development methodologies are not typically used in large

or distributed projects, we found that our case study projects succeeded in implementing

Agile practices to their advantage. While doing so, the projects also realized the benefits

associated with the organizational maturity of the company. We found that standard

Agile practices were adapted in various ways to leverage their value in this setting. For

example, to cope with project size, there was a clear division of roles and responsibilities

within the projects. While this might appear contrary to the Agile principle of direct

lines of communication, this division allowed large, distributed projects to cope with

requirements in an effective way in a situation where it is not feasible for everyone

to directly communicate with everyone else. In this division of responsibilities, we

identified three levels of project activities; (i) ‘strategic’, where the project as a whole is

managed, high-level sequence of requirements is defined, and requirements are allocated

to requirements areas, (ii) ‘tactical’, where requirements are translated into low-level

specifications and sequences of iterations are planned, and (iii) ‘operational’, at the level of

individual iterations, where the actual development takes place. At these three levels, we

identified distinct prioritization events, dependency considerations, levels of abstraction

ii

with respect to requirements, and levels of client involvement.

Domain knowledge supported project activities on all three levels. Due to the

relative lack of documentation in Agile projects, and the added pressure of working in

short iterations, it was essential that project members possessed the information they

required. As such, knowledge management played an important role in our case study

projects. Knowledge was gathered via the vendor’s own domain competency group, and

by interviewing client representatives. Furthermore, it was disseminated through training

seminars, certification programs and presentations given by client representatives, as well

as through a comprehensive induction process for new project members.

In practice, Agile methodologies are rarely implemented ‘as is’. Rather, practitioners

cherry-pick those practices that fit their particular context. We found this was also true

for our case study projects. Through examining the three different projects, we attempted

to link the findings to each project’s unique circumstances. The first project had a tightly

controlled change management process in place, and did not easily accept changing

requirements. This can be linked to the fact that the project had a fixed-price, fixed-term

contract with its client. In the second project, requirements were subject to change at

any time, and as such, were not extensively defined up-front. Furthermore, this project

had the greatest degree of distribution, which led to the most elaborate organization of

collaboration. The third project, being smaller than the other two, had a less elaborate

project organization in place. Still, this organization was more formal and well-defined

than typical for Agile projects. As such, it appears as the ‘missing link’ between small

and very large projects, which allowed us to trace the evolution of project formalization as

projects scale up.

Using the methods we describe here, our case study projects effectively coped with

size while integrating Agile practices to their advantage. At the same time, Agile methods

actually had a positive impact on development in a distributed setting, due to greater

transparency, close interaction and lower barriers to communication. Our observations are

captured in a descriptive model that structures and summarizes our findings with respect to

the way in which our case study projects coped with requirements.

The methods, tools, and approaches that we describe in this work can serve as a

precedent for practitioners with a desire to do things differently, in other large organizations.

iii

Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

List of tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

List of figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Tata Research Development & Design Centre . . . . . . . . . . . . . . . 2

1.2 Principles of Agile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Traditional, Agile, and requirements . . . . . . . . . . . . . . . . . . . . 3

1.4 Areas of interest in this research project . . . . . . . . . . . . . . . . . . 5

1.5 Theoretical Significance . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.6 Practical Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2. Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1 Project Alpha . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2 Project Beta . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3 Project Gamma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3. Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1 Survey Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Inclusion and exclusion criteria . . . . . . . . . . . . . . . . . . . . . . . 12

3.3 Survey Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.3.1 Distributed Agile . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3.2 Large-scale Agile . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.3.2.1 Scaling of Agile practices . . . . . . . . . . . . . . . . 173.3.2.2 Organization of teams . . . . . . . . . . . . . . . . . . 18

3.3.3 Agile requirements prioritization . . . . . . . . . . . . . . . . . . 193.3.4 Comparison of other large Agile case studies . . . . . . . . . . . 19

3.4 The use of literature in this project . . . . . . . . . . . . . . . . . . . . . 20

iv

4. Research Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.1 Research methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.2 Questionnaire design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.3 Conducting the interviews . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.4 Data capturing and analysis process . . . . . . . . . . . . . . . . . . . . 25

4.5 Memo structuring and core concept . . . . . . . . . . . . . . . . . . . . 31

5. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.1 Is Agile worth the risk? . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.1.1 Agile adoption . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.1.2 Agile and the developer . . . . . . . . . . . . . . . . . . . . . . 395.1.3 Agile and the client . . . . . . . . . . . . . . . . . . . . . . . . . 405.1.4 Agile and collaboration . . . . . . . . . . . . . . . . . . . . . . . 41

5.2 Development process in general . . . . . . . . . . . . . . . . . . . . . . 435.2.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.2.1.1 Project Alpha . . . . . . . . . . . . . . . . . . . . . . 435.2.1.2 Project Beta . . . . . . . . . . . . . . . . . . . . . . . 445.2.1.3 Project Gamma . . . . . . . . . . . . . . . . . . . . . 45

5.2.2 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.3 Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485.3.1 Facilitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.3.2 Coordination . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.4 Knowledge management . . . . . . . . . . . . . . . . . . . . . . . . . . 525.4.1 Domain knowledge acquisition . . . . . . . . . . . . . . . . . . . 525.4.2 Client involvement . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.5 Elicitation of requirements . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.6 Delivery Stories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.7 Requirements prioritization . . . . . . . . . . . . . . . . . . . . . . . . . 575.7.1 Prioritization process . . . . . . . . . . . . . . . . . . . . . . . . 585.7.2 Factors triggering (re-)prioritization . . . . . . . . . . . . . . . . 625.7.3 Prioritization criteria . . . . . . . . . . . . . . . . . . . . . . . . 645.7.4 Classification of requirements . . . . . . . . . . . . . . . . . . . 645.7.5 Down-stream impact of reprioritization . . . . . . . . . . . . . . 66

5.8 Business value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

v

5.9 Dependency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.10 Project segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

6. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

6.1 Formalization of RE process . . . . . . . . . . . . . . . . . . . . . . . . 74

6.2 Organization of collaboration . . . . . . . . . . . . . . . . . . . . . . . . 79

6.3 Descriptive model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 806.3.1 Strategic level . . . . . . . . . . . . . . . . . . . . . . . . . . . . 836.3.2 Tactical level . . . . . . . . . . . . . . . . . . . . . . . . . . . . 856.3.3 Operational level . . . . . . . . . . . . . . . . . . . . . . . . . . 866.3.4 Domain knowledge support . . . . . . . . . . . . . . . . . . . . 876.3.5 The full picture . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

7. Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

7.1 Internal validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

7.2 External validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

7.3 Model validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

8. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

8.1 Application of Agile . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

8.2 Agile project management constructs . . . . . . . . . . . . . . . . . . . . 96

8.3 Project organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

8.4 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

9. Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

9.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

9.2 Future research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

9.3 Lessons learned for researchers . . . . . . . . . . . . . . . . . . . . . . . 102

9.4 Implications for practice . . . . . . . . . . . . . . . . . . . . . . . . . . 104

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

Appendices

A. Theoretical Model - Racheva & Daneva . . . . . . . . . . . . . . . . . . . . . 112

B. TRDDC Organization Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

vi

C. Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

C.2 Main characteristics and context of the project . . . . . . . . . . . . . . . 115

C.3 Section 1. Prioritization and re-prioritization process. (concrete case) . . . 117

C.4 Section 2: Prioritization process (general observations) . . . . . . . . . . 118

C.5 Section 3: Value creation . . . . . . . . . . . . . . . . . . . . . . . . . . 119

C.6 Section 4: Domain knowledge assistance . . . . . . . . . . . . . . . . . . 120

D. Visualization of interview transcripts . . . . . . . . . . . . . . . . . . . . . . . 122

E. Traceability between narative and data . . . . . . . . . . . . . . . . . . . . . . 123

F. Relational overview of codes . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

G. Code occurrences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

H. Code saturation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

I. Unique codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

J. Web-based data analysis tool . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

K. Memo clustering mindmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

L. The Thesis Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

vii

List of tables

2.1 Context of embedded cases . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2 Different types of outsourcing . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.1 Literature study concept matrix . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2 Comparison of large Agile case studies . . . . . . . . . . . . . . . . . . . . 20

4.1 Overview of interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5.1 Overall project organization . . . . . . . . . . . . . . . . . . . . . . . . . . 46

6.1 Specificity of findings (1/2) . . . . . . . . . . . . . . . . . . . . . . . . . . 73

6.2 Specificity of findings (2/2) . . . . . . . . . . . . . . . . . . . . . . . . . . 74

G.1 Code frequency per interview . . . . . . . . . . . . . . . . . . . . . . . . . 125

G.2 Code frequency averages per project . . . . . . . . . . . . . . . . . . . . . 125

H.1 Code saturation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

viii

List of figures

1.1 Changing perception of requirements . . . . . . . . . . . . . . . . . . . . . 4

1.2 Changing perception of requirements - Agile . . . . . . . . . . . . . . . . . 5

3.1 Preliminary literature survey . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 In-depth literature survey . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.3 Literature on requirements prioritization in general . . . . . . . . . . . . . . 14

4.1 Grounded Theory process . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2 Number of new codes for each interview . . . . . . . . . . . . . . . . . . . 28

4.3 Code frequency and code percentage graphs . . . . . . . . . . . . . . . . . 30

4.4 Percentage of code occurrence per project . . . . . . . . . . . . . . . . . . 31

4.5 Memo clustering mindmap - collapsed . . . . . . . . . . . . . . . . . . . . 32

4.6 Brainstorming about core concept, why-how tree . . . . . . . . . . . . . . . 34

4.7 Memo clustering mindmap after identification of core concept . . . . . . . . 35

4.8 The three lenses through which we interpet our observations . . . . . . . . . 35

5.1 Structure of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.2 Generality of findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.3 Project organization - Alpha . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.4 Project organization - Beta . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.5 Organization of tasks in case study projects . . . . . . . . . . . . . . . . . . 47

5.6 Delivery story process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.1 Prioritization considerations: Project Beta . . . . . . . . . . . . . . . . . . 78

6.2 Prioritization considerations - fixed price: Project Alpha . . . . . . . . . . . 78

6.3 Descriptive model of project activities: overview . . . . . . . . . . . . . . . 80

6.4 Legend - Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

6.5 Descriptive model of project activities: the requirements flow . . . . . . . . 82

ix

6.6 Separation of concerns as related to the project timeline . . . . . . . . . . . 83

6.7 Legend - Category colors . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

6.8 Descriptive model - strategic level . . . . . . . . . . . . . . . . . . . . . . . 84

6.9 Descriptive model - tactical level . . . . . . . . . . . . . . . . . . . . . . . 85

6.10 Descriptive model - operational level . . . . . . . . . . . . . . . . . . . . . 86

6.11 Descriptive model - Domain knowledge support . . . . . . . . . . . . . . . 88

6.12 Conceptual model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

A.1 Theoretical model by Racheva and Daneva . . . . . . . . . . . . . . . . . . 112

B.1 TRDDC organization chart . . . . . . . . . . . . . . . . . . . . . . . . . . 113

D.1 Word Cloud visualization of interview transcripts . . . . . . . . . . . . . . 122

E.1 Traceability between source data and the narative . . . . . . . . . . . . . . 123

F.1 Codes and nesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

J.1 Data analysis tool example - dependency handling . . . . . . . . . . . . . . 130

J.2 Data analysis tool example - reading transcripts . . . . . . . . . . . . . . . 131

J.3 Example of comment adding and editing functionality . . . . . . . . . . . . 132

K.1 Memo clustering mindmap . . . . . . . . . . . . . . . . . . . . . . . . . . 133

K.2 Memo clustering mindmap - collapsed . . . . . . . . . . . . . . . . . . . . 134

K.3 Memo clustering mindmap - core concept . . . . . . . . . . . . . . . . . . 134

L.1 The Thesis Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

x

Acknowledgement

It will be impossible to adequately express my gratitude to the people who made this

research project possible, but I will do my best. First and foremost, I would like to thank

the Tata Research Development & Design Centre for offering me the opportunity of a

lifetime. Specifically, my thanks go out to Smita Ghaisas, my supervisor at TRDDC,

without whom none of this would have happened and whose constant help, guidance and

insight (not to mention patience) not only enabled my work, but made it a pleasure to do.

Also her team, consisting of Preethu Rose, Nirav Ajmeri, Manoj Bhat, Mini Jain, Kumar

Vidhani and Kush Shah, for being incredibly considerate and supportive, and showing

me more hospitality than I would have thought possible. I would also like to express my

gratitude towards my interview participants, who graciously donated their valuable time

and energy and unfailingly did their best to help.

Furthermore, I would like to thank Chintan Amrit and Maya Daneva, my supervisors

from the University of Twente, for their constant advice and encouragement, which

provided me with the means and motivation to carry out this work. I also thank Klaas

Sikkel, whose insights and contributions proved invaluable. Last but not least, I would like

to thank Zornitza Racheva for her work on requirements prioritization in Agile projects,

which formed the foundation for this project.

xi

1. Introduction

This Master’s project arose out of an emerging collaboration effort between the University

of Twente (UT) in the Netherlands and the Tata Research, Development & Design Centre

(TRDDC) in Pune, India. Specifically, between the UT’s Information Systems Group and

TRDDC’s Requirements Engineering team. Shared areas of interest gave rise to a desire for

more academic collaboration, part of which was to receive a Master’s student for a period

of five months for a research project. I was lucky and privileged enough to be that student,

and this thesis is the result of my work. Common areas of interest were found in the study

of requirements engineering in Agile software development projects, based mainly on

the work of Zornitza Racheva, in collaboration with Maya Daneva and others1. This area

of interest overlapped with TRDDC’s own interests and activities [9]. Specifically, all

parties were interested in further examining the conceptual model for client-driven Agile

requirements prioritization proposed in [10] (reproduced here in appendix A). However,

the case studies conducted by Racheva and Daneva were all of relatively small-scale Agile

projects that were all co-located. Collaboration with TRDDC offered us the opportunity to

examine large to very large projects that operated in a geographically distributed setting.

The goal was to gain new perspectives on the process of Agile requirements prioritization

by examining it in very large, and distributed, software development projects. Because of

the opportunities offered by our area of study, our original research interests were expanded

to include an investigation of how Agile projects cope with distribution. An element of

knowledge management was included to further reflect TRDDC’s interest in this area [11].

In the rest of this introduction, we will elaborate further on the context of our

research project, as well as take a step back and look at some of the history behind Agile

software development. Following this, we will describe our areas of interest and discuss

the theoretical and practical significance of our work. Our actual case study projects will

be discussed in the next chapter.1See [1, 2, 3, 4, 5, 6, 7, 8]

1

2

1.1 Tata Research Development & Design CentreFounded in 1981 as the first R&D centre in the Indian software Industry2, TRDDC

is a division of Tata Consultancy Services (TCS), and the largest of its ‘Innovation Labs’.

TRDDC is subdivided into the labs of Process Engineering, Software Engineering and

Systems Engineering. Its Requirements Engineering (RE) group, of which I was part,

belongs to the software engineering lab (basic organization chart provided in appendix

B). A main area of interest of this group is how domain knowledge can support the RE

process. A research and development effort is underway that aims to create a tool to

allow practitioners to retain domain knowledge gained in projects [9]. This knowledge can

then be re-used for future projects. This group in TRDDC frequently collaborates with

the case study organization to capture domain knowledge and organize it in an effective

way, adapting the system as necessary. For each business domain, the system starts with

a ‘seed’ requirements specification, from which a comprehensive picture of the domain

evolves over time. The eventual goal is to make the tool a general-purpose framework

for capturing and organizing domain knowledge. For example, once it is known that

financial applications in a certain country require certain features for legal compliance,

future projects can use this knowledge to accelerate their RE process. The ambition is

to grow this system into an exhaustive repository of domain knowledge and give it the

necessary features to optimally support project teams in knowledge-intensive activities.

Further insight into the requirements prioritization process, from a new perspective

and in a new setting, was a shared area of interest between the UT, TRRDC, and me. All

parties were interested in producing a conceptual model for thinking about prioritization

from this angle, as a supplement to existing, internal models, as well as the model proposed

by Racheva and Daneva.

1.2 Principles of AgileBecause Agile plays an important role in this thesis, we will briefly discuss its

chief characteristics. Agile software development / project management is very much a

reaction to traditional, or plan-based development methodologies. These are often seen as

cumbersome, with unnecessary bureaucratic overhead. In 2001, a group of 17 practitioners2TRDDC website; http://www.tcs-trddc.com/, accessed 25-10-2011

3

from the software engineering discipline composed the ‘Manifesto for Agile Software

Development’, often called simply ‘the Agile manifesto’. The core values described in this

manifesto, as originally published in [12], are as follows;

• Individuals and interactions over processes and tools

• Working software over comprehensive documentation

• Customer collaboration over contract negotiation

• Responding to change over following a plan

The authors make the point that, while there is value to the items on the right,

the emphasis should be on the items on the left. These points are further elaborated

upon in twelve principles3. Surveys in the literature [13] indicate that the most popular

Agile methodology in practice is Extreme Programming (XP) [14]. Another popular

methodology is Scrum [15]. It should be noted that Agile methodologies are themselves

agile, as pointed out by Hildenbrand et al. [13]. As we shall see in our study of the

literature, Agile methodologies are rarely adopted wholesale. Practitioners can pick and

choose the elements that make sense to them, and fit best in their unique context. Giving

a comprehensive overview of Agile is outside the scope of this introduction, but the

consequences for requirements engineering are discussed in the next section.

1.3 Traditional, Agile, and requirementsGenerally speaking, traditional software development methodologies operate accord-

ing to a strict segmentation of activities in the project timeline; all requirements are defined

up-front, development then occurs in relative isolation, and implementation of the product

as a whole is performed at the end. Experience from practitioners in the software industry

has shown that there can be issues with this type of development [16, 17]. For the scope

of our research project, we are mainly concerned with the way in which requirements are

handled. Because most of the development effort in traditional methodologies is performed

without interaction with the client, the perception of the requirements changes over time3The twelve principles are enumerated in full at http://agilemanifesto.org/principles.html, accessed

25-10-2011

4

on both sides. If, for the sake of argument, we assume that all requirements were captured

fully and understood 100% correctly at the start of the project, there can still be issues. In

these cases, the evolving understanding of both parties causes a drift in understanding from

what was originally specified. Changing circumstances on the client’s side also play a role

in this [18]. Evolving understanding of requirements on both the vendor’s and the client’s

side can lead to situations where software products that were developed over the course of

six months, a year, or even longer, completely fail to meet client’s expectations. This is in

spite of the fact that both parties agreed on the requirements at the outset of the project [13].

For example, Gat et al. [19] describe this situation as follows; “In their traditional 12-18

month development cycle defining requirements often occurred many months in advance

of the start of development, sometimes resulting in a two year lag from customer need to

delivered product. This delay often produced stale requirements that were out of sync with

the latest user needs and market opportunities” [19]. Obviously, matters are even worse

when parties differed in their perception of the requirements from the start. This process of

diverging understanding is schematically depicted in figure 1.1. Please note that the figure

servers illustrative purposes only and is not an empirical statement.

Figure 1.1: Changing perception of requirementsHow the understanding of the requirements by both the client and the developer changes

over time.

In contrast to ‘traditional’ (or waterfall-like) software development models, Agile

methodologies proceed in a much more iterative fashion. They are characterized by

5

frequent interactions with the client and very short release cycles of only weeks in duration.

After each cycle, working software is released to the client. This causes rapid feedback

and allows developers to adjust efforts regularly if needed [20, 21, 22]. It also triggers

the process of evolving understanding on both sides; through the discussion over features

and the exact shape they are to take, proponents of Agile development argue that a

better product emerges. Schematically, this can be depicted as shown in figure 1.2, to be

contrasted with figure 1.1.

Figure 1.2: Changing perception of requirements - AgileThrough frequent interaction with the client, developers can course-correct often, leading

to an end-product that more closely matches the client’s expectations.

As stated by Cao and Ramesh [18]: “Customers can steer the project in unanticipated

directions, especially when their requirements evolve owing to changes in the environment

or their own understanding of the software solution” [18]. Thus, the eventual end-product

is much closer to what the client requires, and delivering working software in increments

reduces risk for both parties. This process of adjustment and re-adjustment is known as

the concept or requirements (re-)prioritization. Thus, it plays an important role in Agile

development, and has been a specific avenue of inquiry for researchers [2].

1.4 Areas of interest in this research projectThe goal of this research is to develop an understanding of the Agile prioritization

process in large, distributed software development projects. Due to our choice of research

6

methodology, on which we shall elaborate in chapter 4, we did not have a precisely defined

problem statement up-front. We did, however, formulate our areas of interest in the form

questions to guide our literature study and help us construct the questionnaire.

Question 1: Who are the decision makers in the prioritization process? Which roles are

involved and what are they responsible for?

Question 2: What criteria do large project teams use to make risk-and-value driven

decisions during Agile prioritization in an outsourced mode?

Question 3: What is the relationship between project settings and requirements prioriti-

zation?

Question 4: How does the vendor’s team combine value creation for their own organi-

zation with value creation for their client?

1.5 Theoretical SignificanceAs we will show in chapter 3, our study is relatively unique in that it looks at both

large-scale and distributed Agile software development. Even if we take these elements

separately, we see relatively little empirical data published in scientific literature. There

are many ‘success-stories’ of small-scale Agile development, but little is known about

large-scale Agile. Also, using Agile development in a distributed setting is relatively

uncommon, and is thus an interesting avenue of in-depth inquiry. In this research project,

we captured a conceptual model for Agile prioritization, and thereby contribute to the call

for continued research on RE processes.

Furthermore, our study contributes to the body of knowledge about Agile software

development in the context of a highly mature, highly structured organization. While there

is anecdotal evidence suggesting that highly mature organizations do adopt Agile (for

example, [23, 24]), the author of this thesis could find no publication that describes how

Agile thinking and Agile principles are implemented in the day-to-day working practices

of project delivery teams. The author’s conclusion on this is based on a systematic search

in digital libraries (more on this can be found in chapter 3 - the literature study).

7

1.6 Practical SignificanceIn the software engineering community, research into new and improved software

development methodologies has been an ongoing effort of significant practical importance.

This is because of the still-disturbingly-high failure rates of projects [25, 26] that either

go over time or over budget, or fail to achieve their targets. Agile is a popular software

improvement approach and understanding how it works in new contexts, for example

outsourcing and distributed ones, is clearly of great value to practitioners in both software

development organizations and in Agile consulting service providing companies. Specif-

ically, our study offers findings for practitioners interested in RE methodologies, best

practices, and how to adapt Agile to their specific context. Our three embedded cases have

some commonalities but also their own unique context characteristics. Practitioners can

match these to their own and consider implementation of Agile methods accordingly.

2. Context

We carried out an embedded case study [27] where we interviewed practitioners across

three different projects within the same large organization. This organization is located

in Asia and is specialized in providing outsourcing solutions to its clients. It is a highly

professional, CMM level 5 organization [28] that has formalized its internal processes to a

large extent.

For confidentiality reasons, our three case study projects will be referred to as

projects Alpha, Beta, and Gamma. Aside from the general setting, each project has its

own individual context, which will be described in its section per project. Table 2.1 offers

an overview of the main characteristics of the projects. Dibbern et al. discuss different

outsourcing modalities [29]. If we apply their classification to our cases, they can be

categorized as shown in table 2.2.

Table 2.1: Context of embedded casesOutsourcing Project Project Alpha Project Beta Project Gamma

Type of engagement Single externalclient

Collaborative exter-nal client

Inter-departmentalproject

Scope Large Large MediumNumber of team mem-bers 124 35-40 + Client

Team of about 100 35-40

Cost and Duration Fixed Flexible FlexibleModularity of ProductArchitecture Low Low High

Table 2.2: Different types of outsourcingDegree ofoutsourcing Internal

OwnershipPartial External

Total Project Gamma Project AlphaSelective Project BetaNone

8

9

2.1 Project AlphaThe practitioners in project Alpha were part of a development team 124 people in

size, involved in the development of a large software solution in the application domain of

insurance process automation. The practitioners had experiences in several large projects

that had the objective to deliver a large enterprise system for a client in the insurance

business. The system was aimed at automating the core business processes of this client’s

organization.

The development project teams were co-located, and the client teams were dispersed

globally. The case study participants included: one Scrum master, one business analysts,

two business analyst leads, one delivery head (responsible for transforming the clients’ user

stories into ‘delivery stories’ that include architecture design decisions and non-functional

requirements), one portfolio manager (who was responsible for the group of client’s

projects managed as a portfolio), and one test scenario team lead (responsible for end-user

acceptance testing and making sure requirements are testable and verifiable).

2.2 Project BetaThe second project consisted of a team between 35 and 40 people in size from the

side of the vendor, complementing an existing development team on the client’s side,

leading to an overall program of 150 people. The engagement was not for a fixed duration

or for a specific project, but to enhance the client’s existing development efforts. The

engagement was started three years prior to the time of our interviews, with Agile having

been adopted as a development methodology for the last year. Various products were under

active development concurrently. These products were divided into ‘releases’, with each

release constituting a development project in itself, as driven by the client.

2.3 Project GammaIn project Gamma, the client was an internal, but not co-located business customer.

The objective of this third project was to create a model-driven development environment.

This was done in collaboration with an internal client, who used the product to develop

solutions for an external customer. For the purposes of this discussion, we will refer to the

former team as the ‘development team’ (or ‘vendor’), and the latter team as ‘the client’, as

10

there was little to no direct interaction between the first development team and the eventual

end-user of the products developed by the second team. The project was 26 people in size,

with 21 people on the developer’s side and 5 on the client’s side (again, we refer here to the

internal client). In contrast to the other cases, the development team was the actual owner

of the product, and used its interaction with the client as a way to expand the product with

the aim to eventually make it widely applicable in a broad range of settings. As the client

used the tool for their own development efforts, bugs or limitations were discovered, and

change requests were raised with the development team. The development team then tried

to accommodate these requests as quickly as possible so as not to slow down the client’s

development effort. In contrast to the other cases, the direction of the product was not only

determined by feedback from the client; requirements are also influenced by other factors.

3. Literature Review

Figure 3.1: Preliminary literature survey

Before the actual case studies were con-

ducted, a comprehensive survey of the

literature was undertaken. We conducted

a systematic review of the current body

of knowledge, using the guidelines put

forward by Webster and Watson [30]. Al-

though this was an exploratory case study,

the literature survey helped to construct

the questionnaire, as well as give the re-

searcher the background knowledge nec-

essary to guide the interviews and pursue

interesting facts.

After the primary data had been

gathered, additional literature was con-

sulted to which findings could be com-

pared and contrasted. This literature also

helped in the theory generation exercise.

This literature study approach is in ac-

cordance with recommendations put forward by Grounded Theory methodologists (and

elaborated upon in chapter 4). The process and results of our literature surveys will be

discussed in this chapter. Search engines, search terms and selection approach will be

discussed in the form of our inclusion and exclusion criteria. Following this, the findings

of some of these papers will be discussed in some detail, along with an evaluation of their

significance for this thesis.

3.1 Survey StructureAside from the primary literature that formed the basis for this research project

[1, 2, 3, 4, 5, 6, 7, 8, 10], a more general literature survey was conducted. Taking our

11

12

areas of interest defined in section 1.4 as a point of departure, we looked for literature

concerning Agile requirements engineering, large Agile projects, and Agile in a distributed

setting. Using the search engine Scopus4, the preliminary literature survey was conducted

in two stages, the first of which is depicted in figure 3.1.

These results gave us the necessary context to cast a broader net and acquire more

of the theoretical background behind the questions asked by Racheva et al. [5] in their

questionnaire. Our search terms for this were ‘business value’, ‘domain knowledge’,

‘requirements engineering tool support’, and ‘requirements engineering customer involve-

ment’. This second-stage literature survey is schematically represented in figure 3.2. The

inclusion / exclusion criteria mentioned in the figure are elaborated upon in section 3.2.

Figure 3.2: In-depth literature survey

3.2 Inclusion and exclusion criteriaWhen scanning papers for relevance, we used explicit criteria that are discussed in

this section. The inclusion criteria are;4http://info.scopus.com/

13

IC1: Recounts results of case studies of large Agile projects in practice.

IC2: Discusses hybridization of Agile practices to fit project’s context.

IC3: Explicitly describes RE in Agile projects.

IC4: Discusses the impact of distributedness on Agile projects.

IC5: Describes how Agile methodologies can be adapted to accommodate dis-

tributed teams.

IC6: Specifically mentions the nature of customer involvement in Agile projects.

Our exclusion criteria were;

EC1: Multiple papers discussing the same study. In these cases, the most relevant

or most cited paper is chosen.

EC2: The study only handles Agile methodologies in the context of small projects.

EC3: The paper is itself a survey of the literature. In this case, the relevant papers

themselves were consulted.

The number and quality of the results that met our criteria suggests that our scope is

neither too broad nor too narrow. These papers were examined in greater detail. Further

studies of the literature were conducted looking specifically for the work of Zornitza

and Daneva, and other authors active in our areas of interest, such as Ramesh and Cao

[18, 31, 32]. Still more literature was acquired through backward and forward searches;

references in our papers were checked for related work, and searches were undertaken

for literature citing our most important papers. Scientific advisors from both the UT and

TRDDC also supplied relevant reference material that we took up in our study. Together,

this overview of the body of knowledge resulted in the theoretical underpinnings of this

thesis.

14

3.3 Survey ResultsAs shown in figure 3.3, there appears to be a wealth of theory on the prioritization of

requirements in general. However, as was immediately evident from our literature survey,

there is a lack of literature looking specifically at prioritization of requirements in Agile

projects.

Figure 3.3: Literature on requirements prioritization in general

The results indicate that, although there seems to be an abundance of theory on

RE in Agile practice, relatively few papers discuss the prioritization process directly.

Furthermore, there is a decided lack of empirical data on the actual implementation of

Agile in practice, especially on the way prioritization is handled.

As far as could be determined, there are very view studies that investigate Agile

requirements prioritization in large-scale, distributed, outsourced development projects.

Still, individual aspects of this case study’s context can be compared to studies dealing

with three contexts specifically; (i) distributed Agile development, (ii) large-scale Agile,

and (iii) Agile requirements prioritization, which we will do in the following subsections.

Finally, we will look at a comparison between other case studies of large Agile projects

that were identified in the literature. A concept matrix of our context of interest and the

most relevant literature is shown in table 3.1. A comparison of other case studies of large

Agile projects specifically is given in table 3.2.

3.3.1 Distributed Agile

Effective communication is described as a crucial success factor for Agile software

engineering by several authors [18, 33, 34]. There is much evidence in the literature

that customer engagement specifically is of paramount importance [34, 35]. Ramesh et

al. describe it as the “most important aspect in Agile RE [Requirements Engineering]”

[31]. Grewal & Mauer cite the lack of direct customer interaction as a primary reason

15

Table 3.1: Literature study concept matrix

Dis

tribu

ted

Agi

le

Larg

e-sc

ale

Agi

le

Agi

leR

E

Agi

leSE

chal

leng

es

Agi

leSE

adva

ntag

es

Agi

lesu

cces

sfa

ctor

s

[13] Hildenbrand et al. x x x[18] Cao & Ramesh x x x x x[19] Gat x x x x[20] Auvinen et al. x x[21] Sulfaro & Marchesi x[22] Koehnemann & Coats x x x x[31] Ramesh et al. x x[33] Lucia & Qusef x x[34] Korkala & Abrahamsson x x[35] Korkala et al. x[36] Grewal & Mauer x x[37] Sutherland et al. x x[38] Elshamy & Elssamadisy x x[17] Valade x[16] Larman & Vodde x[39] Eckstein x x[40] Gary et al. x x[41] Hong et al. x[42] Baskerville et al. x

for discontinuing the distributed part of their development effort [36]. This creates a

potential problem for the implementation of Agile in a distributed setting, also with

respect to traceability [31]. Moreover, one of the core principles of Agile is that face-

to-face communication is favored over written specifications [12, 18]. Hildenbrand et al.

identify the heavy reliance on communication as a significant challenge for the adoption

of distributed Agile [13], additionally stating that “[a]n application of XP [Extreme

Programming] principles without any face-to-face meetings is not feasible”. With this

heavy reliance on direct communication, it is perhaps not surprising that most cases in the

literature describe projects where the entire Agile development team is co-located [22].

Sutherland et al. [37] argue that “typically, remote teams operate independently and

16

communication problems limit productivity”. While they discuss a specific way in which

they have found that Agile can be successful in a distributed setting, they recognize that

most distributed Agile projects degenerate to a state in which each location is isolated

in doing its own specific task. This is described as the ‘isolated Scrum’ distributed team

model in which there is no real cross-location collaboration or teamwork [37] (see section

3.3.2.2 for more on this). One way to facilitate communication in a distributed setting is

by using a communications bridge [19]. In this case, there is no direct communication

between distributed developers, but decisions are communicated through people with

specific roles that have counterparts at each location [19]. However, this model seems best

suited for a situation where there is one primary location and one (or multiple) subsidiary

locations. These subsidiaries are not equal partners in the discussion and, in a sense, only

follow the decisions made by the primary.

Even if communication in itself is not the issue, time zone difference can also be

prohibitive. This is the case especially if business hours do not, or barely, overlap. This

can make it difficult for the development team to stay in touch with representatives from

the client [36]. Another characteristic of Agile that can cause difficulties for dispersed

development efforts is the lack of documentation; “since there was no written documen-

tation, conversation by phone for every little detail seemed awkward” [36]. Lacity et al.

state that ‘formal processes’ are the primary mechanism with which outsourcing relations

are facilitated [43], which is potentially incompatible with an Agile approach.

In addition to the previously noted points, cultural differences are also a factor to

consider [37]. While not unique to Agile development by any means, cultural issues could

become at least more pronounced when a methodology is used that necessitates close and

frequent interaction.

3.3.2 Large-scale Agile

One of the core assumptions of Agile SE is that the project operates in a relatively

flat hierarchy, with short communication lines, where everyone can talk directly with

everyone else. Requirements are gathered directly from the customers by the developers

themselves, and getting clarification is a matter of walking over to the relevant business

person. Traditionally, Agile is mostly applied in relatively small-scale projects [44] with

17

little reporting overhead or governance constraints [22], where these characteristics are not

unrealistic. However, in large projects, this type of arrangement becomes impractical. As

famously described by Brooks [45], as the size of a development team increases linearly,

the amount of communication lines increases quadratically (because everyone needs to

talk to everyone else). This leads to a point where the increase in productivity from an

extra team member is completely negated by the burden of the added communications

overhead. Plan-based (that is, non-Agile) development methodologies have addressed this

issue in various ways, for instance by segmenting the development effort so there is no

need for direct communication lines between all project members. However, as mentioned

in section 3.3.1, this runs contrary to one of the core principles of Agile. Elshamy and

Elssamadisy [38] state very explicitly that “[l]arge software development projects have

their own set of problems that need to be addressed” continuing that “[m]any of the

standard development practices in agile methodologies do not provide their expected

consequences [in large projects]”. However, authors such as Gary et al. argue that Agile

can be adapted to support the “right amount of formality” [40] when necessary. The most

commonplace way to deal with this is to adopt some sort of hybridized Agile methodology

that is adapted for the context. We will talk about this point in two parts; first, how (and

how well) specific Agile practices scale, and second; how the literature describes Agile

teams can be organized in large projects.

3.3.2.1 Scaling of Agile practices

Koehnemann and Coats looked specifically at adopting Agile for large programs,

and concluded that some Agile practices scale well, while others do not [22]. For instance,

the idea behind collective code ownership and rotated pair-programming is that all the

programmers end up being familiar with the entire system. For large projects, this is no

longer practical, as no single programmer can work on, or understand, the entire system

[22]. Other experience from practice shows that new management and tracking tools

are needed to maintain a high-level overview of the entire development effort as Agile

projects grow beyond a certain point [19]. On the subject of making Agile suitable for

large projects, Valade argues; “Agile is a great software development and implementation

methodology, but you still need people to oversee the project initiation and governance

18

model, as well as the deployment process” [17]. In the book by Larman & Vodde [46], in

their discussion of Agile scaling practices, the authors suggest the backlog be subdivided

into multiple ‘requirement areas’ as the number of requirements grows.

3.3.2.2 Organization of teams

A way to facilitate large-scale Agile development is to subdivide the development

team into multiple sub-teams. However, in this situation, special care must be taken to

prevent these separate teams from creating ‘silo’ applications within the larger develop-

ment effort [38]. This can be avoided by promoting communication between the teams,

for example by applying a ‘team of teams’ [39] (or ‘stream of streams’ [36]) approach.

Several modalities for distributed Agile development are mentioned in the literature. In

addition to the ‘isolated Scrum’ distributed team model mentioned in section 3.3.1, other

models are ‘distributed Scrum of Scrums’ and ‘totally integrated Scrums’, with partially

and completely integrated teams respectively [37]. Examples of roles that companies

introduce in large Agile projects and that provide input to requirements negotiation and

decision-making are: (i) ‘requirements architects’, responsible for “taking the high-level

features defined by product management and decomposing these, on a just-in-time basis,

into the more detailed requirements and stories needed to drive iteration planning” [40],

and (ii) business area owners [36]. The people in these roles collaborate with the overall

project manager (or the Scrum master) who ensures that teams are working in parallel in a

timely manner. The Scrum master is well aware of dependencies among teams and uses

this knowledge to resolve them as soon as they arise [41]. Furthermore, Baskerville [42]

discusses how organizations also adopt both Agile and traditional (plan-based) method-

ologies concurrently; “we find organizations which are not exactly integrating agile and

planned software processes; rather they are operating these two different ways of working

consistently within separate boundaries”. Another way of approaching the situation is

discussed by Auvinen et al. [20]; rather than taking an existing Agile methodology and

adapting it, they describe a situation where an organization opts to “inject select Agile

practices into a large program”, these practices being pair programming, the planning

game and collective code ownership. The key point here is reported to be the process of

making existing practices explicit; in some ways, the team was already following Agile

19

practices without knowing it. In this situation, adoption of Agile methods served to make

these processes more concrete and robust.

3.3.3 Agile requirements prioritization

The Agile RE literature discusses two characteristics of the project setting which

influence decision-making in requirements (re) prioritization at inter-iteration time: need to

embrace change and project constraints. However, the Agile authors point out that in large

projects these characteristics are complemented with scope and the number of project staff.

The last two are deemed ‘first order dimensions’ [39] that define the ‘largeness’ of a project.

Scope characterizes the complexity of the requirements, while number of project staff is

related to the exposure to risk; the more people involved in a project, especially the larger

the number of client representatives, the higher the risk of contradictory requirements.

The Agile literature sources acknowledge that Agile RE practices need to be implemented

differently in large projects because at a specific project size “things do not work out the

normal way anymore” [39], and because new problems may surface due to the size of

the team that would not be apparent at smaller scales. Koehnemann and Coats state that

“[l]eft to their own devices, multiple stakeholders can identify infinite features” [22], which

leads to the well-known problem of feature creep [47] or scope creep [48]. Especially for

large systems, there can be a disconnect between the desires of one stakeholder and those

of another, or between what is good for a single stakeholder and what is good for the client

organization as a whole. These authors recommend modeling the system behavior as a

whole using modeling languages as pre-development work as a way to stay ‘on the same

page’ with the clients as well as a way to maintain a high-level overview of the project

[22].

3.3.4 Comparison of other large Agile case studies

Table 3.2 offers an overview of the literature describing cases that relate most closely

to this case study. All these studies concern Agile projects involving at least 60 people. The

‘x’ or ‘-’ marks signify whether or not, respectively, the use of user stories (see [49]) and

new/modified Agile practices is explicitly stated in the paper. The last column explicates

to which extent the requirements prioritization process is discussed.

20

Those projects that did not adopt the user stories as an approach for their business

requirements, redesigned their requirements specification practices to “become more agile”

[36, 40, 50, 51]. We observe that while the literature sources explicitly discuss the user

story elicitation and documentation practices, they mostly only provide hints regarding the

requirements (re)prioritization process at inter-iteration time. We make the note, however,

that the number of publications on Agile practices in large projects is significantly less

than the overwhelming amount of ‘success stories’ in Agile projects in SME. This is not a

surprise, as Ambler indicates that most Agile teams consist of less than 10 people and are

co-located [44].

Table 3.2: Comparison of large Agile case studies

Source Business Sector Userstories

HybridAgile

Discussion ofprioritization?

[20] Auvinen et al. Mobile systems x x hints[38] Elshamy et al. IT service delivery (a vendor) x x hints[39] Eckstein Telecom x x explicitly[52] McDowell &Dourambeis Telecom x - not discussed

[21] Sulfaro et al. National Post Services x - not discussed[22] Koehnemann IT service delivery (a vendor) x - not discussed

[19] Gat Enterprise IT infrastructuremanagement solutions x - explicitly

[16] Larman Large government organiza-tions x - explicitly

[40] Gary et al. Safety-critical systems - x hints[36] Grewal &Mauer Oil & gas (a vendor) - x hints

[37] Sutherland et al. Digital library management (avendor) x x hints

[17] Valade ERP Implementation x x hints

[53] Bosch Software product business (avendor) x x hints

[41] Hong et al. E-commerce x x hints

[50] Kendall et al. Physics-based computationalengineering - x not discussed

[51] Christou et al. Banking - x not discussed

[54] Shatil et al. Defense system engineering(a vendor) x x hints

21

3.4 The use of literature in this projectFor our case studies, a firm theoretical basis proved essential for success. First, it

gave us the context necessary to formulate a comprehensive questionnaire for the interviews

that followed.

Second, it allowed us to recognize and pursue interesting facts that came up in

the interviews. In other words, the literature study helped the author develop theoretical

sensitivity, as Grounded Theory methodologists [55] call it. It helped us understand

the importance of looking at the investigated phenomena from multiple angles, make

comparisons while interviewing representatives in various roles, and build on ideas.

Third, once data collection was complete, our literature study allowed us to reflect

findings back to the existing body of knowledge, and compare and contrast our findings

to others in the published literature. The comparisons helped us understand what novel

results this study yielded and how and where our findings converge with or extend what

has been already published.

4. Research Approach

In this chapter we will discuss and support our choice of methodologies, and elaborate on

their application in this project. The data gathering and analysis process will be explicated,

after which we will arrive at our central concept and the structure of the rest of this thesis.

4.1 Research methodologyOur embedded case study design follows the guidelines by Yin [56] as well as Scholz

and Tietje [27]. We chose the embedded case study form because we wanted to obtain a

detail-rich, holistic and contextualized description - from multiple projects undertaken by

a large outsourcing vendor, regarding how requirements reprioritization took place. Yin

[56] makes a distinction between a holistic and an embedded case study. While a holistic

case study examines the global nature of a program or an organization, an embedded case

study includes outcomes from individual projects within the program [56]. We applied

purposeful sampling [27] when looking for suitable projects. Our purpose was to find

projects that had dispersed users and a project team that already earned experience in using

an Agile delivery model, or at least used a collection of Agile practices. Hence, our case

study involves more than one unit, or object, of analysis [27]. We used in-depth interviews

[57] as a data collection technique and Grounded Theory [55] as a data analysis technique.

The in-depth interviews technique was selected because of two reasons: (i) it is a suitable

technique for an inquiry like ours, and (ii) the resulting data offers a robust alternative [57]

to more traditional survey methods. This is especially the case when the absolute number

of participants is less important than a rich investigation of content. In our data collection,

we triangulated data from multiple sources (e.g. participants across three different projects

and in different roles, whose experiences varied broadly based on the specific role each one

played). This was done to ensure that our interviews provide a multidimensional image of

composing activities in our particular project setting.

As mentioned above, we carried out the data collection and analysis by implementing

the Grounded Theory (GT) building techniques according to Kathy Charmaz [55]. GT is

a qualitative method applied broadly in social sciences to construct general propositions

22

23

(called ‘theory’ in this approach) from verbal data. This approach is exploratory and

well suited for situations where the researcher does not have pre-conceived ideas. By

this, GT methodologists [55] mean situations in which the research does not start with

hypotheses or a predefined theory and then seek proof. Instead, the researcher is driven

by the desire to capture all facets of the collected qualitative data and to allow ‘theory’ to

emerge from the data. In the field of empirical software engineering (SE), researchers have

been using GT to find answers to questions that address relatively “unchartered land” (as

Baskerville et al. refer to it [42]), or a phenomenon about which little is known. Recent

examples of GT studies in SE include the publications [42, 58, 59]. Other recent papers

offer guidelines for applying GT in the software engineering field, specifically [60, 61].

In the RE field, examples of applications of GT as an empirical research approach are

published by Urquhart [62], Martin [63] and Martin et al [64]. The GT process is presented

in more detail in section 4.4.

4.2 Questionnaire designThe questionnaire that we used in this case study is based on a questionnaire previ-

ously used by Racheva and Daneva [5], and adapted to our specific context. It was also

expanded based on new areas of interest identified from the literature, and to reflect the

areas of interest of TRDDC. Most notably, our version of the questionnaire also looked into

properties of the distributed nature of the project, as well as the way in which knowledge

management was organized. We also expanded the existing questionnaire with questions

that would give us insight into the methods used to support distributed development.

Precise script was added to the questionnaire that was read literally to each participant

as the interview progressed. This was done to ensure validity by maintaining consistency

in the way interviews were conducted. For reference, the full questionnaire, as it was used

during the interviews, is provided in appendix C.

This questionnaire was verified by senior researchers, and tested in a pilot interview.

The pilot interview did not result in major changes in the questionnaire, and the results of

this interview were taken up in this study together with the other data.

24

4.3 Conducting the interviewsWe carried out 16 interviews, interviewing a total of 17 participants in different roles

across the three cases (see table 4.1). The interviews usually took between 50 and 90

minutes, with a few outliers as can be seen in the table. All interviews were conducted

either face-to-face or facilitated through video conferencing. For each of the embedded

cases, the first interview was treated as a pilot interview in order to establish whether or not

the questionnaire was applicable in the specific context. A special set of factual questions

related to context, scope, etc. were asked in these interviews. Before each interview, the

interviewee was provided with information on the research purpose, the research process

and the rights and responsibilities of the organizations involved in the case study. At the

meeting, the researcher and the interviewee went through the questionnaire, which served

as a guide to the interviews. Because the interviews were semi-structured, interesting

answers were often pursued further through ad-hoc follow-up questions.

Table 4.1: Overview of interviewsIncludes duration of interview (in minutes) and a word count of resulting transcript.

Project Participant Role Modality Duration WordsAlpha P0 Business analyst Face-to-face 92 min. 13,457Alpha P1 Delivery head Face-to-face 94 min. 13,492Alpha P2 Portfolio manager Face-to-face 63 min. 7,862Alpha P3 Scrum manager Face-to-face 67 min. 10,274Alpha P4 Business analyst lead Face-to-face 60 min. 8,491Alpha P5a & P5b Business analyst lead (2x) Face-to-face 82 min. 12,140Alpha P6 Test scenario team lead Face-to-face 36 min. 4,772Beta P0 Business analyst Video-con 87 min. 11,840Beta P1 Project lead Face-to-face 82 min. 14,883Beta P2 Tech lead Face-to-face 75 min. 12,005Beta P3 Project lead Face-to-face 11 min. 1,799Beta P4 Project manager Video-con 78 min. 11,597Beta P5 Lead developer Video-con 84 min. 12,004Gamma P0 Project lead Face-to-face 100 min. 14,006Gamma P1 Architect Face-to-face 59 min. 8,213Gamma P2 Consultancy lead Face-to-face 52 min. 7,171

After the first round of interviews, it was indicated that interviewees would like to

have some idea beforehand of the kind of questions that would be asked by the researcher.

Because the flow of the semi-structured interview process depends on the answers being

25

spontaneous rather than rehearsed, it would have been methodologically unsound to

circulate the questionnaire as a whole to future interviewees. To still be able to give

participants a general indication about the kind of questions that would be asked, an

abstract questionnaire was created that contained sample questions from each section. This

put participants at ease and increased the willingness to participate in the study.

4.4 Data capturing and analysis processSchematically, the process we used in utilizing Grounded Theory can be visualized

as shown in image 4.1. Although this model by Adolph et al. [60] is based on ‘classical’

or ‘Glaserian’ ([65]) Grounded Theory, it is a general representation that shares enough

similarities with the version of GT put forward by Charmaz [55] to be useful for our

purposes. Although GT is traditionally mainly applied within the disciplines of Nursing

and Sociology, recently, SE researchers have shown the methodology to be applicable

within their domain as well. Even looking at the study of Agile teams specifically, we see

many recent studies using GT [60, 61, 66, 67, 68, 69]. In GT, researchers should approach

their area of study without preconceived notions and let the theory arise from the data

itself. Therefore, we started our data gathering process without a precisely defined problem

statement. As stated by Hoda et al.; “having a preconceived research problem can cause

the research to be influenced by the existing research literature in the area” [61]. As

illustrated in the model, data collection and data analysis happened side-by-side. This way,

researchers know when theoretical saturation is reached and there is no need to collect

more data. Still, in our study, data accumulated more quickly than we could analyze it.

Because of this, after all interviews were completed, there was still some backlog of data

to analyze. However, this did not pose any problems for our study as all concepts were

thoroughly saturated by the end of the data analysis phase.

All interviews were recorded electronically. Our choice to record the interviews is

contrary to recommendations by Barney Glaser, one of the founders of classical GT [65].

Glaser argues that it is sufficient to take notes during the interviews and work these out

later. However, we found recording to be essential for capturing all details of our long

and information-dense interviews. Also, recording the interviews allowed us to focus on

listening to and guiding the interviews. This is echoed by other practitioners who used GT

26

in a SE setting, namely Adolph et al. [60] and Hoda et al. [61].

Figure 4.1: Grounded Theory processDiagram describing the Grounded Theory process of analyzing and making sense of the

data, and working towards a theory. From Adolph, et al. [60]

After the interviews were completed, the recordings were manually transcribed.

Because data analysis was performed collaboratively, special care was taken to transcribe

the information as precisely as possible. These transcripts could then be treated as the ‘raw

data’ by the other researchers after all the personal and confidential information had been

removed5. For example, notes were added to the transcripts about the non-verbal language

of the interviewees. These indicated, for instance, whenever an interviewee paused and

expressed doubts regarding the completeness of the information provided. This turned out

to be critical information to the other researchers who conducted their own analysis. The

result of the transcription process was a very large dataset containing a total of 163,8915As required by the case study organization.

27

words across 16 transcripts. To give an indication of the size of this dataset; when pasted

into a single document, it amounts to 331 pages of plain text6. The contents of this dataset

are visualized in the form of a ‘word cloud’, which can be found in appendix D, as well as

in another form on the title page.

Every time a transcript was ready, the data analysis guidelines of the GT approach

[55] were applied (as data collection and analysis were done side-by-side). Essentially, GT

analysis includes ‘coding’ and ‘constant comparison’ of the interview data. Coding is the

way of learning to know the data. It is the process of conceptualizing the data by reading

the data line-by-line and marking a segment of data with a descriptive word or phrase.

Constant comparison is the process through which we constantly compare instances of

data that we have named as a specific category with other instances of data, to see if these

categories fit and are workable. This process helps to group the data into categories. The

resulting codes and categories guide the writing-up of our results and aid in improving the

accuracy of the claims [55]. It also allows for traceability throughout the research process,

so claims and conclusions can easily be traced back to their source.

As we mentioned, GT methodologists recommend that coding and constant compar-

ison be done iteratively. Two researchers (Van der Veen and Daneva) coded the interview

text independently, at different locations and with little communication between them.

This was done to ensure code validity. However, the results of the coding and interpreta-

tions of the data were discussed and peer-reviewed iteratively by two other researchers

(Amrit and Sikkel) to establish consistency of the emerging clustering into categories.

Our analysis proceeded as follows: the researchers first read the interview texts and at-

tached a coding word (or phrase) to a portion of the text - everything from a single word

to multiple paragraphs. The coding words were selected to reflect the relevance of the

respective portion of the interview text to a specific part of the studied phenomenon. This

could be a concept (e.g. ‘requirements dependency’, ‘distribution’), or an activity (e.g.

‘knowledge sharing’). Some of the codes were a logical continuation of the composition

of the interviews, as standard aspects of the process were discussed, e.g. ‘size of the

team’, or ‘decision-maker’. In the case of specific incidents, we asked the interviewee

what concept or activity the interviewee had been talking about, which we duly noted. We6Using Times New Roman font size 12 using default margins and spacing in Microsoft Word 2011

28

then clustered all pieces of text that relate to the same code, in order to analyze them in a

consistent and systematic way. We then compared the codes to each other for the purpose

of searching for similarities and differences between them. This inter-code comparison

enabled us to identify underlying and emerging uniformities in the meanings of the codes

(i.e. the concepts or activities) and with this we produced categories. Other categories,

and respectively - codes, emerged during the coding process as a result of observations

we had not anticipated. These primarily concerned concepts and aspects of the process

we had not explicitly addressed in the questionnaire. Those were, for example; ‘delivery

stories that describe the non-functional requirements and architecture’, ‘domain owners’,

or ‘risk’. Code saturation occurred rapidly; most codes were created in the early interviews,

with only a few new ones being added occasionally. This is shown in the graph in image

4.2. Participants prefixed 1, 2, or 3 were members of projects Alpha, Beta or Gamma,

respectively. The data on which this graph is based can be found in appendix H. An

increase in new codes is evident for the first interview(s) of new projects, but not so much

as to warrant classifying these as completely separate cases.

Figure 4.2: Number of new codes for each interview

The codes themselves were applied manually using the software tool TAMS An-

alyzer7. While other researchers often use more sophisticated tools such as Nvivo8, we7Open-source, Mac-based qualitative data analysis tool; http://tamsys.sourceforge.net/8Nvivo qualitative data analysis package; http://www.qsrinternational.com/products nvivo.aspx

29

found these tools overly complicated for our purposes, and in some senses actually dis-

tanced us from the actual data. This sentiment is shared by Adolph et al. in their treatment

of the application of GT in SE [60]. Additionally, Hoda et al. stated that “[w]e tried using

this software (Nvivo) but found that its structural framework limited the ways in which we

could organize and interact with the data.” [61]. Using a more ‘hands-on’ data analysis

method enabled us to interact with the data directly, also allowing us to become more

familiar with it in the process.

The coding process resulted in 83 unique codes (see appendix I for a list of all unique

codes, and appendix F for an overview of their relationships) being applied 1293 times

across all transcripts. The frequency and distribution of the (top-level) codes is visualized

in figure 4.3. Two interviews immediately stand out, namely Alpha P6 and Beta P3. This is,

however, not unexpected. As Alpha P6 is the ‘Test scenario team lead’, this participant is

relatively far down-stream in the project. In a project as large as Alpha, it is not unexpected

that someone in this position would be unaware of some characteristics of the rest of the

project. Participant Beta P3, the project lead, had only a limited amount of time available

for the interview due to a visit from the client. Because of this participant’s area of

expertise, we focussed our interview on the section of the questionnaire dealing with value

creation (see C.5), which resulted in the codes for this interview leaning in that direction.

Also, codes for ‘context’ occur mainly in the first interviews of each project. This, too, is

not unexpected as we had a separate section of ‘facts’ questions in our questionnaire that

we only asked in the first interviews. Also, the semi-structured nature of the interviews

lead us to ask more about the project’s circumstances in the early interviews, and to a

lesser extent later on when we had a clear view of this.

30

Figu

re4.

3:C

ode

freq

uenc

yan

dco

depe

rcen

tage

grap

hs

31

The percentages with which codes occur in the three projects are displayed in figure

4.4. The raw data underlying these graphs is provided in appendix G. The codes shown

in these figures are only the high-level codes and include all ‘nested’ or sub-codes. The

organization of further sub-division of codes is represented schematically in appendix F.

Figure 4.4: Percentage of code occurrence per project

When coding was completed, a web-based tool was created to aid in data analysis.

This helped facilitate knowledge sharing and collaboration in the research project, as well

as make it easier to find specific information. Screenshots of this tool are provided in

appendix J. The text of the actual thesis itself arose from theoretical analysis of the coded

information. Traceability between text and data was maintained by keeping the literal

source material as comments in the thesis source document next to the narrative that arose

from it (see appendix E).

4.5 Memo structuring and core conceptDuring the coding we took extensive notes, called ’memos’ in GT [65]. As explained

by Adolph et al. [60]: “Memos capture, make real, and preserve the ideas emerging

from an analyst’s pre-conscious processing as the data are analyzed . . . Whenever an idea

emerges about a concept or the connections between concepts, analysts should stop coding

32

and write a memo”. For us, memos were taken in a simple text editor and initially ordered

only chronologically (that is, in the order in which the interviews were analyzed). This

resulted in a document with short paragraphs of observations, hypotheses and quotations.

After the coding and accompanying memoing processes, GT methodologists indicate

that a process of ordering and structuring of the memos should take place. Doing so, “the

analyst sorts the memos to organize them into the outline of a theory for writing and

presentation” [60]. Thus, the ordering of the memos leads to the structure of the thesis and

of the eventual theory. Our memos were structured by placing all the separate paragraphs

of text into one large mind-map9, and grouping and clustering them according to theme

or overarching concept. Purely as an indication, a high-level view of this mind-map is

provided in appendix K. When the nodes are collapsed so that only the concepts and

categories are visible, this mindmap appears as depicted in figure 4.5.

Figure 4.5: Memo clustering mindmap - collapsed

At this stage, GT practitioners search for a central, unifying concept to take the place

of the core concept. In our case, this would take the place of the center node. Hoda et al.

[61] offer four criteria to consider for choosing a category as the core, namely;

1. It must be central and related to several other categories and their properties.9Using mind-map editor ‘Freemind’ (http://freemind.sourceforge.net/)

33

2. It must reoccur frequently in the data.

3. It takes the longest to saturate.

4. It relates meaningfully and easily with other categories.

Several candidates for core category were considered in brainstorming sessions in

collaboration with senior researchers. The main candidates were Hybrid Agile, Managing

Risk and Coping with Requirements. When these options are considered with the above

criteria in mind, we can start a process of elimination. ‘Hybrid Agile’ was dropped first for

not relating directly to much of the data and being a specific expression of more high-level

factors (more on this later). We also see that ‘Managing Risk’ satisfies criteria 1 and 4,

but not 2 and 3; ‘Risk’ was not elaborated upon as a consideration by our participants and

was not among our codes (see appendix I) or memo categories (see figure 4.5). ‘Coping

with Requirements’, on the other hand, satisfies all four criteria; (i) all top-level categories

in figure 4.5 can be related directly to it, (ii) the way requirements were handled was

elaborately discussed by all participants, as evident from figure 4.4 and even the Word

Cloud visualization of the data in appendix D, (iii) as can be seen in the code overview

graph in appendix F, ‘requirements’ has the most extensive sub-category structure, and

(iv) our other concepts and categories can be easily related to ‘Coping with Requirements’.

The role of risk in relation to coping with requirements can be thought of in terms of a

why-how tree [70]. A why-how tree is a tool for thinking about solutions and root causes.

Asking the question ‘why’ moves one up the tree, toward the reasons behind potential

solutions. Asking the question ‘how’ constitutes a move down the tree, moving toward a

concrete solution.

Our why-how tree is depicted in figure 4.6. The top-level memo concepts are the

blue boxes at the bottom of the tree. For each of these, asking the question ‘why?’ leads us

up, towards ‘Coping with Requirements’ (except for ‘separation of concerns, which is a

specific solution for project management). This shows that it is not the case that ‘Managing

Risk’ is unimportant or completely out of place, but it can be said to be a higher-level

concern, up from ‘Coping with Requirements’. The tree also shows ’Hybrid Agile’ and

its more specific position within our broader set of considerations. Thus we assert that

‘Coping with Requirements’ is a suitable core category for our study.

34

Figure 4.6: Brainstorming about core concept, why-how tree

Our memo clustering mind-map is thus restructured to reflect this, removing ‘re-

quirements’ as a separate concept and promoting its former sub-concepts to top-level

concepts (see figure 4.7). The structure of this mind-map led to the structure of the ‘results’

chapter (page 36) in this thesis, and both directly and indirectly gave shape to the eventual

theory. We use here the definition of ‘coping’ as “to face and deal with responsibilities,

problems, or difficulties, especially successfully or in a calm or adequate manner”10. Thus,

dealing with requirements in an effective way was identified to be the central concern in

our dataset.

Having identified the core concept, we can relate the top-level categories to our

coded information, as well as to what we know about these concepts from the existing body

of knowledge. The literature study provides us with the necessary context to interpret our10Using the 2nd definition of ‘cope’ from Dictionary.com, http://dictionary.reference.com/browse/cope,

accessed 03-11-2011

35

Figure 4.7: Memo clustering mindmap after identification of core concept

observations from a theoretical perspective and thus, forms the basis for the interpretation

of our data. Following this, the results of the coding process help us categorize the dataset

and make it more approachable through our analysis tool. Finally, the list of observations,

notes, hypotheses and quotes that we have called ‘memos’ has been structured, which

helped us identify our core concept. Thus, the triple lenses of structured memos, nested

codes and literature study, helped us focus and interpret our observations (as illustrated in

figure 4.8).

Figure 4.8: The three lenses through which we interpet our observations

5. Results

The structure of this chapter comes from the memo structuring exercise as outlined in the

previous chapter. There, we grouped our findings around the core concept of ‘coping with

requirements’, as discussed in section 4.5. How the sections of this chapter relate to the

categorization of memos is depicted in figure 5.1.

Figure 5.1: Structure of Results

The results of this analysis will be discussed in this chapter, and will serve as a basis

for the theory building exercise described subsequently. Our foundational literature [1],

and consequently our questionnaire, placed a strong emphasis on requirements engineering,

and especially on the requirements prioritization process. Our other areas of interest, aside

from being interesting in and of themselves for researchers as well as practitioners, are

essential in understanding this process; they serve to set the context for our description of

the way in which our case study projects coped with requirements, and will allow us to

trace choices to properties of the project’s circumstances.

We will often refer to interview participants, and/or give direct quotations; please

refer to table 4.1 for an overview of roles and responsibilities of our interviewees. Some

of our findings are universal to all cases (↵ \ � \ �), while others are only found in two,

or are unique to specific projects. This can be thought of in terms of a Venn-diagram as

shown in figure 5.2, where some characteristics overlap while others do not.

As we discuss our findings we will explicate how general or specific they are. In

the next chapter, this will be revisited as findings are structured in tables depicting those

finding’s generality or specificity.

36

37

Figure 5.2: Generality of findings

5.1 Is Agile worth the risk?As we noted in chapter 3, Agile is not commonly used in projects that are either

large or distributed, let alone in projects that are both. It is, therefore, not unreasonable

to ask the question “is it worth the risk to go Agile in such projects?” It is worth noting,

at this point, that Agile was relatively new to our case study organization. According to

participant Beta P3, the organization did not have any Agile projects three years prior to

our study. In our interviews we asked specifically about the added value of Agile as a way

to answer the question whether the benefits outweigh the obvious risks we identified from

the literature.

We will look at this question from a few different perspectives; first, we will look at

the Agile adoption process and discuss some of the reasons our participants mentioned

for choosing Agile as their development methodology. In the two subsections after that,

we will discuss the effects of Agile adoption on the developer and the client organization

respectively. Finally, we will look at the role of Agile in collaboration in a distributed

setting.

5.1.1 Agile adoption

In contrast to what our literature study showed to be typical, the desire to use

Agile development methodologies in our cases came from the client, rather than from the

vendor’s organization. Participant Beta P2 stated specifically that the reason for using

38

Agile is that, for the client, previous projects were not able to reach the market on time and

did not meet the customer’s expectations. The primary reason why the client organization

opted for Agile was to minimize turn-around time, according to Beta P1. In project Alpha,

usage of Agile was also a request from the client’s side for similar reasons. For project

Gamma, usage of Agile was inherited from another team in the organization with which

there was collaboration; “We realized the benefits they were getting” (according to project

Gamma’s project lead).

In our study, we saw that there were some change management issues when Agile

was first introduced. This is illustrated in quotation 1.

“In terms of Agile, the clients themselves were adopting Agile, and there was quite

a lot of opposition to change, from Waterfall as well as a lot of standard process in

terms of documentation and all that. So it was a slow process. You have to change the

people’s mindset to adopt Agile, and once you are into Agile.”Quotation 1, Beta P5

Agile adoption was organized using a ‘core team’ of people who were trained in the

methodology. These people served as evangelists (also discussed by Gat [19]), promoting

Agile internally and making the required organizational adjustments. On occasion, it was

necessary to educate client’s representatives about Agile. This was not a comprehensive

end-to-end training effort but a practical course so stakeholders could get a working

understanding of the process and effectively participate in the process.

Participants from all projects mentioned a learning process when Agile development

was first adopted; it took a while for everyone to get used to working in iterations. This

corresponds to experiences reported in the literature [19]. There were initial hick-ups

(mainly planning issues) due to inexperience with Agile, but these cleared up after the first

few iterations. According to participant Beta P4, it took around three to four months for

the project to get fully accustomed to the Agile methodology.

Some Agile practices were tried out, but ultimately found to be unsuitable. Partic-

ipant Beta P1 mentions that test-driven development was initially attempted, but it was

found to be too time-consuming to have all test cases ready before starting the development.

Another Agile practice, pair programming, was also tried out. However, there were issues

39

due to the teams not being co-located, similar to findings in the literature [22]11.

The point was raised that the choice between Agile and traditional (waterfall) method-

ology can depend on the type of product being developed; Participant Beta P1 shared with

us the belief that Agile is primarily useful when a product has to reach the market as soon

as possible, while for legacy systems, traditional methods can be more appropriate.

5.1.2 Agile and the developer

Increased speed of development was mentioned by most of our participants as the

greatest single benefit of using Agile. The project lead for project Gamma stated that,

though sometimes being more chaotic, Agile ‘keeps things moving’ in a way that is not

normally seen in traditional methods. Increased motivation played a significant role in

this. Many of our participants stressed the boost to morale that they experienced when

using Agile. The close collaboration with the client, and delivering working pieces of

functionality on a regular basis, had a very positive impact on motivation. It is said to give

developers the confidence that what they are developing will actually be of benefit to the

client and will actually be used. This is illustrated in quotation 2.

“It really gives a good feel to see, ok, the system to this extent is working, it’s not just

designed, not just developed, it is fully tested, it is working”Quotation 2, Alpha P1

Participants also stated that the development effort as a whole was more efficient

because, thanks to rapid feedback, no effort was wasted on activities that would not

contribute to business value. At the same time, the open nature of the Agile philosophy

ensured that people were better aware of what was going on in the project as a whole. This

led to high scores on user acceptance tests; 90% on project Alpha’s entire first collection12.

However, there were also some challenges associated with Agile development

from the perspective of the developer. Some of our participants mentioned difficulty

concentrating on tasks due to shifting requirements and aggressive release cycles. This is

stated explicitly by one interviewee, Gamma P1, who mentioned that because of the high11It should be mentioned that the literature shows at least one successful implementation of distributed

Pair Programming, reported by Maurer [71]12first set of iterations, see 2.1

40

pace of development in Agile projects, and the pressure introduced through interactions

with the client, there is less time for research or exploratory work. Also, it becomes difficult

to perform large, overarching changes that might not fit in the space of a single sprint. It is

interesting to note that this participant, an architect, was the most elaborate in identifying

challenges posed by Agile vs. traditional methods. This is not unexpected as architecture

is much more in flux in Agile development, as opposed to a Waterfall-approach where it

can be defined up-front and then remains more or less static.

Another issue is confusion in the early stages of a project, as touched upon when

we discussed adoption in 5.1.1. Interviewees also mentioned extra issues when inducing

new members into the project, because most new recruits will be unfamiliar with Agile in

general, and of course with the specific implementation of the methodology in the project.

Also, although we mentioned positive effects on motivation above, confidence and morale

can sometimes suffer if already developed items are scrapped due to re-prioritization

events.

Having experienced developers is an important success factor for Agile development.

One of our participants expressed that if you have a relatively inexperienced development

team, it is sometimes difficult to accurately estimate the time necessary for development13.

This can lead to situations where targets cannot be met by the end of a sprint, causing

re-prioritization and/or de-scoping of requirements.

5.1.3 Agile and the client

Our data seems to indicate that the main advantages of Agile development for the

client are twofold; decreased time-to-market and more business value being delivered.

These reasons were, in fact, why the client in project Beta opted for Agile, as illustrated in

quotation 3.

“So as part of this Agile, why [CLIENT] went to Agile . . . They had many projects,

which did not reach the market on time, and which were not able to meet the customer’s

expectation.”Quotation 3, Beta P2

13The role of experience on the estimation and planning process is elaborated upon in subsection 5.7.1

41

The constant interaction with the client raises the business value of the delivered

functionality. The client is more likely to get what it expects. Early feedback, frequent

discussion and transparency ensured that effort was directed towards functionality that

would actually benefit the client; the client had insight into the whole process and could

redirect effort whenever necessary. One interviewee summed up the difference between

traditional methods and Agile from the client’s perspective as given in quotation 4.

“From the client’s perspective, [in traditional methods] sometimes it becomes difficult,

because whatever has been approved or signed off in requirement analysis, that is the

only thing they will get at the end of the project. Whereas part of Agile methodology,

the client has better control over [the process], especially for changing requirements,

or changing market”Quotation 4, Gamma P2

In our cases, the clients actually specifically asked for Agile, as we noted earlier.

As we know from the literature, one of the most important success factors for Agile

development is active participation of the client, and the client’s willingness to invest the

time and effort necessary. All interviews indicated that there was a very good working

relationship with the client, with a very active engagement in the process (more in this in

the next subsection).

5.1.4 Agile and collaboration

In this subsection we will discuss our participants’ experiences with collaboration

as it relates directly to the use of Agile. We examine collaboration as a whole in more

detail in section 5.3. In our case study, we found that Agile methodologies actually helped

to increase transparency across distributed locations. This was especially noticeable in

project Beta, where the development effort was not mainly concentrated in one area, but

was spread across multiple sites in Asia and Europe. Agile was said to decrease the gap

between the business and the development team, and actually helped promote the idea that

everyone was part of a single team, rather than an ‘us’ and ‘they’ mentality. This is stated

explicitly in quotation 5.

42

“This Agile made us feel that [CLIENT] was part of [DEVELOPER ORGANIZA-

TION]. So we did not find any major difference”Quotation 5, Beta P1

This point is further illustrated in quotations 6 and 7;

“. . . because if it was on the waterfall it was only the project manager or the project

lead who interacts with the client and gives us the information. But here it is not that.”Quotation 6, Beta P1

“(Agile is...) about making the entire team to understand what they are going to do,

what they are going to deliver. Including the business, including the IT. Typically,

what happens is, the business spends some time, defines the requirements, they forget

it and give it to IT. But here, once we make both the parties understand what we are

going to deliver in terms of business value, then both will work together and... will try

to help each other”Quotation 7, Beta P3

Multiple participants explicitly mentioned that the transparency inherent in the Agile

development methodology improved collaboration in the distributed setting, as exemplified

in quotation 8.

“Typically on a waterfall project, if it’s an eight-month project, I don’t think the

business will have a very clear visibility on where the IT team is at any point of time

. . . But then here, here you’re talking about daily stand-ups, daily burn-downs and all

that”Quotation 8, Beta P4

The previously mentioned good working relationship with the client certainly plays a

role in this. In project Alpha, where most of the development work was done in Asia, there

were multiple representatives from the client actually at ‘off-shore’14 to act as product

owners and provide the development teams with a readily available point of contact.14That is, on location with the developers

43

Of course, to achieve these benefits, adjustments had to be made to the ‘standard’

Agile practices. This is further discussed in the following sections.

5.2 Development process in generalPart of our study was to get a general idea of the overall management of the devel-

opment process, and in this section we will report our general observations. Aside from

being interesting for researchers and practitioners in and of itself, this examination will

also help put our other results in context.

5.2.1 Approach

All projects used different hybrid forms of Scrum, the major features of which will

be discussed per project, and summarized in table 5.1. For a more general description of

the individual cases, see chapter 2. After discussing the specifics per project, we will look

at the overall project structure that we found to be in common across all three cases.

5.2.1.1 Project Alpha

Our first case applied, what they called, a Scrum/Stream approach. High-level seg-

mentation of the project was based on five domains, which represented the separate aspects

of the client’s business. Each domain had a domain owner, which was a representative

from the client who could provide clarification and had the authority to sign off on features.

Development work for each domain operated roughly independently. The order in which

requirements were taken up was based on pre-established high-level business process

maps which show the functional sequence. Scrums are organized in collections of seven

weeks, which were further split up into sprints of two weeks. For each of these sprints,

two or three days were used for planning, preparing releases, etc. The last week in the

collection served as a buffer in case of overruns, and also provided time for consolidation

and packaging, as well as planning for the next collection. There was a collection manager,

who was the person ‘above’ the product owners in the decision-making hierarchy. This

person made decisions if the product owners could not agree, or if the right person for

a particular decision was not available. Aside from the functional sequence, the order

in which work items were taken up was also influenced by dependencies between the

44

features. Initial prioritization came from looking at the business processes and estimating

the required effort. The business process model is made at the start of the project and

sliced to fit into the collections.

Figure 5.3: Project organization - Alpha

Concurrent to the Scrums were Streams, which ran the entire length of a collection,

without being further subdivided into sprints. Streams were created for work that could not

realistically be divided into smaller chunks. This could be due to a fundamental monolithic

nature of the work in question, such as the architecture and attribute definition, or because

of a high degree of interdependency within a feature.

Project Alpha’s high-level organization is depicted schematically in figure 5.3.

5.2.1.2 Project Beta

Rather than creating a single product from end-to-end as with project Alpha, the

engagement in this case was based on supplementing an existing development effort at the

client’s side (see also section 2.2). As such, high-level segmentation and organization of the

project was different in several respects. Development centered around several products,

which were then segmented into releases of around three months in duration. These

releases, in turn, consisted of sprints of either two or four weeks. In the latter case, this

amounted to 2 12 weeks of development, followed by 1 1

2 weeks of testing, consolidation and

preparing the release. There could be multiple sprints running concurrently for different

products. Rather than functional sequence, the order of development is determined mostly

by the amount of business value a feature is expected to deliver. At the start of a new

release, some pre-development work is carried out, as one of our interviewees explained in

45

quotation 9;

“The first month, you are typically trying to do some kind of iteration 0, where you

are trying to build up your product backlog and all that, you are trying to prioritize

that, you have a first pass at the product backlog.”Quotation 9, Beta P4

An iteration report was prepared at the end of each iteration, for internal reporting

purposes as well as for progress monitoring by the client. There is evidence that at least

some parts of the approach used in project Beta originated from project Alpha and were

adopted as best practices (such as the use of ‘delivery stories’, see section 5.6).

Beta’s organization is summarized in figure 5.4 Note that the number of concurrent

releases on which work is done simultaneously is hypothetical (and varied over time); the

point is that the project’s organization supported more than one concurrent release at a

time.

Figure 5.4: Project organization - Beta

5.2.1.3 Project Gamma

Although the overall project to develop a model-driven development environment

had already existed for at least 10 years, Agile had only been adopted as the development

methodology for four months at the time of our interviews. As discussed previously,

project Gamma differed significantly from our other two cases. Most of these differences

can be traced to two characteristics; (i) the client was an internal customer within the

46

same organization, and (ii) the product was part of an ongoing research and development

effort into model-driven engineering. These characteristics have a number of important

consequences for the organization of the project. For one, in contrast to what we found

in the other cases, here it was the development team that had the final word on which

requirements are be taken up and how re-prioritization takes place. Another consequence

was that multiple extra factors influenced the course of the development effort (see section

2.3). Agile was adopted after the example of the internal client, which had been using the

methodology for its own development work. There does not appear to be a higher-level

segmentation of the project beyond sprints. These sprints were four weeks in duration

when the methodology was first adopted, but were later extended to six weeks. Of these

additional two weeks, 112 is used for testing, with the remaining time reserved for preparing

the release. There was a period of one week between the end of one sprint and the start of

the next; one of our participants, the project lead, believes it was essential to leave some

‘breathing-room’ between sprints to alleviate some of the pressure and to allow people to

“tend to their own things for a while”, before continuing with the sprint process.

The overall organization of the three projects is summarized in table 5.1 and schemat-

ically depicted in figure 5.5. Note that the activities of project Alpha were subdivided

into domains, and development activities took place either in collections of sprints, or in

so-called ‘streams’. Project Beta used either two- or four-week sprints depending on the

requirements, and were part of releases of around 3 months. Project Gamma maintained a

one-week intermission between the end of one sprint and the start of the next.

Table 5.1: Overall project organizationHigh-levelsegmentation

Low-level seg-mentation

Iterationduration Iteration segmentation

Alpha Collections Sprints & Streams 2 wks 80/70% dev, 20/30% otherBeta Releases Sprints 2-4 wks 621

2% dev, 3712% other

Gamma - Sprints 6 wks 6623% dev, 331

3% other

5.2.2 Structure

Several different teams went into the organization of the project as a whole. There

was a Master Project Backlog (MPB) team (consisting of business analysts), which actually

wrote the requirements in collaboration with the client. These requirements were then

47

Figure 5.5: Organization of tasks in case study projects

elaborated upon by a design team (more on this in section 5.6). A quality assurance

team reviewed the requirements for attributes such as correctness, sufficiency, and consis-

tency. At this stage, a technical team looked into the technical dependency between the

requirements. This overall structure was seen in all cases, but was implemented in slightly

different ways. For project Beta, not all of this process was in the hands of the developer’s

organization - being part of a larger development effort. Project Gamma, being smaller

than the other two, seemed to carry out roughly the same tasks but in a more compact

setup; a consultancy team on one hand gathered the requirements, with the development

team on the other providing technical input.

Quality checks were part of the sprint planning, as discussed in quotation 10;

“We plan in such a way that we have quality checks also, in between. For example,

whatever design we do, that undergoes internal review, ok? And even the code

development also undergoes review within that time frame.”

48

Quotation 10, Alpha P2

There was a specific regression testing team that made sure that the code-base as a

whole remained consistent. Defects discovered by this team were considered in the sprint

planning process and were thus taken up as work items in subsequent sprints.

One of our participants discussed the impact of using an Agile methodology on the

structure of their project, as shown in quotation 11.

“The Scrum is a mingled team, it will have all the people. It will have the managers, and

the developers, testers, everybody, and the task will be thrown in front of everybody,

and everybody will pass their opinions and we will do the rough estimation as well on

the spot. And based on the developer input, and analysis, analyst explanation.. we will

go for the high-level estimation and the tasks will be allocated and we will go with the

task”Quotation 11, Beta P2

The main metric used for project management was Scrum velocity. This was the

main way in which performance was tracked. No other formal methods seemed to be in

use, although project Gamma’s project lead did express that they would like to use such

methods if they existed and were found to be applicable in their setting.

An important concept mentioned by participant Beta P5 is technical debt (see [72] for

a discussion of this concept). As the project progressed and the software architecture took

shape in a more or less ad-hoc way (since it was an aggregate of many short-term iterations

of development) this was something that needed to be considered as the product grew.

Technical debt can make it necessary to spend an iteration on refactoring the architecture

to accommodate hitherto unforeseen requirements.

5.3 CollaborationIn our questionnaire, we looked specifically at the organization of collaboration

between the developer and the client. It is also a subject that came up naturally in the

course of most interviews (being semi-structured). This led to several interesting findings,

which we will discuss here.

49

Something that came up again and again in our interviews was the necessity for a

good working relationship between the client and the developer. One participant stressed

that a critical success factor for the smooth operation of the requirements prioritization

process was for the business and development teams to be able to work together well, as

stated in quote 12.

“It’s not like a... ‘we’ and ‘they’ kind of approach; it’s really one single team, which

has actually benefitted the project big-time.”Quotation 12, Beta P4

Project Alpha, with its co-located client representatives, provides strong evidence

for the benefits such an arrangement can have. Being able to simply walk up to people

from the client and ask them questions face-to-face made the wishes and motivations of the

client significantly less opaque to the developer, and is an important factor in promoting

the idea of ‘teamness’15. However, there were some issues to overcome at the start of the

project. Initially, issues arose because on-site representatives were either not completely

knowledgeable, or not authorized to make certain decisions. This led to situations where

previously clarified requirements or authorized decisions would be overruled later on in

the process by others from the client’s side. This is illustrated in quotation 13;

“Earlier, what happened was; they confirmed a few things verbally, so we implemented.

In the mail also they confirmed. But when it was raised in the Sharepoint site the

answer was altogether different.”Quotation 13, Alpha P3

These issues were addressed by making sure that representatives had the proper au-

thority, and by further explicating which stakeholders from the client were knowledgeable

on which topics (see section 5.4 on knowledge management). Having an on-site product

owner for each Scrum-team was said to have been ideal. However, practically, it was only

possible to have one representative per domain, which increased the time necessary for

clarifications. In project Beta, interaction between distributed developers was consciously15As discussed in section 5.1.4

50

promoted, ideally to the point where there was no longer any distinction between people

from the client and those from the developer’s organization. In project Gamma, there was

a clear push from the client to become part of the development effort, so there were no

difficulties integrating them into the team as far as motivation was concerned.

In the following sub-sections, we will discuss how collaboration was facilitated and

coordinated, respectively.

5.3.1 Facilitation

For projects Alpha and Beta, distribution was facilitated in various ways; just

using email was found to be insufficient and caused a significant lag in communications,

sometimes delaying efforts by a day or more. Interaction was augmented by more direct

means of communication, starting with instant messaging. An application called Webex16 is

used for activities such as showing screenshots and giving demos. Distributed collaboration

played an especially important role for project Beta, where the vendor’s team supplemented

a team of the client’s developers. Therefore, it was especially important for project

members to effectively communicate with distributed counterparts. Although many of this

additional facilitation was organizational (see the next subsection, 5.3.2), interviewees

from Beta were the most elaborate in discussing the tools used by them and the client to

collaborate. Here, an application called Sametime17, was used to facilitate direct video-chat

between people from the client and the vendor. The client played an active part in setting

up this arrangement, installing webcams and enabling direct lines of communication. The

client also helped facilitate distribution by extending support after office hours to increase

overlap between the different time zones.

For project Gamma, collaboration was facilitated through a wiki-like discussion

forum, email, and sometimes telephone. The client published their ‘wish-list’ on the wiki,

which was then used as a platform for discussing feasibility.

5.3.2 Coordination

In projects Alpha, collaboration was mainly between representatives from the client,

and business analysts or intermediaries from the vendor. Effectively coordinating distri-16http://www.webex.com/, accessed 30-08-201117http://www-01.ibm.com/software/lotus/sametime/, accessed 30-08-2011

51

bution mainly resolved around being able to determine which members of the client to

approach with which questions, and to effectively negotiate around differences in time-

zone. Due to the greater need for distributed collaboration, project Beta had coordinators

in place on each of the separate locations to ensure that interaction went smoothly, as

described by one participant in quotation 14;

“The coordinator is responsible to make sure that the communication happens properly

and make sure that whatever requirements have been communicated appropriately to

their off-shore counterparts”Quotation 14, Beta P0

Project Beta had people from the vendor’s team at the client’s side, where they

acted as a bridge between the business and the development team. However, not all

interaction had to necessarily be mediated through this coordinator bridge; it was possible,

for instance, for a developer at one location to directly contact a business analyst at another.

Every day there is a 15-minute Scrum at ‘off-shore’ (which is to say; at the devel-

oper’s location). Between on-site and off-shore, the same happens between the coordinators.

Later an intermediate Scrum will take place between the top persons from each team18.

There is reporting to the higher-level analysts on a weekly basis. Also, at the start of an

iteration, everyone gets together for planning. A similar meeting is held at the end of each

iteration for a retrospective.

The development work in project Gamma was supplemented by a small team of

developers from the client. This was based mainly on initiative from the client, which was

eager to be involved more closely in the product’s development. To facilitate coordination,

each developer from the client was paired with a developer from the vendor. Discussion

about the needs of the end-user were facilitated through corporate architects that acted as a

catalyst for new avenues of discussion18When compared to the distributed team models described by Sutherland et al. [37], this modality can be

placed in the ‘distributed Scrum of Scrums’ category.

52

5.4 Knowledge managementA learning process was involved with the knowledge management itself; participants

stated that it took some time for both the client and the developer to fully appreciate the

importance that domain knowledge would have in the project. This expressed itself in

‘startup problems’, as mentioned in quotation 15;

“Better knowledge on [BUSINESS DOMAIN] would have definitely improved the

requirement practice and prioritization process”Quotation 15, Alpha P2

A specific example in project Alpha is the data mapping process. As a part of data

migration, existing data was mapped to elements in the new system, and lack of domain

knowledge actually caused problems. Here, a greater familiarity with the client’s business

domain could have prevented issues. In project Beta, one participant expressed that the

client had not given much thought to the domain expertise of the contractor; rather, the

selection process was based on technical expertise and knowledge. There were some

issues as a result of this, and the participant believed that it was not fully realized that the

business and technical side needed to be considered together. Realization of this gap led to

comprehensive knowledge management efforts, which we will discuss in the course of this

section.

In the following subsections we will look at the way in which domain knowledge

was acquired and disseminated, and how the client was involved in this.

5.4.1 Domain knowledge acquisition

The organization in which our case studies took place had an internal division called

the domain competency group, which maintained domain knowledge of all the different

business domains in which the company was active. Practitioners in all projects could

draw on this source of information by recruiting team members with the necessary domain

experience from previous projects. However, not all necessary information was always

available (for instance; information on practices in a specific country), so other means

of domain knowledge acquisition were also necessary. Some of this knowledge was

gained from testing and feedback, but also from people who had established the initial

53

project backlog and were coming back into the project. The project lead for project Alpha

elaborated on the acquisition process as shown in quotation 16;

“The knowledge was gained predominantly from the testing area; we were doing the

testing, so that helped us develop our domain knowledge. And we also get people

who are senior from [DEVELOPER COMPANY] domain competency group, and

they have experience in the domain. Not specifically in [SUB-DOMAIN], but as

a wide practice. So they come up with, let’s say, 60 to 70% of the knowledge on

that domain. This is generic. So then they just need to deep dive and know what

[CLIENT COMPANY] specific domain knowledge is required, and it’s just a matter

of time before they can get used to it. They get through their... documentation process

map details, requirements, and as we progressed, the delivery story preparation, the

knowledge level has gone up, and they were quite happy.”Quotation 16, Alpha P1

Additionally, participant Alpha P5a described an extensive domain knowledge

acquisition process involving certification programs, knowledge sharing sessions, quizzes

and brainstorming sessions. All new members that were inducted into the project were

required to follow a comprehensive certification program before they could start. At

first, this caused some issues due to the lag this introduced for new resources to become

available; ideally, new people should have been recruited between 112 and 1 month before

they were actually required to allow for the domain training process. However, this did

not always happen at first. In spite of this extensive training program, some difficulties

remained. Participant Alpha P5b stated that still “some reverse-engineering had to be

done”, mainly because “[i]t’s too large an organization with too many processes”.

For project Alpha, most of the domain knowledge dissemination happened at the start

or during the induction process, as the project was centered around a single product with a

fixed scope and duration. In project Beta, domain knowledge dissemination processes were

started as required when work began on new products that required knowledge about that

particular aspect of the client’s business. In project Gamma, necessary domain knowledge

was mainly acquired by recruiting people with the required experience into the team

(possibly through the aforementioned domain competency group).

54

5.4.2 Client involvement

We observe clear differences in the amount of client involvement in the dissemination

of domain expertise. In project Alpha, the client actively participated, giving workshops

and training seminars, and providing readily available sources of information (the on-site

customers mentioned earlier). Practitioners in project Beta, on the other hand, were much

more on their own in acquiring the necessary information on the client’s business domain.

As mentioned before, the client here was mainly concerned with the technical expertise

of their development team, and not so much the vendor’s insight into the client’s internal

functioning and domain-related properties. This is described explicitly in quotation 17;

“. . . once we came into the project, we were given a very very brief overview of the

domain knowledge. It was... rather, very less. It was just basic information that we

could get. And from then on it was our responsibility to pick up and learn the domain

expertise. So... it was a bit of a challenge, because the view that the client has, to do

cost cutting, you have to box off the technical portion of it, and they don’t really seem

to be aware that technology as well as the business goes together.”Quotation 17, Beta P5

However, statements from other participants in project Beta suggest that this difficulty

was handled successfully by the business analysts in their interactions with the client, as

others reported that there were no difficulties with respect to uncertainties about the internal

functioning of the client’s organization.

For project Gamma, customer involvement in domain knowledge acquisition did not

happen according to a fixed plan, but as and when required they provided information to

the development team.

5.5 Elicitation of requirementsIn our case study projects, extensively defining requirements up-front, at the start

of projects, seemed to be typical. Especially for project Alpha, the vast majority of

requirements were established before the actual development process started. Although

this is not unexpected given the context (discussed further in chapter 6), it is certainly

a departure from ‘pure’ Agile practice. Requirements are captured in the form of user

55

stories and then further elaborated and operationalized (which is the subject of section 5.6).

Case Alpha’s project lead shared with us that their method for capturing requirements is

customized to support (i) capturing of domain-specific information, (ii) dependency, and

(iii) functional sequence. The first of these relates to our findings in section 5.4, while

the second will be further elaborated on in section 5.9. The third, functional sequence,

will play an important role in our discussion of prioritization in section 5.7. Functional

sequence was established by defining the client’s business processes in collaboration with

the client. Concretely, this was done in the form of UML process flow diagrams. The

separate elements of these process diagrams are then elaborated upon in the form of user

stories. What is subsequently done with these user stories is described in the next section.

5.6 Delivery StoriesDelivery stories are created by taking user stories, and turning them into a functional

specification, high-level design, and test scenarios.

The project lead for project Alpha described delivery stories in quotation 18;

“. . . delivery story is like a pre-development work for the Scrum”Quotation 18, Alpha P1

Delivery stories were introduced by the team of project Alpha and later adopted

by project Beta as an internal best practice. The concept provides large projects with

something more measurable and concrete than relatively less explicit user stories. Multiple

participants from projects Alpha and Beta stressed the importance of delivery stories. From

the interviews, as well as with the help of project documents made available to us, we have

pieced together the steps that go into the user story ! delivery story translation process.

First, user stories are acquired from the client by the vendor’s business analysts in

the fashion discussed in the previous section. This user story is then assigned an owner,

usually the person responsible on the client’s side. Individual user stories are placed in

context with each other in a process map, which is a way to explicate dependency and

functional sequence. There is a phase at the start of each iteration in which delivery

stories are prepared. At this time, user stories are taken up from the project backlog and

turned into delivery stories. Groups of user stories are associated with a generic feature.

56

A global complexity estimate (high, medium, or low) is applied to each feature. The

interviews indicate that this complexity metric is based on experience and gut feeling

rather than formal criteria, although participants did indicate that they would like to adopt

a more formal approach if one was available/applicable. Features are then mapped onto

subprocesses. These subprocesses become the eventual delivery stories as they are further

specified. For each of these subprocesses, the client defines business rules, together

with the acceptance criteria for each business rule. The vendor’s business analysts then

take up these business rules and acceptance criteria, and translate them into functional

specifications. At this point, the design team takes over and creates design specifications

from the functional specifications, while the testing team uses them to create test cases.

This process is summarized in figure 5.6.

Figure 5.6: Delivery story process

It should be noted that all the different elements depicted in figure 5.6 are mapped

onto one another through a system of numbering of the specific user story, business rule

and functional specification. This ensures traceability throughout the process, and enables

project management to maintain a high-level overview.

At the end of this process, the following properties are associated with each delivery

story;

• Number of use cases

• Required time for use case implementation in person days

57

• Number of GUI Screens

• Required time for GUI Screen implementation in person days

• Business rules and validations

– Number of business rules

– Number of validations

– Number of field validations

• Number of test cases

• Required time for test case preparation in person days

• Required time for data models in person days

• Total time required for sub-process implementation

These delivery stories are created in close collaboration with the client’s representa-

tives. After this process, requirements are concrete enough that they can be handed over to

the actual development team while minimizing uncertainty; formalizing and concretizing

user stories in this way helps bridge any gaps in understanding that may exist between

clients and developers.

While the use of delivery stories makes requirements more explicit, clarification

requests can still be raised, which was discussed in section 5.3.

5.7 Requirements prioritizationHaving discussed delivery stories, we can move on to requirements prioritization.

Here, the actual requirements that are being (re-)prioritized are the delivery stories19. At

the start of a project, however, user stories have not yet been translated into delivery stories.

Still, an initial prioritization has to take place when the requirements are entered into the

main project backlog. This is done mainly based on the high-level process flow, which is

also called the functional sequence. This was determined by looking at the process maps

we discussed in section 5.5, and with a very rough and high-level estimation of effort.19Project Gamma, again, is a special case that will be discussed separately as the need arises

58

Dependency also played a role here, as we will discuss in section 5.9, although this was

only the highest-level dependency that is apparent from the process flow diagrams. We will

first discuss our findings with regard to the prioritization process in general, after which we

will look at specific factors related to prioritization; triggers, considerations, classification,

and impact.

5.7.1 Prioritization process

Based on the functional sequence, requirements were roughly grouped and allocated

to the various sprints of the project. In project Alpha, this was the stage in which require-

ments were grouped into collections. For each sprint, requirements were taken up in the

order in which they appeared in the project backlog. Reprioritization at the level of the

backlog occurred occasionally due to requests from the client, new insight with regard to

dependency or a greater understanding of the client’s process flows.

Before each new sprint, user stories were taken from the project backlog and trans-

lated into delivery stories20. These delivery stories were then taken up in the actual

sprints. After the initial, high-level prioritization, most reprioritization happened at the

‘Scrum-level’, as participant Alpha P5a described it. The first order of business in a new

sprint was to look at the allocated delivery stories and to sequence them for that sprint.

This also helped the Scrum-team to better understand technical dependency between the

requirements, and plan the sprint accordingly.

Central to the entire prioritization process was the discussion between the business

perspective, and the technical perspective. These perspectives are usually represented,

respectively, by people from the client and the developer’s organization (although, when

necessary, vendor’s business analysts can represent the ‘business perspective’ in the

client’s absence). The final decision was almost always made collectively, rather than

being forced by either party. Usually this resulted in a compromise with which both

sides were satisfied. However, when necessary, the developer organization was willing

to make hard decisions, with which the business might not be happy, for the sake of

keeping the technical architecture consistent. Similarly, situations arose in which critical

but unforeseen functionality had to be demanded by the client. The discussion was usually20as described in 5.6

59

triggered by the client, but it did happen both ways; sometimes a developer would think

of an easier or more elegant way to do something, or best practices learned from other

projects would suggest a certain course of action. This was then explained to the client, and

proposed as a solution. Considerations that were taken into account in the prioritization

process are discussed further in subsection 5.7.3.

The discussion between the business and technical perspectives shows itself explicitly

in the dichotomy between, as one participant put it, doing things properly and doing

things quickly. Sometimes, to maintain Scrum velocity, ‘dirty fixes’ or work-arounds

were implemented that can lead to re-factoring at a later stage in the project. Pressing

circumstances such as legislative changes could make it necessary to implement these

work-arounds to get products released on time. In this tension between proper solution

and dirty fix, there is much discussion, as shown in quotation 19;

“It is very difficult to convince them [the client] to spend more money out of which

they don’t have a tangible benefit . . . It’s partly the development team’s responsibility

to make them understand, and then take a call on that”Quotation 19, Beta P5

This participant suspected that the current implementation of ‘dirty fixes’ was going

to come back and negatively impact the project in the future. This relates to the concept of

‘technical debt’ that we mentioned in subsection 5.2.2, and will be explored further when

we discuss impact in 5.7.5. Participant Beta P1 stated that when discussing prioritization of

tasks such as refactoring and testing, the expenditure of resources must be explained to the

client in terms of added value. In this way, the vendor can represent its side from a technical

perspective while at the same time taking the client’s perspective into consideration.

Project Alpha, with its fixed-price, fixed-term nature, had a more formalized change

process in place than is typical for Agile projects. Per sprint, there were only four hours

of allowable change in the requirements. The link between the context and the process is

explicitly mentioned in quotation 20;

60

“It’s the way the project is priced. For example if it’s a fixed project, you know, from

the part of the vendor, we’re not going to accept much changes, much reprioritizations.

Any kind of change in BRs [Business Rules] is going to be resisted. But if it’s not

a fixed term project, you can keep improving the process, we can keep accepting

requirements and we can keep developing that. So... I think that might be one of the

most important ones. Because if it is fixed price, it’s based on a set of requirements

which is frozen and received. Without much scope for deviation. From the side of

the vendor, it’s not going to be feasible to accept a lot of changes, accept a lot of

prioritizations”Quotation 20, Alpha P5b

In project Beta, the requirements change process was less tightly controlled and

more in accordance with ‘standard’ Agile practice. Still, reprioritizations raised by the

client mid-sprint were usually reserved for the next sprint; project management preferred

to let each current sprint achieve what it set out to do, rather than make significant changes

mid-sprint. Because sprints were relatively short, the impact of this waiting time was

minimized. The current requirements could be finished and the new ones taken up in a

matter of weeks. In Beta, most of the reprioritization is performed by representatives from

the client. Reprioritization typically happened at the level of the project backlog. This

did not involve a lot of discussion, as no effort has been expended yet. Reprioritization

on the lower levels could cause more issues, such as technical dependency (more on

this later). Intra-iteration reprioritization did occur, however. When it did, it was due to

business criticality or the necessity of quick reaction time. While for the overall project,

the business was responsible for the decisions, on the sprint-level, there was usually a joint

decision between the business and the development team. Teams were empowered to make

decisions at the sprint-level. However, delays sometimes occurred when decisions needed

to be taken at a higher level, which could be disruptive for the project. Participant Beta P4

estimated that intra-iteration reprioritization occurred rougly 25% of the time, although

this seemed to vary depending on the stage of the release or work package; reprioritization

was especially common in the first three months. The development team tried its best to

accommodate changes whenever possible, even if they came late in the iteration. This

61

flexible mentality is illustrated in quotation 21;

“Maybe it is not a compulsion that we have to, but we say that it is Agile”Quotation 21, Beta P1

Practitioners in project Gamma preferred to de-scope features in a sprint rather than

drop them to the next sprint. This relates to promises made to their immediate client, that

certain functionality would be available at a specified time. Should these features be moved

to a later sprint, the client’s own development effort would be slowed down. This was

because the client relied on these features becoming available for its own project planning.

Therefore, the client is provided with functionality with which it can ‘make due’ until the

full feature can be implemented in a later sprint. There is also an element of trust-building

involved, as explained by project Gamma’s project lead in quotation 22;

“Though this [feature] is not a critical development at this moment, we feel delivering

this release on that day when we have announced is important to create that credibility

and faith that this team can deliver”Quotation 22, Gamma P0

Changing priorities during a sprint was avoided. New features that came in during

a sprint were put on the backlog. It seems that, in this project, requirements did not

change mid-sprint, except through re-scoping triggered by the development team itself.

The amount of reprioritization (also of requirements still on the backlog) varied; from once

a week to not at all for a whole iteration. Compared to the other two projects, development

in Gamma seemed to be driven much more by the development team, and hence changes

in requirements initiated mainly from the development team. This is to say; ownership

of the project here was much more in the hands of the development team, and this had

consequences for the prioritization process. Still, when specific interests of the client were

involved, they were still allowed to drive the decision process. Quotation 23 illustrates this

point;

62

“We are working on a development framework, which is not related to any particular

business domain as such, so we are the final decision makers. But in the case of

our customer projects and so, definitely, if the client understands in a better way

the business value and the other impacts and such, in that case the client is the final

decision maker.”Quotation 23, Gamma P2

Because the software under development (SuD) was part of an overarching research

project into model-driven development, the research goals also played a role in which

features would be taken up for the product. As participant Gamma P2 described it, the

driving factors for taking up requirements were; (i) needs of the research team, (ii) desires

for extra features from the product team (the ‘client’ in this context), (iii) fixing any

defects that the product team came across as they use the SuD, and (iv) issues or desired

features that the development team itself comes across while developing proof-of-concept

applications using the SuD.

5.7.2 Factors triggering (re-)prioritization

Now that we have looked at the prioritization process itself, we can examine the

factors that trigger it. Our case studies show that the drivers for reprioritization are diverse,

and are different depending on the context. We will describe the common elements, and

also the differences we found between our cases.

In project Alpha, the main reason for changing priorities was clearly dependency21;

dependency between business processes, but also technical dependency. Most participants

from project Alpha confirmed that reprioritization events were mostly triggered by pre-

viously unknown dependencies. These technical dependencies were usually discovered

by the development team during the actual sprints, and are then communicated to the

client. This gave rise to some of the ‘technical- vs. business-perspective’ discussions

mentioned in 5.7.1. Participant Alpha P4 was active in a Stream development (see 5.2.1.1)

and mentioned ‘scope’ as the main reason why prioritization decisions were difficult.

Dependency was not mentioned at all. This is likely because Streams already consisted21See section 5.9 for more on dependency, specifically.

63

of tightly interdependent requirements that were grouped in such a way that they did not

have to be divided into two-week sprints. It should also be noted, of course, that the whole

concept of ‘Stream development’ was implemented to deal with tightly interdependent

functionality.

Dependency was also a concern in project Beta. Technical constraints and dependen-

cies played an important role, but change was most often triggered by the business analysts

or the regression testing team, and not so much the development team. It was mentioned

that an external trigger for re-prioritization was change requests from the client. If these

requests were urgent, other requirements needed to be reprioritized around this. One of the

ways development teams did trigger reprioritization is with technical issues put forward

during retrospectives at the end of sprints, but mostly reprioritization came from the client.

One of our participants described it as stated in quotation 24;

“. . . when the business has to re-think what they really want, also get a better idea of

what the estimates are, what the sizes are, they tend to kind of reprioritize also.”Quotation 24, Beta P5

Prioritization usually proceeded by the development team offering the client multiple

options, accompanied by the necessary explanation to make an informed decision. The

final decision was then made by the client.

Dependency seemed to be much less of an issue in project Gamma. None of our

interviewees in Gamma mentioned technical dependency as a consideration without being

specifically prompted, while it was central in project Alpha and explicitly mentioned on

multiple occasions in Beta. In Gamma, participants stated that feature complexity itself

could be a trigger for reprioritization; features could turn out to be more complex than

expected, which could necessitate a reshuffling of priorities. Alternatively, the feature

could be scoped-down. Scoping-down was usually the preferred solution, as discussed

previously. Reprioritization was usually not triggered by the actual development team.

However, when the developers did trigger reprioritization, it was due to changes in the way

in which requirements were understood, or due to technical constraints of the architecture.

Sometimes it was realized that there was a better way to implement a feature so it would be

more coherent with the overall architecture. This could cause reprioritization as a general

64

component is developed, based on which more specific modules would be dependent.

5.7.3 Prioritization criteria

The business- and technical perspectives were not the only considerations driving the

decision-making on priorities. Cost was discussed in meetings with clients, and sometimes

risk as well. Size was also discussed, but this can be seen in terms of ‘scope’ and its

influence on cost and risk, or the effort estimation from the developer. Estimates on cost

and size come from the development team, and risk from the business. However, although

they originated from the developers, the metrics of cost, size and risk are obviously mainly

important for the considerations of the client. For the developer, the primary consideration

aside from the technical architecture was the required effort. Furthermore, interviewees in

project Beta explicitly stated that business value, which we will return to in section 5.8,

played an important role in the client’s considerations when prioritization was discussed.

The client performed a cost-benefit analysis when assigning priority to items on the project

backlog.

In project Alpha, the situation was slightly different. Project Alpha’s project lead

stated that criteria taken into account for prioritization were criticality, schedule (with

respect to Scrum velocity) and downstream dependency. Still, the main consideration

is functional sequence, modified mainly by other forms of dependency. The distributed

nature of the project also played a role in prioritization; requirements were arranged in such

a way that they could be handled when the people involved were available, such as when

the relevant business person was on-location or not otherwise engaged. Maintainability

was also mentioned as a consideration, as the developer’s organization could be contracted

in the future to maintain the application. Because the project was fixed-price and fixed-

duration, ‘cost’ became a concern for the developer, and ‘risk’ becomes a shared concern

(though from different perspectives).

5.7.4 Classification of requirements

The projects we examined in our case study did not seem to have a highly formalized

system in place for the classification of requirements. Still, categorization was applied

and a relative ‘weight’ and ‘value’ was attached to individual requirements. At least in

65

project Alpha, on the highest level, requirements were grouped in MoSCoW categories;

Must have, Should have, Could have and Would have (in descending order of criticality).

Weight of individual requirements (that is, delivery stories) is then established based on

the number of business rules. Aside from these metrics, decisions on the value of priorities

were rather informal. According to participant Alpha P4, the desired value (more on this

in section 5.8) was only subjectively quantified. This is illustrated in quotation 25;

“It’s more of a subjective decision than a specific formalization.”Quotation 25, Alpha P4

One interviewee in project Beta expressed that their main considerations for priori-

tizing were (i) customer need, and (ii) the available ‘bandwidth to accommodate change’.

As discussed before, the goal in project Beta was to accommodate changes whenever

possible, so their main consideration when a change request came in was their ability to

oblige. Similar to project Alpha, the ‘weight’ or ‘value’ of individual requirements was

not formally calculated, as described in quotation 28;

“We do not have any... spreadsheet or formula to calculate the complexity, so it is

people driven.”Quotation 26, Beta P2

When we asked about methods for establishing the characteristics of requirements

in our questionnaire, we mentioned the planning game as an example. In reaction to this,

participant Beta P4 mentioned that something like this only makes sense if you have a

relatively inexperienced team and you want to have everyone’s view to arrive at some

sort of consensus for the planning. On the other hand, if you have experienced people,

estimates can be arrived at more directly. In fact, in the experience of this participant,

these estimates are usually more accurate than taking some sort of average from all people

involved, experienced and inexperienced alike. Interviewee Beta P5 similarly expressed

that experience is the main source when judging requirements, rather than any formal

method or explicit effort estimation. Business value (discussed specifically in 5.8) plays

an important role in the classification of requirements, as stated in quotation 27;

66

“Even in [DEVELOPER ORGANIZATION], what we are trying to do is... nowadays

we are not talking in terms of metrics. Earlier when we talked about metrics, we said

“we delivered 10 functionalities, without defects”. We are not trying to do it nowadays.

We are trying to [define] what we have delivered in terms of business value.”Quotation 27, Beta P3

Project Gamma’s project lead stated that, although no explicit criteria are currently

used for prioritization, they would be interested in using a more formal method if one was

available;

“We would definitely want to use some concrete method where everyone participates

and there’s a rationale.”Quotation 28, Gamma P0

However, when determining which requirements came first and which second, the

end-user perspective was the guiding factor. The fact remains, however, that prioritization

was very often done only on the basis of ‘gut feeling’.

Interviewee Gamma P1 was the most explicit on metrics and criteria. Cost was

discussed with the client, and was defined in concrete monetary terms. Size also played

a role in in the discussion, and was measured in properties of the source code such as

number of classes and lines of code. Sometimes broader categories were used, such as

‘small component’, ‘large component’, etc. Complexity was also considered, but it was not

mentioned how this complexity would be measured.

5.7.5 Down-stream impact of reprioritization

The reprioritization of requirements can have a significant impact on project activ-

ities. Because delivery stories needed to be translated from user stories, re-ordering of

requirements on the backlog could have consequences for the entire execution of a sprint.

Reprioritization events could even trigger other reprioritizations in otherwise unrelated

areas of the project due to dependency and mutual targets. Also, higher-level redefinition

of requirements could trigger re-prioritization ‘down-stream’ due to re-adjusted effort

estimation. One business analyst in project Alpha illustrates this in quotation 29;

67

“We are at the down-stream end. So once the basic process changes, you will have to

re-prioritize all your processes.”Quotation 29, Alpha P5a

In our questionnaire, we asked participants whether changing requirements had

ever impacted the system as a whole in a significant way, but this did not seem to be the

case. In project Alpha, one participant mentioned that it was too early in that project for

such impacts to have been observed. There was relatively little mention of the impact of

reprioritization in the other cases. We hypothesize that the impact of changing requirements

is mainly a factor in Alpha due to the highly interdependent nature of the requirements in

that project. We will return to this subject in the discussion (chapter 6).

5.8 Business valueAn important factor in the requirements engineering process for all projects was

business value. Participants defined business value in various ways, the most elaborate of

which is given in quotation 30;

“To me personally, it means... one thing is the time to market, and the benefit that

they’re going to get. And when I say benefit, it’s benefit over the competitors as well

as the benefit in ease of use, benefit in terms of improved efficiency and productivity,

quite a lot. So, all this means to me business value. It’s not just the money thing, it’s

also the satisfied end-users that say “Ok, previously I spent about half an hour doing

this and now it’s just a couple of minutes”. That’s also business value according to

me”Quotation 30, Alpha P5a

In our case study projects, better understanding of requirements is linked directly to

understanding desired business value, and the client’s related business goals. At the high

level, clients made an explicit link between desired business value and their business goals.

However, as requirements became further operationalized (i.e. after the user story !delivery story translation process), value of requirements is seen less in terms of business

goals and more in concrete technical terms. For instance, at the level of participant Alpha

68

P2’s involvement, business value was not explicitly connected to the client’s business goals.

At the initial discussion of any new set of requirements, clients communicate business

value in terms of the basic reasoning behind the requirements. Doing so, they discuss the

risk involved and what the dependencies are from the business perspective.

Although clients do not always directly discuss the value of requirements, they are

encouraged to do so by the developer’s business analysts. Business value was not typically

discussed by the client in absolute terms (“worth x amount”) but rather in relative terms;

relative to other requirements and the goals of the business. In project Gamma, business

value of requirements was expressed in terms of things useful for developers, as the SuD is

a tool for developers. These properties included usability, performance, maintainability

and functionality.

We observe a clear separation of concerns in all projects when it comes to business

value; we will return to this in the discussion.

5.9 DependencyAs touched upon previously, we observed significant differences between our case

study projects in the role played by dependencies. Practitioners in project Alpha most

elaborately discussed dependency and its impact on their project, and they identified several

different types of dependency. As we discussed when examining the organization of each

project, Alpha subdivided sprints into five separate domains. The highest-level grouping

of dependency thus becomes inter-domain and intra-domain dependency. Interdependence

between the domains is avoided whenever possible, but cannot be entirely prevented as

the system as a whole is based on the same architecture and also needs to form a single

coherent application. Another type is cross-functionality dependency, which arises when

the implementation of one feature is dependent on another. According to participant Alpha

P5a, when prioritizing, generally no consideration is given to cross-domain dependencies;

in this respect the domains operate in ‘silos’.

Although dependency was mentioned specifically by a number of participants in

project Beta, it was clearly less of a consideration than in project Alpha. This was partly

because the volume of requirements in Beta was simply much smaller (because they were

not all defined up-front). Also, the development effort was not so much concentrated on a

69

single, end-to-end development project, but rather concerned multiple different applications

progressing in releases. New requirements were also coming in frequently, so there was

relatively little incentive to map them out precisely beforehand. Thus, for Beta, dependency

played more of a role at the iteration level. Here, it definitely needed to be taken into

account when reprioritizing, as multiple interviewees confirmed.

None of the interviewees from project Gamma mentioned dependency between the

requirements without being specifically prompted. During the interview with Gamma’s

project lead, the point was made that the separate features of their product are largely

atomic, and are based on a well-defined and thorough architecture. With these characteris-

tics, it is not surprising that dependencies would play less of a role.

None of our case study projects seemed to use a formal way of mapping and keeping

track of requirements. On a high level, dependency was only expressed in the process flow

diagrams. In Alpha and Beta, low-level dependency tracking was done based on individual

delivery stories. There did not seem to be any central tracking mechanism in place such

as a dependency matrix. Rather, the definition of each delivery story included a column

which listed all other delivery stories on which it was dependent.

5.10 Project segmentationIn this section we will discuss examples of our observations with regard to the

way in which our case study projects were organized to cope with their size. In the

discussion (chapter 6) we will compare and contrast these findings with the existing body

of knowledge, and integrate them into an overarching model.

In project Alpha, we learned that on the high level, there is a general ‘design

guidelines document’ that all the teams use for keeping overall design consistent. Some

things in this document are ‘uplifted’ from the user stories. As our participant explained;

“[I]t was becoming a challenge; one user story was talking about the X Y Z and the

other one talks about the A B C D, and it is not in line... so some of these were uplifted

and agreed as a team between the business architecture and the technical architecture.

And it was published as a guiding principle.”Quotation 31, Alpha P1

70

Furthermore, when discussing priorities of requirements, we asked whether knowl-

edge of priorities was distributed to all project members. One participant responded as

shown in quotation 32;

“Not always, because when they wrote the business requirements... I think there was

an altogether separate team, I think they would have discussed it that time.”Quotation 32, Alpha P3

For another example; as we noted when we discussed the code frequency graph

(figure 4.3), the testing team is ‘down-stream’ from most other project activities. For

instance, the testing team does not usually interact with the customers directly; clarification

queries go through the design team or the business analysts. However, when these are not

available, the on-site customer can be approached directly. From these examples and our

other observations, it seems clear that project Alpha has a clear hierarchy and division of

roles and responsibilities in place.

Project Beta seems to be similarly segmented. Participant Beta P1 told us that the

explicit criteria used for requirements prioritization are discussed between the business

analyst and the business customer, and not necessarily propagated further ‘downstream’ in

the project. When we discussed requirement dependency, this same participant explicitly

mentioned a sprint-level, or development front where the effects of this are mainly felt,

while also distinguishing between customer front, analyst front in other answers. This

subdivision of tasks came forward most explicitly when the participant stated;

“[O]n the development front you will know only the specified area which you are

working on.”Quotation 33, Beta P1

This was confirmed by participant Beta P2, who shared with us that on the ‘Scrum

level’ there is not always direct feedback from the client, as this is handled at the ‘business

analyst level’. Perhaps most explicit on the segmentation of responsibilities was intervie-

wee Beta P4, when discussing responsibilities as related to the prioritization process. This

participant indicated that prioritization can happen at three levels. One at the highest level

of business; here, reprioritization mainly involves re-ordering of the items in the backlog.

71

This happens often, but mainly at the start of the project (presumably at the start of a

particular ‘work package’). The next level of prioritization is when user stories are broken

up and further defined in the form of delivery stories. Reprioritization at this level involves

taking up these stories when something goes ‘out of sprint’, i.e. fails to meet targets. Here,

reshuffling of stories occurs to maintain Scrum velocity and keep sprint targets. The third

level at which reprioritization takes place is within the sprint itself, when teams are actually

implementing the delivery stories. Change here is triggered by increased insight into the

client’s process flow, as well as technical considerations. These include dependency and

the system’s architecture. Participant Beta P4 also told us that reprioritization typically

happens at the highest level or strategic level, which does not involve much discussion

as no effort has been expended yet. Reprioritization on the lower levels can have more

consequences, for instance due to technical dependency.

Another example was encountered in our discussion about the extent to which

requirements raised by the on-site customers and other stakeholders are representative

of the client’s organization as a whole. Here, one interviewee mentioned that there were

definitely instances where this was not the case. This is an interesting statement, as this

view was not shared by most other project members. The discrepancy can be explained by

realizing that this participant is much further ‘up-stream’ in the project than most other

interviewees, and interacts directly with customers on a regular basis. Even here, though,

there seems to be a separation of responsibilities, as our next participant expressed in

quotations 34 and 35;

“[W]hen we say stakeholders, it might be the customers or the managers or the program

managers who have higher strategic priorities, so... we really didn’t have the visibility

towards how they achieve it”Quotation 34, Beta P5

“[A]t the program level I’m not sure if everyone knows what exactly is the business

value. But at the project level yes, they are aware of business value”Quotation 35, Beta P5

Aside from the specifics, these quotes show that there is a clear stratification or at

72

least segmentation in place in project Beta. Even project Gamma, although it is not as

large as projects Alpha or Beta, shows clear signs of separation of concerns. As Gamma’s

project lead expressed in quotation 36;

“Normally we shield our developer team from the end users”Quotation 36, Gamma P0

In Gamma, there is a separate consultancy team that acts as a buffer between the

developers and the user. Still, when things need to be clarified directly to the developer,

there is a three-way interaction between clients, consultants and developers. However,

there is presumably no direct, unmediated interaction between clients and developers.

The examples discussed in this section indicate that our case study projects have a

clear and explicit separation of concerns in place. This topic will be further elaborated in

the discussion chapter.

6. Discussion

For the purposes of this chapter, findings that we discussed in chapter 5 are structured and

summarized in tables 6.1 and 6.2 to form the background for our discussion. The colored

cells in the tables indicate whether those properties were observed for project Alpha, Beta,

and / or Gamma. The tables show that while some observations were universal to all three

cases, others occurred in only one or two of our case study projects. In this section we will

offer some hypotheses about the reasons behind this, by linking findings to the project’s

context. After this, we will have the necessary structure and context in place to present a

descriptive model of the requirements engineering process in our case study projects.

Table 6.1: Specificity of findings (1/2)

73

74

Table 6.2: Specificity of findings (2/2)

6.1 Formalization of RE processThere are certain consequences to using Agile methodologies in large projects.

Usually, in Agile projects, teams are arranged in a relatively ‘flat’ hierarchy, where there

is not a very strongly defined segmentation of responsibilities. In these teams, the same

people who gather the requirements also make the designs and code the actual software.

However, as we discussed, this type of arrangement is not feasible in large projects. One

should expect, therefore, that projects beyond a certain size have a clear division of roles

and responsibilities in place.

Contrary to what our study of the literature showed to be typical for Agile projects,

our case study projects had quite extensive processes in place for eliciting and (re-

75

)prioritizing requirements. One novel concept that was discovered with regard to the

implementation of Agile in our cases, was that of delivery stories, as described in section

5.6. Delivery stories provide the necessary translation step in a situation where developers

do not directly receive requirements from the clients. As one participant remarked in

quotation 37;

“. . . it is not easy to comprehend from the business rules and directly develop. There is

no proper flow sometimes, so, we formulated the delivery story idea”Quotation 37, Alpha P0

We hypothesize that the degree to which the RE process needs to be formalized in

an Agile project, increases with the size of the project. In typical, small Agile projects, the

process is hardly formalized at all, in accordance with the light-weight nature of Agile.

In these situations, there is less need for formal processes because the communication

lines are much shorter and the hierarchy much flatter. This allows clients and developers

to collaborate closely, obviating the need for a precisely-defined approach. The situation

is significantly different in very large projects such as Alpha and Beta that involve more

than a hundred people, necessitating the use of delivery stories to accurately communicate

the desires of the client to the actual developers. In this respect, the medium-sized project

Gamma represents a ‘missing link’ between these two extremes, and allows us to trace

some of the evolution of the delivery story concept. Project Gamma, although it did

not use delivery stories specifically, did formalize its requirements process to a certain

extent. Project Gamma has a consultancy team that receives requirements from the client

in the form user stories. The consultants then translate these short descriptions into more

concretely defined specifications that could be handed over to developers.

As we mentioned previously, the vendor organization is a highly structured, CMM

level 5 company. In their study of the relationship between organizational culture and

the deployment of Agile methods, Iivari and Iivari [73] state that organizations with a

hierarchical orientation may have difficulty adopting Agile practices. Indeed, in our cases,

we found that adjustments were necessary to accommodate Agile development. The

main driver for this was the client’s desire to use Agile, but it was also recognized as an

opportunity for the developer’s organization to gain valuable experience with Agile. This

76

was discussed by participant Beta P3 in quotation 38.

“We were flexible in organizational level processes, [. . . ] There were some mandatory

processes which might not be suitable for Agile at that time. So we tried to convince

[DEVELOPER ORGANIZATION] for that, we gave very high level importance

to [CLIENT ORGANIZATION]. But over a period of time, what happened was...

whatever we have done, our experience automatically paid back to [DEVELOPER

ORGANIZATION]. Because of our experience, we were able to define the processes

for Agile”Quotation 38, Beta P3

The idea that Agile can be adapted to support the ‘right amount’ of formalization

is echoed by Gary et al. [40] in their study of Agile for safety-critical systems. Other

authors have shown that Agile practices can, indeed, be successfully implemented in highly

mature organizations [74]. Formal user acceptance tests in Alpha and Beta helped to make

these project’s progress more explicit. Mapping out the client’s business in the form of

comprehensive business process maps allowed practitioners in project Alpha to gain the

necessary insight to essentially automate large parts of the client’s business. We observe

that our case study projects consciously undertook an effort to create an environment,

within the larger vendor organization, where Agile practices could be applied. Reporting

demands and other governance overhead was handled at the ‘interfaces’ by people with

specific roles, whose task it was to comply with corporate policy while still allowing the

project to be lean and flexible.

We note similarities between the high-level structuring of projects Alpha and Beta

(i.e., ‘domains’ and ‘releases’, respectively), and what Larman and Vodde described as

‘requirements areas’ [16].

Resistance to accepting new requirements in project Alpha was a direct consequence

of the fixed price, fixed duration nature of their engagement with the client. In Gamma, the

interaction with the client was only one of the factors driving development, and so played a

comparatively smaller role in determining the direction of the project. These projects also

had most requirements defined up-front, in contrast to project Beta. For Beta, the flexibility

with requirements can also be traced to their interaction with the client. Here, interaction

77

was more in accordance with common Agile practice, be it on a larger scale. Developers

from the vendor supplement an existing development effort on the client’s side. There was

no fixed end-date, or single monolithic product to be developed. Rather, developers worked

on multiple products over variable time periods. Developer allocation and requirements

here were constantly in flux. Therefore, extensively defining and formalizing requirements

up-front would have been inefficient, as priorities were likely to shift over short periods of

time.

The preference of project Gamma to de-scope existing requirements when new ones

come in due to reprioritization events can also be traced to that project’s context. Namely,

as we discussed in 5.7.1, its commitments to their client. De-scoping in this context is

to simplify a feature currently under development in such a way that time is freed up for

other work, but the core functionality of the feature is still delivered on time. This way, the

internal client is still provided with the necessary means to continue its own development

effort.

As we noted in section 5.3, the projects were still flexible in such a way that direct

communication could take place when it was necessary. It is possible, for instance,

for a developer at one location to directly contact a business analyst at another. Our

participants stated that this was not standard practice in the organization as a whole, but

was a consequence of the greater need for timely information in Agile development.

We also discussed (in 5.7.3) how a project’s context influences which considerations

are important to which party. We saw how, in a fixed-price project, the concern over

‘price’ shifts from the client to the developer; with the price already decided upon, every

additional unplanned investment is essentially a loss for the vendor. ‘Risk’, here, is also

more of a concern for the vendor, as there are explicit and immediate consequences to

inability to meet the agreed upon target 22. This can be expressed schematically as shown

in figures 6.1 and 6.2.

Although planning and estimation was mostly experience-based, our participants

indicated that they would like to use a more formal approach if one was found to be

applicable to their situation. This is similar to sentiments reported in a study by Grewal

and Maurer [36].22This is not to say that risk does not play a role for the vendor in the other projects, only that it comes

more explicitly to the foreground here.

78

Figure 6.1: Prioritization considerations: Project BetaConsiderations influencing the prioritization process from different perspectives.

Figure 6.2: Prioritization considerations - fixed price: Project AlphaConsiderations influencing the prioritization process from different perspectives in a

fixed-price, fixed-term project.

79

6.2 Organization of collaborationThe literature shows that “Communication and coordination are significantly harder

to manage when a project is distributed over multiple geographic sites” [75]. Other studies

show that distributed work items very quickly have a negative impact on development

performance [76]. At least some of this can be traced to distributed developers not being

aware of the roles and responsibilities of people at other locations. A distributed team can

get the sense that each location is a separate unit [76]. This can be related to the concept of

‘teamness’ that we find in literature on distributed collaboration, where the point is made

that close interaction increases productivity by making people who are not co-located

nevertheless feel like a single team [76, 77].

From our case studies we conclude that the adoption of Agile actually helped

‘teamness’; greater transparency and flexibility in roles meant that people found it easier

to collaborate directly. Agile’s emphasis on interaction and direct communication forced

both the client and the vendor to organize the project in such a way that this was not only

possible, but encouraged. This was illustrated most explicitly in quotations 6, 7 and 8 on

page 42, which clearly underscore the impact of Agile on the client-vendor relationship.

In our case study, the emphasis of the knowledge management efforts lay on the

acquisition and efficient dissemination of domain knowledge. Agile methods demand of the

developers a greater degree of familiarity with the client’s organization, due to constantly

shifting requirements and relative lack of documentation. In our case studies, we found

knowledge management to be of variable importance depending on the case’s context. It

was most explicitly discussed by participants from projects Alpha and Beta. Especially for

Alpha this should not be surprising as this project was developing an application for the

end-to-end automation of a large insurance provider; intimate knowledge of the client’s

business was deemed crucial for success, so management of this domain knowledge plays

an important role. Domain knowledge was also of significant importance for project Beta in

their development effort for their client’s in-house applications, although here, it was more

difficult to get the necessary client involvement. As multiple participants in Beta discussed,

the client was mainly concerned with the vendor’s technical know-how, and less with its

domain knowledge. Perhaps surprisingly, domain knowledge also played a role in project

Gamma. This in spite of the fact that there was not a great deal of interaction with the

80

eventual end-user, and the product being developed was a general-purpose model-driven

development environment. Project Gamma did in fact take domain-specific attributes into

consideration; their aim was to make their environment comprehensive to fulfill the needs

of a specific business domain, before moving on to others.

6.3 Descriptive modelIn this section we present the descriptive model that aims to capture our observations

of the way our embedded cases coped with requirements. This model is descriptive, in that

it offers a description of our observations in our case study projects.

On the highest level, the project’s activities can be divided into four separate cat-

egories; strategic, tactical, and operational levels, with domain knowledge supporting

activities on all levels. This is depicted in figure 6.3.

Figure 6.3: Descriptive model of project activities: overview

The choice for this high-level segmentation of activities comes from many repeated

observations, some of which we discussed in the results chapter’s section on project

segmentation, section 5.10. It comes directly from explicit statements from interviewees

concerning levels at which discussion takes place, decisions are made, and people interact.

There was also ample implicit evidence supporting such a segmentation. For instance,

when asking about business value, we observed that the further down-stream (and thus,

removed from the client) participants were, the less specific knowledge they had concerning

81

business value, while people further up-stream (and closer to the client) were more

extensive in their answers. Still, the link between the client’s business goals and the desired

business value was made explicit only to those project members who were involved in the

initial requirements gathering phase. Another example can be found in the contradictory

responses that we received from participants when asking about the extent to which the

client’s business processes had to be reverse-engineered. Most participants stated that this

was not the case at all, and that all necessary information was always readily available.

However, several participants explicitly stated that significant reverse-engineering did, in

fact, have to take place. This discrepancy can be explained by noting that this latter group

of people were the ones actually eliciting requirements from the client, and establishing

the top-level sequence. Any reverse-engineering that had to occur, thus took place at this

stage. People further ‘down-stream’ were not only unaffected by this, but were unaware

that it had been necessary in the first place.

Figure 6.4: Legend - Se-mantics

This segmentation of responsibilities, and thus of

project-related knowledge, we have termed ‘separation of

concerns’. The segmentation of project activities in strate-

gic, tactical and operational levels reflects our findings in

our case studies. In this section we will examine the re-

quirements engineering process, i.e. how our case study

projects coped with requirements, by discussing the relevant

factors and their positions in our model. For the diagrams

presented in this section, we will use semantics as defined

in the legend in figure 6.4.

Looking at the way the projects cope with requirements, the central concepts on

each level can be understood as depicted in figure 6.5.

At the strategic level, the requirements are acquired and high-level prioritization is

applied. At this stage, requirements enter the Master Project Backlog (MPB). This is the

main repository of requirements for the project. At this point in time, the requirements still

only exist in the form of ‘user stories’. From here, requirements are allocated to specific

segments of the project, such as the ‘domains’ in project Alpha, and from there on to

parallel collections, streams, or individual sprints. This is where we enter the tactical level;

82

Figure 6.5: Descriptive model of project activities: the requirements flow

this includes activities at the level of individual ‘domains’ in project Alpha, or of ‘releases’

in project Beta. At the tactical level, activities are concerned with managing at the inter-

sprint level. Tracing the requirements flow, we see that this is where the user stories are

turned into concrete specifications that have explicit metrics attached to them. For Alpha

and Beta, this was where user stories were turned into delivery stories, as we discussed in

results section 5.6. It is also where project Gamma’s requirements are made more formal

and explicit by that project’s consultancy team. These concretized requirements are then

placed in an ‘intermediate backlog’. From here, they can be taken up in the actual sprints,

where the activities take place at the operational level. This level is where the actual

development work takes place. Reprioritization here can cause requirements to be moved

back to the backlog for future sprints. Activities at all these levels are supported by a set of

activities related to capturing, maintaining and disseminating domain knowledge. Figure

6.6 shows our segmentation of project activities superimposed on the project timeline;

here, we see that the ‘strategic level’ is also concerned with the project’s planning phase

and the necessary implementation and change management activities. It should be noted

that other long-term considerations also play a role, such as the possibility of having to

maintain the application in the future, or retaining the client for new projects. In this

figure we also see that multiple instances of activities at the ‘tactical’ level can be running

83

concurrently. Examples of this are the separate collections and streams in project Alpha,

and the different releases in project Beta.

Figure 6.6: Separation of concerns as related to the project timeline

Figure 6.7: Legend - Cate-gory colors

Having discussed the main requirements flow, and

how this categorization of activities can be understood in

relationship to the project timeline, we can now take a

detailed look at each level in turn. In our discussion of

the individual levels, the items shown in bold on a blue

background will be the components of the ‘requirements

flow’ from figure 6.5. Categorization of elements in the

model will be represented by color as defined in figure 6.7.

6.3.1 Strategic level

As we stated, requirements in our cases were gathered

in the form of user stories; “as a [role], I need to be able

to [action]”. These requirements are determined by the

project’s business analysts or consultants, in collaboration with client representatives or

stakeholders. This group of people also creates the process flow diagrams. After being

gathered, high-level prioritization is applied to the requirements. At this level, priorities

are derived from the high level sequence, as explicated in process flow diagrams, and from

84

business value. The determination of business value arises out of a discussion between the

client, reasoning about this in terms of business goals, and the vendor’s business analysts

or consultants. This is where the metric of ‘cost’ plays a role. It is in the order determined

by business value and high-level sequence that priorities enter the MPB. Here, risk and

initial effort estimation is applied to the requirements. As we discussed previously, it

depends on the context whether ‘risk’ is primarily a concern for the client, or also for the

vendor. At this stage, the vendor’s analysts / consultants also apply a very high-level effort

estimation to the requirements (this should be seen as categorizations along the lines of

‘easy’, ‘medium’ or ‘hard’). These requirements on the MPB are then allocated to different

areas of the project, depending on the project’s high-level segmentation. In the case of

project Alpha, this is where requirements are allocated to specific domains. Weekly project

status meetings are held between the senior business analysts. These meetings can result

in a change or re-ordering of requirements on the MPB, or a reallocation of requirements

to different segments of the project. This part of the model is depicted in figure 6.8.

Figure 6.8: Descriptive model - strategic level

85

Figure 6.9: Descriptive model - tactical level

6.3.2 Tactical level

At this level, requirements are transformed into low-level specifications. As we

discussed, this means the user story ! delivery story translation process for projects

Alpha and Beta, but a concretization process also takes place in project Gamma. Part of

the detailing process is also to apply concrete effort estimation to the requirements. It

is here that we see the ‘business perspective $ technical perspective’ discussion taking

place. Here, developers and / or designers give input on the technical side, and the client

or the vendor’s business analysts provide the business perspective. There is a separate

Quality Assurance (QA) process to ensure correctness, sufficiency, and consistency of

requirements.

From these more concrete requirements, it is now possible to derive a detailed

process map; this is a more explicit and detailed version of the process flow diagrams

that we saw at the strategic level. This process map details how each component in the

system connects to each other component in a functional sense. Thus, the process maps

are the main input for determining functional dependency. Based on this dependency,

requirements are placed in an ‘intermediate backlog’ (IB) of detailed requirements, on

86

which client representatives sign off. It is from this collection that requirements are selected

for the individual sprints. For project Alpha, this is the stage at which the determination is

made which requirements go into which sprints in a collection, or which go into a stream.

Priority of requirements on this level is influenced by inter-iteration reprioritization

events. This can be due to new understanding of functional dependency, or from require-

ments moving back to the backlog due to events at the operational level (which we will

discuss in the next subsection). Daily team meetings, between the top persons from each

of the development teams, influence the order of requirements on the IB, as well as the

selection of requirements for (future) sprints. Processes at this level are schematically

represented in figure 6.9.

6.3.3 Operational level

Figure 6.10: Descriptive model - operational level

At the operational level, requirements are actually translated into code. The transla-

tion of the relatively unspecific user stories into low-level specifications allows developers

at this level to concentrate on the development activities themselves. Still, this does not

mean that this is always a straight-forward process. Technical dependency, that is, depen-

dency between actual functionality or modules in the code, is often only discovered at

this stage. This can lead to intra-iteration reprioritization. Thus, although this is avoided

87

whenever possible, requirements sometimes need to be changed during the actual sprints.

While dependency is one possible cause for this, it is not the only one.

Although the concretized requirements aim to provide developers with a comprehen-

sive picture of what is required, clarification requests still need to be raised occasionally.

Insight gained from these requests can thus also trigger reprioritization at this level. In

accordance with Agile development, frequent interactions with the client create a feedback

process that is close to the development cycle, so developers can change course as and

when required. There are daily meetings between all members of a team that took the

form of ‘daily stand-ups’ in our cases. Here, concerns are raised that can be discussed

with the client if necessary. However, for all these processes (clarification, feedback and

daily meetings), interaction with the client’s business representative is usually mediated by

business analysts. While it is possible, in principle, for individual developers to contact

people from the client side directly, this seems to be the exception rather than the rule in

our case study projects.

As with the tactical level, there is a QA process at the operational level that oversees

the development process and makes sure that it lives up to corporate standards. As

components are developed, they are tested by a separate team. This team can then provide

feedback to the developers, or play a role in the allocation of requirements to future sprints

(as we will discuss later). The process that we discussed in this subsection is depicted in

figure 6.10.

6.3.4 Domain knowledge support

Activities on all the aforementioned levels are supported by domain knowledge.

Efforts in this regard were very extensive in project Alpha, and also played an important

role in project Beta. We have seen that even in project Gamma, with its internal customer

and general-purpose product, domain knowledge was important. Domain knowledge was

gathered and disseminated through domain experts, training seminars and certification

programs. In project Alpha, certification was part of the induction process for new project

members. These activities were supported by the organization’s domain competency group

on the one hand, and client involvement on the other, although client involvement does

vary from project to project, as we discussed. These processes are depicted in figure 6.11.

88

Figure 6.11: Descriptive model - Domain knowledge support

6.3.5 The full picture

In figure 6.12, we present the full descriptive model composed of the strategic (figure

6.8), tactical (figure 6.9) and operational (figure 6.10) levels, and the domain knowledge

support (figure 6.11), discussed previously. As we examine the model as a whole, a few new

connections become visible, as well as another process. We see that the ‘testing’ process

at the operational level can have an impact on the selection of requirements for iterations

on the tactical level. This is because, if developed software is determined to be defective

or incomplete, it will have to be taken up again in a future sprint. We see furthermore that

technical dependency on the operational level can have consequences for inter-iteration

reprioritization as well. The reason for this is that technical dependency can be of such

a nature that it cannot be resolved in the sprint itself, and can cause reprioritization at a

higher level, which necessitates rearranging of sprint planning to accommodate this. The

new process that we see is ‘evolving business needs’ of the client. This is an external

factor that can potentially affect each level of the project. On the strategic level, changes

in the client’s organization can cause practitioners to reconsider the order of requirements

in the MPB. On the tactical and operational levels, it can cause reprioritization at either the

inter-iteration or intra-iteration level, respectively. In this last case, requirements currently

under development might need to be dropped to respond to urgent requests from the client,

for example in response to legislative changes.

With our descriptive model, we have composed a comprehensive picture of how our

case study projects were organized to cope with requirements. It shows how vendors can

scale up in such a way that they can accommodate very large projects, while still retaining

some of the core principles of Agile development, and reaping the benefits thereof.

89

Figu

re6.

12:

Con

cept

ualm

odel

7. Validity

We evaluated the possible threats to validity of the observations and conclusions in a case

study [56]. For this purpose, we used the checklist for case study researchers recommended

in [78]. The issues of internal and external validity will be discussed separately in this

section. Additionally, we will evaluate the validity of our model presented in 6.3, as

described by GT methodologists [55, 79, 80, 81].

7.1 Internal validityWe acknowledge the inherent weakness of interview techniques that they are driven

by the researcher, meaning that there is always a residual threat to the accuracy of what

interviewees say. However, we believe that in our study, this threat was reduced, because

each interview conversation was completely transcribed and every transcript is available

for reference purposes.

Moreover, a validity concern in interview-based studies is that the researcher influ-

ences the interviewee. To counter this threat, special attention was paid to conducting the

in-person interview in such a way that the interviewee felt comfortable in answering the

question. The researcher who interviewed the participants made sure the interviewee is not

avoiding the question and feels at ease. This created a safer environment and increased the

chances to get a truthful answer rather than one aimed to please the interviewer. Further-

more, ‘script’ was added to the questionnaire as a way to maintain consistency across the

interviews, so the researcher did not inadvertently make certain statements only to some

interviewees and not to others.

To minimize potential bias of the researcher, we also considered construct validity

of our study. We followed Yin’s [56] recommendations in this respect, by establishing a

chain of evidence. First, we included participants with diverse roles in the same project,

and this allowed the same phenomenon to be evaluated from diverse perspectives (data

triangulation). Multiple participants in a variety of roles were interviewed separately and

asked the same questions. If these separate participants then consistently report the same

facts (from different perspectives), a stronger case can be made for the validity of the data.

90

91

Furthermore, the outcome of the coding process was internally verified for consistency by

an experienced researcher (Daneva) applying her own coding to the interview transcripts.

Although using different terminology, these two independent coding processes identified

roughly the same concepts. This is strong support for the assertion that coding was valid

as well as comprehensive. Furthermore, as per Yin’s second recommendation [56], draft

versions of this work were reviewed and verified by a senior person in the case study

organization. Furthermore, case study participants verified details with respect to project

organization and, for instance, the user story to delivery story translation process.

Internal validity is furthermore supported through our ability to verify data from

our interviews with facts reported in project documents that were made available to us

(providing the ‘multiple sources of evidence’ from Yin’s third recommendation [56]).

These documents serve as a secondary dataset to triangulate certain statements from the

interviews. Information that we have access to through these documents includes;

• User story templates,

• Delivery story templates,

• Examples of fully documented user- and delivery stories,

• Sprint task lists,

• Burn-down charts (for multiple sprints),

• Lists of internal queries raised,

• Logs of development hours,

• Lists of technical impediments encountered,

• Retrospective reports,

• Documentation of clarification processes.

92

7.2 External validityBecause we planned and executed exploratory qualitative research, the key question

to address when evaluating the validity of its results, is [56]: to what extent can the

practitioners’ experiences and the requirements prioritization mechanisms we observed, be

considered representative for a broader range of companies? We think that the conclusions

we draw will hold for other companies in contexts similar to that of our case study

organization, namely company size, project size, presence of process excellence units,

and maturity level. In higher maturity companies, it is intuitive to assume that there are

teams (similar to our case study organizations Program Management Office and Process

Excellence Team) that scrutinize the application of Agile practices and assess the fit to the

established systems delivery framework. We also think that there are many companies

(especially in Europe and North America) who may not be CMM-certified, and yet may

have established an effectively functioning project management office. The formation of

these teams is, by and large, market-driven as (i) many clients increasingly prefer that

vendors use Agile practices, and (ii) vendors are expected to possess and demonstrate

competence in “Agile”. While clients are well-versed in Agile practices and the benefits

thereof, they might not necessarily be involved first-hand in adapting the original Agile

Manifesto guidelines [12] to large distributed (and possibly multicultural) project settings.

More often than not, they expect the vendors to take care of these issues [43], while

insisting that the system be delivered using Agile practices. In the light of this market

development, we assume that organizations that have already formed internal program

management/process excellence units are most likely to find our conclusions relevant and

useful.

External validity is strongly supported by the fact that we conducted three separate

embedded case studies. While the projects we studied were part of the same organization,

they operated largely independently of each other. Therefore, a better case can be made

for generalizability of our findings than if they had been derived from only a single case.

The projects have certain features in common, while they differ in other aspects 23. This

gives us the opportunity to (i) trace characteristics of the project to its context, and (ii)

determine which attributes are universal rather than dependent on the setting. Linking23See chapter 2 for a general overview and the first part of chapter 6 for an in-depth discussion

93

findings to project setting allows us to make a case for generalizability to projects with a

similar setting.

The choice of the projects in the study could represent a threat to the validity of

the results in a number of respects: (i) The choice of the projects was not motivated by

any other criteria except that they all needed to be Agile. The authors relied on their

professional and personal network to establish contacts with the project members. (ii) The

studied projects are not representative for all the possible ways in which prioritization is

performed in large Agile organizations. We, however, consider that our findings can be

observed in companies and projects that have similar contexts to those included in our

study.

7.3 Model validityIn section 6.3 of the discussion chapter, we proposed a descriptive model based on

the results of our case studies. Traceability between the elements in this model and the

source data was maintained as discussed in the methodology chapter. In the results chapter,

the actual data is kept side-by-side with the resulting narrative, as depicted in appendix E.

The elements of our model, which derive from these findings, can thus quickly be linked

back to the source data. Still (as also described in [82]) GT methodologists [55, 79] state

that the model should not be validated against the data, but according to three evaluation

criteria [80, 81]:

1. Adequacy

2. Fitness (or relevance)

3. Modifiability

Ensuring the first criterion, adequacy, was done by closely following the GT process

and extensively documenting this process for traceability. Full transcripts of all interviews

are available for verification, codes were applied to the transcripts, and our online analysis

tool allows us to quickly perform evaluation. Furthermore, as discussed in section 7.1,

coding was applied by multiple researchers independently to ensure concept validity.

94

The second criterion, fitness, or relevance for both researchers and practitioners,

was ensured by choosing a level of abstraction that would make the model more widely

applicable, and useful beyond the context of our case studies. We abstracted away from

the specific implementation of processes in our case study projects, such as going from

‘user story ! delivery story’ to ‘requirements translation into low-level specification‘,

and speaking of ‘iterations’ in general rather than ‘Scrum sprints’ specifically. Doing so,

we make the model more broadly applicable. This was also ensured by linking specific

findings to elements in the context, as described in the first part of chapter 6.

Concerning modifiability, the third criterion, we make the point that our model is

descriptive rather than prescriptive. Verification can be performed by comparing it to

situations in other projects in similar context. Also due to the aforementioned process

of abstraction from our particular cases, the model is more easily comparable to other

settings. It also allows us to link our findings to concepts discussed by other researchers in

the literature.

8. Conclusions

At the start of our research project, we formulated our areas of interest in the form of four

questions: (i) Who are the decision makers in the prioritization process? Which roles are

involved and what are they responsible for? (ii) What criteria do large project teams use to

make risk-and-value driven decisions during Agile prioritization in an outsourced mode?

(iii) What is the relationship between project settings and requirements prioritization? (iv)

How does the vendor’s team combine value creation for their own organization with value

creation for their client? These questions formed the foundation for our questionnaire, and

thus the data that we collected. In this chapter, we will discuss the conclusions that we can

draw from this data with regard to the ways in which our case study projects ‘coped with

requirements’.

First we will discuss our conclusions with regard to the applicability of Agile, and

the considerations that should be taken into account for this. Next, we describe the project

management constructs that our case study projects had in place for effectively using Agile

in their setting. The last group of conclusions discusses project organization and the way

in which that organization was adapted to cope with size, distribution, and each project’s

unique context. Following this, we take a step back and consider the implications of our

conclusions as a whole.

8.1 Application of Agile• Based on our findings, we conclude that Agile can not only be applied successfully in

a distributed setting, but can be greatly beneficial. Agile promoted communicationin distributed setting. Greater transparency, close collaboration and flexibility in

roles improved project performance and promoted ‘teamness’. This reduced barriers

between vendor and client, and allowed close integration of efforts.

• In the view of our case study participants, the main advantage of the use of Agile was

increased speed of development, and was the main reason our case study projects

opted for this type of methodology. Our case study participants also mentioned

increased developer motivation. Furthermore, customer satisfaction was increased

95

96

because close collaboration and rapid feedback ensured that developed products

conformed to client expectations. In all cases, use of Agile was triggered by the

client. This is an important factor to consider, customer involvement being an crucial

for successful implementation of Agile practices. For all parties, the process as a

whole was more transparent, giving the client greater control over the direction of

the project.

• There are trade-offs to using Agile that should be considered. Our participants

described that a learning process was necessary for all parties involved, client and

vendor alike. Project members were not typically familiar with Agile development,

and even less so for the specific hybrid version of Agile used in a particular project.

Similarly, client representatives needed to acquire a working knowledge of Agile

processes to effectively carry out their role. This led to a more chaotic period at the

start of projects, where all parties needed to become accustomed to the methodology.

The necessary learning process also led to a longer induction-time for new project

members, which can cause issues when people are needed in the short term, but

become available later than when they were actually required.

8.2 Agile project management constructs• Modeling of requirements into business process diagrams in the early stages of

elicitation as pre-development work is an effective way to establish the project’s

high-level structure and flow.

• The user story ! delivery story translation process provides large projects with a

way to elicit requirements from the client in an Agile fashion, while nevertheless

being able to provide developers, who are further ‘down-stream’, with a low-level

specification that can be more directly translated into software.

• The concept of ‘Streams’ was used by project Alpha as an effective way to accom-

modate large work packages, or large monolithic features. These streams, running

in parallel with multiple sprints, address an issue with ‘standard’ Agile development

where some work cannot realistically be subdivided into short sprints.

97

• We observed that which prioritization considerations play a role for which party

depends on the type of contract. Typically, ‘cost’ and ‘risk’ are concerns mainly for

the client. However, we have seen that for fixed-price, fixed-duration projects, ‘cost’

becomes a concern primarily for the vendor, while ‘risk’ becomes a concern shared

by both the client and the vendor.

• In all our case study projects, we observed the use of experience-based effortestimation, rather than a more formal, metric-based approach. This necessitates the

availability and involvement of experienced developers and business analysts.

8.3 Project organization• An effective way to accommodate large project size using a ‘Scrum of Scrums’

approach was implemented in projects Alpha and Beta. Here, there were daily

stand-ups with all members of the separate Scrum teams, followed by a meeting

between the top persons from each team. This method ensured consistency and

provided project management with a comprehensive view of project status.

• Projects Alpha and Beta had dedicated project roles in place, that we can link back

to the literature. An example is that of requirements architects eliciting requirements

from the client, as also described by Gary et al. [40]. Furthermore, the role of

‘business area owner’, described by Grewal and Maurer [36], manifested itself in the

roles of product owner, domain owner and collection manager, all of whom were

representatives from the client.

• In our case study projects, being able to rely upon client representatives was

critical. It was important for these representatives to be knowledgeable in the area

of question, authorized to make decisions, and accurately represent the interests of

the client organization as a whole. In the early stages, there were issues in projects

Alpha and Beta where decisions by representatives would be overruled later by other

client stakeholders, or clarification processes within the client organization slowed

down the project significantly.

• Our case study project applied a separation of concerns with regard to project

98

activities as a way to cope with size. Using this method, it was not necessary for

all members of the project to be aware of all aspects of the project, as this would be

unfeasible in projects of this size.

• Correctly managing dependencies was vital for delivering tightly interdependent

functionality on a timely basis. We observed multiple levels of dependency. At the

top level, requirements are ordered in accordance with the high-level process flow. As

requirements are concretized, a detailed process map allows for ordering according

to functional dependency. At the third level, technical dependency between the

actual software components dictates the flow of the development work.

• Multiple levels of (re-)prioritization allowed project managers, as well as the

client, to maintain control over the direction of the project, in recognition of the

consequences of Agile on the prioritization process. On the highest level, prioriti-

zation occurred mainly as new requirements were established, and reprioritization

resulted from an evolving understanding of business value, as well as changing client

business goals. The next level is inter-iteration (re-)prioritization, occurring between

the actual sprints themselves. This is triggered by changing desires from the client,

functional dependency, and technical insights from the development work in the

sprints. On the third level, Intra-iteration (re-)prioritization occurs within the sprints.

Although this is generally avoided, it can be triggered by newly discovered technical

dependencies, as well as requests or clarifications from the client.

• Domain knowledge supports all project activities, and was deemed crucial for

project success. Through training seminars, certification programs and advice from

domain experts, the development effort was jointly supported by representatives

from the vendor, and the client’s domain competency group.

• With our descriptive model, we captured our observations in our case study projects.

With it, we show the processes involved in coping with the requirements, and how

these processes interact with each other.

99

8.4 OverviewOverall, one of the most important, and perhaps surprising, conclusions from our

work is that it is possible to successfully integrate Agile practices into a large, highly

structured organization, while still realizing the benefits that come with maturity. Our case

study organization realizes the benefits of having an infrastructure of processes, people

and tools in place, while injecting Agile practices into its project management. The result

is the project organization that we described with the model presented in section 6.3. The

benefits of practices such as daily stand-ups, short iterations, rapid feedback and focus

on technical excellence expressed themselves in high customer satisfaction, increased

turn-around time and greater developer motivation. At the same time, development efforts

are supported by the artifacts of maturity, such as the domain competency group, program

management experience, ARIS process flow diagrams, etc., instead of being encumbered

by them.

Furthermore, we see the importance of client involvement; In all three case study

projects, choosing Agile was instigated by the client, so convincing the client to commit

the necessary resources was not an issue. When considering client involvement in the

acquisition of domain knowledge, however, we see more differences. In Alpha, the

client was very committed in this respect, and participated actively in the extensive

domain knowledge efforts. In project Beta, however, more of a ‘pull’ was necessary from

the vendor side to extract the necessary information from the client. Here, the client

was predominantly concerned with the contractor’s technical expertise, and did not - at

first - consider the importance of domain knowledge. Participants stated that this had a

measurable negative impact on that project. For our consideration, it further underscores

the importance of client involvement, also in the area of domain knowledge support.

As we saw in the discussion in chapter 6, our projects had elements in common while

differing in other aspects. Many of these differences were traced to the project’s context.

An example is project Alpha’s formal change management process, and the fixed-price,

fixed-duration nature of its engagement with the client. Projects Beta and Gamma, having

no fixed deadline, accommodated changing requirements whenever possible.

A further example of differences due to context can be seen when we examine

distribution, and its effect on collaboration. In Beta, not only were client representatives

100

distributed, but the project centered around supplementing a team of developers at the

client’s location. As such, close interaction was necessary, and the project was organized

to accommodate this. The daily stand-ups between the project leads now also included a

discussion between the top persons at ‘on-site’ and ‘off-shore’. Furthermore, distributed

collaboration was comprehensively facilitated with desktop video conferencing and demo-

ing software. In project Gamma, where a small team from the client contributed to the

development effort, each remote project member was paired with someone at the primary

location.

Another example is project Gamma’s preference to de-scope functionality (that is,

reduce its intended feature-set) rather than drop it from the sprint completely in the event

of reprioritization. Here, the reason can be traced to commitments made to their client to

deliver functionality at specific times, so the client could use this in its own development

effort.

We also made a link between Alpha and Beta’s size, and the extent to which these

projects formalized their requirements specification process. We hypothesized that project

Gamma, which is medium in size, presents a ‘missing link’ in the way it formalized its

requirements; less than Alpha and Beta, but more than in a ‘typical’ Agile project.

9. Reflections

In separate sections in this chapter, we will highlight our contributions, offer suggestions

for future research, describe our learnings from a methodological point of view, and discuss

implications for practice.

9.1 ContributionsIn section 6.3, we presented our model that shows how our case study projects coped

with requirements. We note again that our model is descriptive, in that it describes our

observations. For practitioners, this offers a description of how a project in this context

could be undertaken. Follow-up studies and broader comparison are necessary to make

the model prescriptive, that is, a formal directive as to how a project of its kind should be

undertaken. Nevertheless, in its current state, our model is useful for practitioners as a

suggested course of action, point of comparison, or as a new way to understand processes

that are already in place. Furthermore, it provides a novel perspective on the shape that

Agile can take in large projects.

Our research project as a whole provides evidence for the applicability of Agile

outside of its usual territory of small and co-located projects. Furthermore, we have

shown how Agile can be successfully applied within a highly formal, highly structured

organization. Our work shows how an approach can be scale up through separation of

concerns, while retaining the important benefits that Agile offers. Furthermore, we have

described ways in which collaboration can be organized and facilitated. We discussed

challenges that could be faced with this approach, and how they can be overcome.

Furthermore, the concept of ‘delivery stories’ for formalizing and concretizing user

stories is novel. Aside from the essential role it plays in the scaling of Agile practices

that we have described, it could also be an interesting and potentially highly useful

tool for methodologists to explore further. Similarly, the concept of ‘streams’ proved

effective in accommodating large or monolithic work packages within the context of Agile

development, and can be of use to practitioners coping with similar challenges.

101

102

9.2 Future researchThe analysis of the dataset that we present in this thesis can be said to be a ‘forrest

from the trees’ analysis, where we attempted to gain a comprehensive understanding of

our dataset as a whole, through the lenses of our areas of interest. Any one of these

separate areas could be an interesting avenue for future research. The separate processes

in our model could each be explored in more detail, and the exact nature of the links

between them could be further explicated. Verification studies could be conducted, or

generalizability could be evaluated with further case studies in other organizations. The

wider applicability of ‘delivery stories’, development ‘streams’ concurrent to collections

of sprints, as well as a deeper investigation into the concept of ‘technical debt’, would be

interesting for research as well as practice. Furthermore, a cultural angle could be explored

by considering the effects of national and corporate cultures on project organization.

9.3 Lessons learned for researchersIn the course of this research project, we had a number of learnings that are potentially

useful to other researchers using similar methodologies in a similar situation. In this section

we will share these learnings in the hope that they may be of assistance in future research.

• We used explicit script in our questionnaire to ensure consistency across the

interviews. This guaranteed that we did not inadvertently influence answers from

participants by making certain statements to some interviewees and not to others.

• An abstract questionnaire was distributed to potential interviewees, which im-

proved willingness to participate. This abstract questionnaire contained sample

questions from each section. This provided participants with a sense of the questions

that would be asked, without diminishing the spontaneity of the answers to most of

the questions in the actual interview.

• Although it is not usually recommended by GT methodologists, we found recordingthe interviews to be vital for capturing data. There were many instances where

concepts only emerged when re-listening to the interviews, or examining the inter-

view transcripts. Having recordings of the actual interviews themselves on file also

103

allowed us to ground our theory more firmly in observation than if we had relied

solely on notes made during or after the interviews. Recording furthermore enabled

us to focus on guiding the interviews and asking follow-up questions, rather than

taking notes.

• Although a very labor-intensive process, we found manual transcription & codingessential for understanding the data. Especially for a dataset as large as ours, the

process of transcribing all interviews and then applying the coding by re-reading

all transcripts line-by-line required much time and effort. However, it did ensure a

greater degree of familiarity with the data than we would have had if we had used

alternatives, such as transcription services or automated coding.

• Although the interviews were manually transcribed and coded, we did use toolsupport to our advantage. With our online analysis tool, we could (i) make the most

recent versions of interview transcripts available to distributed researchers, (ii) see

which codes were applied for each interview, (iii) perform a cross-section analysis by

viewing selected coded fragments across all interviews, (iv) use hyperlinks to specific

lines of interviews to maintain traceability, and (v) apply comments to individual

lines of interviews, which could then be shared with other researchers. Once coding

was applied to the interview transcripts, the online tool was used extensively in the

theory building exercise.

• During the coding process, extensive memos were kept. We found that structuringmemos in a mind-map provided us with a flexible way of structuring and re-

structuring our memos as our theory building exercise and evolving understanding

progressed.

• When identifying the core concept in our dataset, we found it was vital to brain-storm with multiple researchers. After deliberating individually, each researcher

proposed a candidate for core concept in a meeting, after which we discussed suit-

ability of each of these concepts. Using the core concept criteria discussed by Hoda

et al. [61], we started a process of elimination. Using a ‘why-how tree’ allowed us

to put these concepts in context with our memo categories, and eventually arrive at a

suitable core concept.

104

• To strengthen the validity of our GT process, we maintained traceability betweennarrative and source data at all times. This was done by keeping the source data

in the form of comments in the thesis source document, side-by-side to the narrative

that arose from it.

9.4 Implications for practiceBy linking findings to their project’s context, we have explicated the connection

between circumstances and the resulting practices that were implemented. Therefore,

practitioners can consider these in detail for their particular setting. However, in this

section, we present findings that we hope can be useful for practitioners in general, within

in the context of large and distributed Agile projects.

• In our case studies, we saw that it is, indeed, possible to successfully integrateAgile practices in projects that are (i) large, (ii) distributed, and (iii) take place

within a highly formalized organizational context. We described practices that can

be used to cope with size and distribution, and showed that organizational maturity

can actually be a benefit rather than a hinderance to Agile development.

• Practitioners should consider that, in all our case study projects, learning processeswere involved when Agile methods were first implemented. This led to a more

chaotic period at the start of projects, as project members were getting used to the

new methodology. It should also be recognized that the required induction time for

new project members is longer due to the necessity to train them in the methodology.

The client also had to be provided with the necessary working knowledge before

being able to effectively participate in the development effort.

• In large projects, delivery stories are an effective way to translate the wishes of the

client into a specification that can be turned into software by developers. Practitioners

can use delivery stories to their advantage in situations where developers do not

directly acquire requirements from the client.

• More long-term development streams can be integrated into Agile development,

parallel to short-term iterations. This is an effective way to cope with highly interde-

105

pendent functionality or large, overarching changes, while still using short iterations

for other functionality.

• A division of responsibilities is necessary in projects where every project member

cannot be aware of all aspects of the project, due to size or distribution. Although

contrary to Agile principles, separation of concerns is necessary in these settings,

and we have shown that it can be used effectively in combination with Agile practices.

As such, it provides a precedent that practitioners can draw upon in organizing their

own projects in similar situations.

References

[1] Z Racheva, M Daneva, and L Buglione. Supporting the Dynamic Reprioritization ofRequirements in Agile Development of Software Products. Second InternationalWorkshop on Software Product Management, 2008. IWSPM’08, pages 49–58, 2008.

[2] Z Racheva and M Daneva. Complementing Measurements and Real OptionsConcepts to Support Inter-iteration Decision-Making in Agile Projects. 34thEuromicro Conference on Software Engineering and Advanced Applications, 2008.

[3] Z Racheva and M Daneva. Reprioritizing the Requirements in Agile SoftwareDevelopment: towards a Conceptual Model from Clients’ Perspective. Proceedingsof the 21st International Conference on Software Engineering & KnowledgeEngineering, SEKE’2009, pages 73–80, 2009.

[4] Z Racheva, M Daneva, and K Sikkel. Value Creation by Agile Projects: Methodologyor Mystery? Product-Focused Software Process Improvement, pages 141–155, 2009.

[5] Z Racheva, M Daneva, A Hermann, and K Sikkel. Do we Know Enough aboutRequirements Prioritization in Agile Projects: Insights from a Case Study.Requirements Engineering Conference (RE), 2010 18th IEEE International, 2010.

[6] Z Racheva, M Daneva, K Sikkel, and L Buglione. Business Value Is Not OnlyDollars–Results from Case Study Research on Agile Software Projects.Product-Focused Software Process Improvement, 6156/2010:131–145, 2010.

[7] Z Racheva and M Daneva. How Do Real Options Concepts Fit in AgileRequirements Engineering? Eighth ACIS International Conference on SoftwareEngineering Research, Management and Applications, pages 1–8, April 2010.

[8] Z Racheva and M Daneva. Clients’ participation in software projects: comparativecase study between an agile and a ‘traditional’ software company. University ofTwente Publications, pages 1–4, July 2010.

[9] M Kumar, N Ajmeri, and S Ghaisas. Towards knowledge assisted agile requirementsevolution. RSSE ’10: Proceedings of the 2nd International Workshop onRecommendation Systems for Software Engineering, pages 16–20, 2010.

[10] Z Racheva, M Daneva, and A Herrmann. A Conceptual Model of Client-drivenAgile Requirements Prioritization: Results of a Case Study. ESEM’10, pages 1–4,August 2010.

[11] P Rose, M Kumar, N Ajmeri, M Agrawal, V Sivakumar, and S Ghaisas. A Methodand Framework for Domain Knowledge Assisted Requirements Evolution (K-RE).

106

107

Proceedings of CONSEG-09: International Conference on Software Engineering,2009.

[12] K Beck, M Beedle, and A Van Bennekum. Manifesto for agile softwaredevelopment. The Agile Alliance, 2001.

[13] T Hildenbrand, M Geisser, T Kude, D Bruch, and T Acker. Agile Methodologies forDistributed Collaborative Development of Enterprise Applications. Complex,Intelligent and Software Intensive Systems, 2008. CISIS 2008. InternationalConference on, pages 540–545, 2008.

[14] K Beck. Extreme programming explained: embrace change. Addison-WesleyProfessional, 2001.

[15] K Schwaber and M Beedle. Agile Software Development with Scrum. Prentice Hall,2001.

[16] C Larman and B Vodde. Practices for Scaling Lean & Agile Development: Large,Multisite, and Offshore Product Development with Large-Scale Scrum.Addison-Wesley Professional, February 2010.

[17] R Valade. The Big Projects Always Fail: Taking an Enterprise Agile. In Agile 2008Conference, pages 148–153. IEEE, 2008.

[18] L Cao and B Ramesh. Agile requirements engineering practices: An empirical study.Software, IEEE, 25(1):60–67, 2008.

[19] I Gat. How BMC is scaling agile development. Agile Conference, 2006.

[20] J Auvinen, R Back, J Heidenberg, P Hirkman, and L Milovanov. Software processimprovement with agile practices in a large telecom company. In Product-FocusedSoftware Process Improvement, Proceedings, pages 79–93. Oy LM Ericsson AB,Telecom R&D, FIN-20520 Turku, Finland, 2006.

[21] M Sulfaro and M Marchesi. Agile practices in a large organization: the experience ofposte Italiane. Agile Processes in Software Engineering and Extreme Programming,2007.

[22] H Koehnemann and M Coats. Experiences Applying Agile Practices to LargeSystems. In AGILE, pages 295–300. IEEE, 2009.

[23] H Glazer. Love and Marriage: CMMI and Agile Need Each Other. Open Forum,2010.

[24] D Jacobs. Accelerating process improvement using agile techniques, 2005.

[25] B Whittaker. What went wrong? Unsuccessful information technology projects.Information Management & Computer Security, 1999.

108

[26] L Wallace. Software project risks and their effect on outcomes.COMMUNICATIONS OF THE ACM, 2004.

[27] RW Scholz and O Tietje. Embedded case study methods: integrating quantitative andqualitative knowledge. Sage Publications, Inc, 2002.

[28] M Paulk, C Weber, B Curtis, and M Chrissis. Capability maturity model, version 1.1.Software, IEEE, 10(4):18–27, 1993.

[29] J Dibbern, T Goles, and R Hirschheim. Information systems outsourcing: a surveyand analysis of the literature. Newsletter ACM SIGMIS Database, 2004.

[30] J Webster and R T Watson. Analyzing the Past to Prepare for the Future: Writing aLiterature Review, volume 26. Management Information Systems Quarterly, 2002.

[31] B Ramesh, L Cao, and R Baskerville. Agile requirements engineering practices andchallenges: an empirical study. Information Systems Journal, 20(5):449–480, 2010.

[32] L Cao, K Mohan, P Xu, and B Ramesh. A framework for adapting agile developmentmethodologies. European Journal of Information Systems, 18(4):332–343, 2009.

[33] AD Lucia and A Qusef. Requirements Engineering in Agile Software Development.Journal of Emerging Technologies in Web Intelligence, 2(3):212, 2010.

[34] M Korkala and P Abrahamsson. Communication in distributed agile development: Acase study. 33rd EUROMICRO Conference on Software Engineering and AdvancedApplications, 2007.

[35] M Korkala, M Pikkarainen, and K Conboy. Distributed agile development: A casestudy of customer communication challenges. Agile Processes in SoftwareEngineering and Extreme Programming, pages 161–167, 2009.

[36] H Grewal and F Maurer. Scaling Agile Methodologies for Developing a ProductionAccounting System for the Oil & Gas Industry. AGILE 2007, pages 309–315, 2007.

[37] J Sutherland, A Viktorov, and J Blount. Distributed scrum: Agile projectmanagement with outsourced development teams. System Sciences, 2007.

[38] A Elshamy and A Elssamadisy. Applying agile to large projects: new agile softwaredevelopment practices for large projects. Agile Processes in Software Engineering,June 2007.

[39] J Eckstein. Agile software development in the large. Dorset House, 2003.

[40] K Gary, A Enquobahrie, and L Ibanez. Agile methods for open source safetycriticalsoftware. Software: Practice and Experience, 41(9):945–962, April 2011.

109

[41] N Hong, J Yoo, and S Cha. Customization of Scrum Methodology for OutsourcedE-Commerce Projects. In 2010 17th Asia Pacific Software Engineering Conference(APSEC), pages 310–315. IEEE, November 2010.

[42] R Baskerville, J Pries-Heje, and S Madsen. Post-agility: What follows a decade ofagility? Information And Software Technology, 53(5):543–555, May 2011.

[43] MC Lacity, SA Khan, and LP Willcocks. A review of the IT outsourcing literature:Insights for practice. The Journal of Strategic Information Systems, 2009.

[44] SW Ambler. Agile and large teams. Dr. Dobb’s Journal, 33(7):60–62, July 2008.

[45] FP Brooks. The Mythical Man-Month: Essays on Software Engineering, AnniversaryEdition (2nd Edition). August 1995.

[46] C Larman and B Vodde. Scaling lean & agile development. thinking andorganizational tools for large-scale Scrum. Addison-Wesley Professional, December2008.

[47] S McConnell. Avoiding classic mistakes [software engineering]. IEEE Software,13(5):112, 111, 1996.

[48] R Kliem. Managing the Risks of Offshore it Development Projects. EDPACS,32(4):12–20, October 2004.

[49] M Cohn. User Stories Applied: For Agile Software Development. Addison-WesleyProfessional, 2004.

[50] RP Kendall, A Mark, and SE Squires. Condor: Case Study of a Large-Scale,Physics-Based Code Development Project. Computing in Science & Engineering,2010.

[51] I Christou, S Ponis, and E Palaiologou. Using the Agile Unified Process in Banking.IEEE Software, 27(3):72–79, 2010.

[52] S McDowell. British Telecom Experience Report: Agile Intervention–BT’s Joiningthe Dots Events for Organizational Change. Agile Processes in SoftwareEngineering and Extreme Programming, 2007.

[53] J Bosch and PM Bosch-Sijtsema. Introducing agile customer-centered developmentin a legacy software product line. Software: Practice and Experience,41(8):871–882, April 2011.

[54] A Shatil, O Hazzan, and Y Dubinsky. Agility in a Large-Scale System EngineeringProject: A Case-Study of an Advanced Communication System Project. In 2010International Conference on Software - Science, Technology and Engineering(SWSTE), pages 47–54. IEEE, 2010.

110

[55] K Charmaz. Constructing Grounded Theory. A Practical Guide Through QualitativeAnalysis. SAGE Publications, July 2006.

[56] RK Yin. Case Study Research, Design and Methods. Thousand Oaks: Sage, 1994.

[57] N King. Interviews in qualitative research. Sage Publications Ltd, 2009.

[58] G Coleman and R O’Connor. Investigating software process in practice: A groundedtheory perspective. Journal of Systems and Software, 81(5):772–784, May 2008.

[59] J Rose, K Pedersen, and JH Hosbond. Management competences, not tools andtechniques: A grounded examination of software project management at WM-data.Information And Software Technology, 2007.

[60] S Adolph, W Hall, and P Kruchten. Using grounded theory to study the experienceof software development. Empirical Software Engineering, 16(4):487–513, January2011.

[61] R Hoda, J Noble, and S Marshall. Developing a grounded theory to explain thepractices of self-organizing Agile teams. Empirical Software Engineering, April2011.

[62] C Urquhart. Exploring analyst-client communication: using grounded theorytechniques to investigate interaction in informal requirements gathering. InformationSystems and Qualitative Research, 1997.

[63] A Martin, R Biddle, and J Noble. XP Customer Practices: A Grounded Theory. In2009 Agile Conference (AGILE), pages 33–40. IEEE, August 2009.

[64] JL Martin, DJ Clark, SP Morgan, JA Crowe, and E Murphy. A user-centred approachto requirements elicitation in medical device development: A case study from anindustry perspective. Applied Ergonomics, pages 1–7, May 2011.

[65] BG Glaser. Doing Grounded Theory: Issues & Discussion. Sociology Press, MillValley, California, 1998.

[66] A Cockburn. People and methodologies in software development. University of OsloNorway, page 239, 2003.

[67] G Coleman and R O’Connor. Using grounded theory to understand software processimprovement: A study of Irish software product companies. Information AndSoftware Technology, 49(6):654–667, 2007.

[68] A Martin, R Biddle, and J Noble. The XP Customer Team: A Grounded Theory.2009 Agile Conference, August 2009.

[69] E Whitworth and R Biddle. The Social Nature of Agile Teams. 2007 Agileconference, August 2007.

111

[70] R Wieringa. Problem Analysis and Solution Design Lecture Notes. University ofTwente, 2009.

[71] F Maurer. Supporting distributed extreme programming. Extreme Programming andAgile Methods—XP/Agile Universe 2002, 2002.

[72] N Brown, I Ozkaya, R Sangwan, C Seaman, K Sullivan, N Zazworka, Y Cai, Y Guo,R Kazman, M Kim, P Kruchten, E Lim, A MacCormack, and R Nord. ManagingTechnical Debt in Software-Reliant Systems. In Proceedings of the FSE/SDPworkshop on Future of software engineering research, FoSER ’10, pages 47–52.ACM Press, 2010.

[73] J Iivari and N Iivari. The relationship between organizational culture and thedeployment of agile methods. Information And Software Technology, 53(5):509–520,May 2011.

[74] C Jakobson. The Magic Potion for Code Warriors! Maintaining CMMI Level 5Certification With Scrum. Aarhus, 2007.

[75] M Bass, JD Herbsleb, and C Lescher. Collaboration in Global Software Projects atSiemens: An Experience Report. Global Software Engineering, 2007. ICGSE 2007.Second IEEE International Conference on, pages 33–39, 2007.

[76] M Cataldo, M Bass, JD Herbsleb, and L Bass. On Coordination Mechanisms inGlobal Software Development. Global Software Engineering, 2007. ICGSE 2007.Second IEEE International Conference on, pages 71–80, 2007.

[77] J Herbsleb and A Mockus. An empirical study of speed and communication inglobally distributed software development. IEEE Transactions on softwareEngineering, 29(6):481–494, 2003.

[78] P Runeson and M Host. Guidelines for conducting and reporting case study researchin software engineering. Empirical Software Engineering, (14):131–164, 2009.

[79] A Clarke. Situational analysis. grounded theory after the postmodern turn. SagePublications, Inc, March 2005.

[80] Glaser BG. Basics of grounded theory analysis: emergence vs forcing. SociologyPress, 1994.

[81] Anselm L Strauss and Juliet M Corbin. Basics of qualitative research. techniquesand procedures for developing grounded theory. Sage Publications, Inc, 1998.

[82] Z Bakalova, M Daneva, and A Herrmann. Agile requirements prioritization: whathappens in practice and what is described in literature. Requirements Engineering:Foundation for Software Quality; Lecture Notes in Computer Science, 2011.

App

endi

xA

Theo

retic

alM

odel

-Rac

heva

&D

anev

a

Figu

reA

.1:

Theo

retic

alm

odel

byR

ache

vaan

dD

anev

aA

spr

opos

edin

‘AC

once

ptua

lMod

elof

Clie

nt-d

riven

Agi

leR

equi

rem

ents

Prio

ritiz

atio

n:R

esul

tsof

aC

ase

Stud

y’in

2010

[10]

.

112

Appendix BTRDDC Organization Chart

Figure B.1: TRDDC organization chartChart showing the segemntation of TRDDC, and its place within the wider organization.Details of the Software Engineering lab shown. Position of Requirements Engineering

group shown in bold.

113

Appendix CQuestionnaire

“First, let me introduce myself; my name is Egbert van der Veen, and I’m a Master’s

student of the University of Twente, the Netherlands. As part of my graduation project

I’m conducting research at the Tata Research Development & Design Centre in Pune.

There, we are trying to better understand Agile requirements prioritization from the client’s

perspective. This interview is part of a research project towards this goal.”

Offer consent form“The questionnaire consists of some factual information up-front, followed by a semi-

structured part. The whole interview should take no longer than 60 minutes. Please rest

assured that all findings will be thoroughly anonymized, and no personal or confidential

information will be used in any way. If you feel uncomfortable answering any of the

questions, you can always choose not to answer. If you have any questions at any time,

please feel free to ask.”

C.1 IntroductionThis case study represents an effort in discovering the way in which the agile

requirements prioritization process produces value to the clients/product owners.

We will study how requirements prioritization and decision-making on priorities

happens, what the factors are that play a role in it, and how domain knowledge can support

the process. The long-term goal is to improve the agile requirements prioritization process

so that both clients’ and developer’s organizations profit: by ensuring a better understand-

ing for the rationale behind the decisions and including the consideration of value-creation

throughout the project.

Research Questions

• RQ1: How do companies handle changing requirements during agile development?

• RQ2: Do companies make value-driven decisions during agile development?

114

115

• RQ3: How can the requirements engineering process be supported with domain

knowledge? (what kind of knowledge is valuable to have when performing require-

ments engineering?)

In this study under re -prioritization we understand a re-arrangement of the initially

prioritized Project Backlog (PB) during the project. This can happen after some of the

following occurs: adding a new item on the PB; removing an item; changing the priority

of an item; change in the scope of the project, change in the external conditions, new

information during the project that require a change in the scope/schedule, etc.

“Let’s discuss a concrete project where you had to (re)prioritize requirements on

multiple occasions. Do you have any questions at this stage?”

“We will first discuss some general characteristics of the project”

C.2 Main characteristics and context of the project• How long was the project?

• What was the type of contract? (e.g. fixed-price, per hour, etc.?)

• How many iterations / sprints did you perform?

• What was the nature of the distribution of the project?

– How many sites were involved?

– What was the distribution of people among these sites?

– How was the work distributed between them?

– How was the collaboration organized?

– Which, if any, Agile distribution practices were implemented?

• Who were the different parties involved?

– What were the different layers of corporate hierarchy that were involved?

– Did the different parties represent different business units within the same

organization?

116

• How did the communication with the client take place?

– Modality

– Frequency

– Level of communication w.r.t. corporate hierarchy

– Were there domain experts involved?

• What type of product was developed?

• How critical was the application?

• What was the size of the developing team?

– How many other people were involved?

• What development method was used?

• What was the experience in all organizations with the methodologies that were

applied? (i.e. project methodology (agile development))

– Was there an existing preferred methodology?

• In what form were the requirements captured? (e.g. story cards, use cases, traditional

requirements documents, etc.)

– Was a specific method or template used?

– Were the requirements written by the client, or by a representative of the

developer’s organization?

– Is the business value of each requirement made explicit?

The following sections of the interview will proceed in a semi-structured fashion.

The questions should be seen more as means to get a conversation started, and the line of

questioning should not be considered to be too rigid. First we will talk about;

117

C.3 Section 1. Prioritization and re-prioritization process. (concrete

case)• Q1-1. How often did re-Prioritization happen?

– Between each iteration / sprint?

– Triggered by outside change?

• Q1-2. Can you remember concrete cases of (re)prioritization in the project?

• Q1-3. Do you know what triggered them? How did you proceed in these cases?

• Q1-4. Which factors played a role during the decision making?

• Q1-5. Are there any reprioritization decisions that were hard to make, or took a lot

of discussion?

– Why were they difficult?

– What were the pros and cons?

– What tipped the balance?

– Who, in the end, made the decisions? (client or vendor?)

– Does it always work like that?

– Was this an exceptional case; does it usually works differently?

• Q1-6. Were there any issues arising specifically from the distributed nature of the

project? Such as; Language barrier, time zone difference, culture clash?

• Q1-7. Are there any reprioritization decisions that, in retrospect, you would have

made differently?

• Q1-8. And some that you feel particularly good about? (i.e. where increased insight

or serendipity made you change the direction of the project for the better?)

• Q1-9. Where did changing requirements initiate from? Were they initiated by the

client or by the developers?

“In section 2 we’ll talk about the Agile requirements process in general, so not in

any specific, concrete case”

118

C.4 Section 2: Prioritization process (general observations)• Q2-1: Who is responsible for the decisions on priorities within one project and in

each iteration?

• Q2-2: Are any processes used to help prioritize and select requirements for an

iteration? For example, the Planning Game?

• Q2-3: Are any software tools used to help prioritize and select requirements for an

iteration?

• Q2-4: Which methodologies/techniques are informally used to select and prioritize

requirements, if any?

• Q2-5: Do you use explicit criteria for the prioritization? If yes, which? Do these

criteria change from project to project or in different products? If yes, why? For

example; if in one product the client is more important than in another, or if one

project is more beneficial to your organization than other.

• Q2-6: Do you use any formal way to calculate the worth of a requirement, e.g.

cost-benefit analysis, a value estimation formula, a risk estimation technique?

• Q2-7: Are you happy with the way requirements are currently prioritized at inter-

iteration time? What do you like in the existing approach?

• Q2-9: Do you have cases when a client imposes his preferred way of prioritizing the

requirements and then you adopted this in the prioritization process for the project?

And this let the team change the way you prioritize, which might be different from

what you typically do in other projects? If so, which parts of the approach do you

adapt? Why?

• Q2-10: Do you prioritize all the necessary work like refactoring, testing, etc.,

together with the clients’ requirements? Do you need to explain to the client why

this is necessary?

• Q2-11: Do you consider dependencies between the requirements? If yes, how do you

handle them at prioritization time? How did you keep track of these dependencies?

119

“now we’ll talk about some of your own past experiences for a moment...”

• Q2-12: in the past, have you experienced cases where...

– 2-12.1 ... you had to educate the clients’ representatives why prioritization is

needed?

– 2-12.2 ... the decision-makers did not have the necessary information to

perform prioritization?

– 2-12.3 ... the decision-makers did not know why and/or how to prioritize? If

yes, please explain why, in your opinion, this happened.

– 2-12.4 ... the stakeholders were not satisfied with the priorities, and this had a

negative effect on client’s or your organization?

– 2-12.5 ... where the client requested changes in the backlog, do you know why

did it happen?

– 2-12.6 ... the prioritization decision had impacted the system in a significant

way (significant refactoring or rework needed).

– 2-12.7 ... the client’s representative did not always represent the desires of all

the client’s stakeholders?

– 2-12.8 ... the client’s representative was not authorized to make crucial deci-

sions, stalling / delaying the project?

“Next is the topic of value creation, specifically”

C.5 Section 3: Value creation• Q3-1. What does business value of a requirement mean for you?

• Q3-2: At your meetings with your clients/product owners, do they explicitly discuss

the business value of the requirements, so that everyone at the meeting understands

why some requirements are of higher priority than others?

• Q3-3: Is ‘value’ connected to the business goals which the clients want to achieve by

deploying the software system? If so, in which way does ‘value’ connect to clients’

business goals?

120

• Q3-4: When judging the value of the requirements, do clients also consider any

other factors (e.g. cost, size, risk)?

• Q3-5: Has the desired value been quantified? If yes, how?

• Q3-6: In which way, in your experience, does the agile process add value to the

client? Can you give a specific example from your practice?

• Q3-7: You, as part of the developing side do you consider the value for your own

organization, or is what the client wants more important?

• Q3-8. Do you share knowledge about business value creation within the organiza-

tion?

“Back to a concrete scenario. This section revolves around the question; have there

been situations where a better understanding of the client’s business domain would have

helped requirements prioritization?”

C.6 Section 4: Domain knowledge assistance• Q4-1: Were you personally familiar with the client’s business domain at the start of

the project?

• Q4-2: Did your company have previous experience with this domain?

• Q4-3: Were there recurring issues that originated from unknown aspects of the

client’s business domain?

• Q4-4: In your opinion; have there been situations where a better understanding of

the client’s business domain would have improved the requirements prioritization

process?

– Customer’s business goals made explicit

• Q4-5: Was there a conscious effort, at any phase of the project, to learn more about

the client’s business domain? If so;

– At what stage of the project did this effort take place?

121

– Was there any methodology or tool used?

– Who was responsible?

– How was the customer involved?

• Q4-6: Were the customer’s policies formalized in a structured way and made avail-

able to you, or were they hidden in the processes, from which they had to be reverse

engineered?

Conclude interview; thank participant for his/her time, ask if there are anyquestions/remarks

App

endi

xD

Visu

aliz

atio

nof

inte

rvie

wtr

ansc

ript

s

Figu

reD

.1:

Wor

dC

loud

visu

aliz

atio

nof

inte

rvie

wtr

ansc

ript

sV

isua

lizat

ion

gene

rate

dus

ing

‘Wor

dle’

;http

://w

ww

.wor

dle.

net/,

com

mon

wor

dsre

mov

edm

anua

lly.

122

App

endi

xE

Trac

eabi

lity

betw

een

nara

tive

and

data

Figu

reE.

1:Tr

acea

bilit

ybe

twee

nso

urce

data

and

the

nara

tive

Trac

eabi

lity

ism

aint

aine

dby

keep

ing

the

data

onw

hich

the

text

isba

sed

next

toth

ete

xtin

the

thes

isso

urce

docu

men

t.C

omm

ents

disp

laye

din

gree

nin

the

cent

reco

lum

n.A

pplic

atio

nsh

own

isth

eLA T

EXed

itor‘

Late

xian

’(ht

tp://

taco

sw.c

om/la

texi

an/).

123

Appendix FRelational overview of codes

Figure F.1: Codes and nestingThe codes and their nesting displayed in relational context. Core concept displayed in the

center, with high-level concepts. Made with the mind-map editor ‘Freemind’(http://freemind.sourceforge.net/).

124

Appendix GCode occurrences

Table G.1: Code frequency per interview

Table G.2: Code frequency averages per project

125

Appendix HCode saturation

Table H.1: Code saturationDecrease of new codes as data collection progressed shows code saturation

126

Appendix IUnique codes

All unique codes applied to the interview texts in the data analysis phase. Also shows

nesting of codes in the form of ‘superconcept >subconcept’.

1. business value

2. business value >business goals

3. business value >client vs developer

4. business value >decision hierarchy

5. business value >definition

6. business value >discussion

7. context

8. context >aims

9. context >aims >automation

10. context >aims >integration

11. context >aims >reusability

12. context >background

13. context >contract

14. context >contract >fixed price

15. context >contract >fixed price >consequence >fine

16. context >contract >fixed schedule

17. context >contract >per hour

18. context >criticality

19. context >issues >existing product

20. context >organizational structure

21. context >scope

22. distribution

23. distribution >challenges

24. distribution >challenges >solutions

127

128

25. distribution >communication

26. distribution >communication >hierarchy

27. distribution >facilitation

28. knowledge management >customer involvement

29. knowledge management >domain knowledge

30. knowledge management >domain knowledge >acquisition

31. knowledge management >domain knowledge >QA

32. knowledge management >knowledge sharing

33. knowledge management >lack of knowledge

34. knowledge management >learning process

35. knowledge management >tacit vs explicit

36. methodology >agile

37. methodology >agile >challenges

38. methodology >agile >collections

39. methodology >agile >collections >sprints

40. methodology >agile >feedback

41. methodology >agile >hybrid

42. methodology >agile >hybrid >streams

43. methodology >agile >perceived benefits

44. methodology >agile >products >releases >sprints

45. methodology >agile >reasons

46. methodology >agile >scrum

47. methodology >agile >scrum >teams

48. methodology >agile >surrogate customer

49. methodology >agile >test driven development

50. methodology >agile >transparency

51. methodology >concurrent development

52. methodology >experience

53. project management

54. project management >change process

129

55. project management >induction process

56. project management >metrics

57. project management >QA

58. project management >ramping up complexity

59. project management >refactoring

60. project management >testing

61. requirements >capturing method

62. requirements >capturing method >delivery story

63. requirements >capturing method >user story

64. requirements >change process

65. requirements >constraints

66. requirements >dependency

67. requirements >dependency >cross domain

68. requirements >dependency >difficulties

69. requirements >dependency >solutions

70. requirements >dependency >tracking

71. requirements >prioritization

72. requirements >prioritization >criteria

73. requirements >prioritization >discussion

74. requirements >prioritization >initial

75. requirements >prioritization >methodology

76. requirements >prioritization >methodology >tool support

77. requirements >reprioritization

78. requirements >reprioritization >decision process

79. requirements >reprioritization >external trigger

80. requirements >reprioritization >impact

81. requirements >reprioritization >internal trigger

82. requirements >reprioritization >intraiteration

83. requirements >reprioritization >triggers

Appendix JWeb-based data analysis tool

Figure J.1: Data analysis tool example - dependency handling

130

131

Figure J.2: Data analysis tool example - reading transcripts

132

Figure J.3: Example of comment adding and editing functionality

Appendix KMemo clustering mindmap

Figure K.1: Memo clustering mindmap

An overall view of the mindmap that resulted from the clustering of all the memos

taking during the data analysis process. Meant to give an indication of structure and size;

not meant to be readable at this level of magnification.

133

134

Figure K.2: Memo clustering mindmap - collapsedThe above mindmap with all top-level nodes collapsed, but showing the sub-nodes of the

‘requirements’ concept for reference.

Figure K.3: Memo clustering mindmap - core conceptRe-ordering of memo organization after identification of core concept, namely, ‘copingwith requirements’. The sub-concepts of ‘requirements’ here promoted to the status of

top-level concepts directly under the core concept.

Appendix LThe Thesis Pyramid

Figure L.1: The Thesis PyramidA breakdown of research activities. Structure shows how activities build upon one another,

while surface area represents relative amount of required effort.

135