NIK’01, Tromsø, 26-28 Nov. 2001 1

An Empirical Study on the Utility of Formal Routines

to Transfer Knowledge and Experience

Reidar Conradi, NTNUTore Dybå, SINTEF Tele og data

conradi@idi.ntnu.no, phone +47 73.593444

Norsk Informatikk-konferanse 2001 (NIK’01)Tromsø, 26-28 Nov. 2001

NIK’01, Tromsø, 26-28 Nov. 2001 2

Contents

• Background• Research context and questions• Knowledge of routines• Use and updating of routines• Routines as a medium for transfer of

knowledge and experience• Discussion and implications• Limitations and recommendations for

future research• Conclusions and postscript

NIK’01, Tromsø, 26-28 Nov. 2001 3

Background

• Formal routines are emphasized in both quality assurance (QA) and software process improvement (SPI).

• At the same time; there is often a considerable skepticism among developers to learn from and adhere to formal routines.

• Therefore, this study was initiated to explore how developers and manager perceive formal routines as a medium to express and disseminate knowledge and experience.

NIK’01, Tromsø, 26-28 Nov. 2001 4

Research Context

SPIQ - Software Process Improvement for better Quality:• National project, funded in part by the Research Council of

Norway, 1997-99• Three research institutions and 12 IT companies, mostly

SMEs. • More than 20 SPI pilot projects were run in these companies.

Method:• Interviews with 23 persons (13 developers, 5 QA managers,

and 5 SW manager) in 5 companies participating in SPIQ (convenience sample).

• Qualitative and quantitative data collected by two last year M.Sc. students, supervised by the authors.

• Data analyzed by the students in cooperation with the authors.

NIK’01, Tromsø, 26-28 Nov. 2001 5

Research Questions1

Q1: What is the knowledge of the routines being used?

Q2: How are these routines being used?

Q3: How are they updated?

Q4: How effective are they as a medium for transfer of knowledge and experience?

1with sub-questions

NIK’01, Tromsø, 26-28 Nov. 2001 6

Software developers(n = 13)

Managers(n = 10)

Frequency Percent Frequency Percent

Little - - - -

Some 6 46 2 20

Much 7 54 8 80

Q1.1: Knowledge of Routines

About the same

NIK’01, Tromsø, 26-28 Nov. 2001 7

Q1.2: Degree of Involvement During Introduction of

Routines

Degree of involvement

Low High

Frequency Percent Frequency Percent

Developers 9 69 4 31

Managers 3 30 7 70

Most developers (9/13) had a LOW degree of involvement!!Most managers (7/10) had a HIGH degree of involvement!!

NIK’01, Tromsø, 26-28 Nov. 2001 8

Q2: Use of Routines

• In general, about 50% of the companies’ routines were in use

•The more experienced developers used the routines to a lesser extent than the more inexperienced developers do.

• It was a common agreement that:– “There is no point in having routines that are not considered

useful”•The status of the routines among the software developers

was highly divergent:– “The routines are generally good and useful, but some

developers are frustrated regarding their use.”– “The system is bureaucratic – it was better before, when we

had more freedom to decide for ourselves what should best be done.”

– “The routines are easy to use.”– “Routines are uninteresting and revision meetings are boring.”

NIK’01, Tromsø, 26-28 Nov. 2001 9

Q3: Updating of Routines

• None of the companies had scheduled revisions as part of the process for updating their routines.

• Most answers to this issue were rather vague.

• Some respondents explained that revisions were informally triggered.

• Other respondents did not know how to propose and implement changes to existing routines.

• Respondents from all of the companies said that the employees could participate in the revision activities if they wanted to.

NIK’01, Tromsø, 26-28 Nov. 2001 10

Software developers(n = 13)

Managers(n = 10)

Frequency Percent Frequency Percent

Yes - - 7 70

Both 7 54 2 20

No 6 46 1 10

Q4.1: Routines as a Medium for Transfer of Knowledge and

Experience

Again, strongly skewed, as for Q1.2.

Fisher’s exact test = 13.02 (p = 0.002, two-tailed)

NIK’01, Tromsø, 26-28 Nov. 2001 11

Q4.2: Alternative Mediafor Knowledge Transfer

Rank (1 is best) Medium Developers Managers

Experience base/newsgroups 1 1 Socialization 2 3 Discussion groups 3 2 Experience reports 4 3 On-the-job-training 5 6 Work with ext. consultants 6 - Group meetings 7 5

Similar ranking, but different perception of contents.

NIK’01, Tromsø, 26-28 Nov. 2001 12

Degree of Involvement vs. Assessment of Formal

Routines

Degree of involvement

Efficient medium? Low High

Yes - 7

Both 5 4

No 7 -

Again, strongly skewed, as for Q1.2 and Q4.1.

Fisher’s exact test = 14.71 (p < 0.0005, two-tailed)

NIK’01, Tromsø, 26-28 Nov. 2001 13

Discussion of Results

• Occupational culture– Managers and developers don’t seem to understand each

other– This leads to failure in organizational learning.

• Participation– All of the companies had a low degree of employee

involvement in the development of their routines.– Consequently, the routines were seen as coercive and not

enabling.

• Situated learning– Significant learning should not be divorced from its specific

context.– Developers viewed stories and tacit social activities (e.g.

newsgroups and discussion groups) as more flexible, adaptable and relevant than formal routines.

NIK’01, Tromsø, 26-28 Nov. 2001 14

Knowledge “Conversion”1

Externalization

Internalization Combination

Socialization

ExplicitTacitT

acit

Exp

lici

t

From

To

1Nonaka and Takeuchi (1995).

NIK’01, Tromsø, 26-28 Nov. 2001 15

Implications

• Studies of the effects of idealized formalizations should focus on their relevance for actual routines and their implementation.

• We must learn how to establish better cross-cultural dialogues between the different occupational cultures.

• Managers should balance discipline and creativity, in order to supplement formal routines with collaborative, social processes.

• Both managers and developers should maintain an open dialogue with feedback regarding the utility of formal routines.

NIK’01, Tromsø, 26-28 Nov. 2001 16

Limitations and Recommendationsfor Future Research

Validity:• The small sample and lack of randomness in the choice of

respondents may be a threat to external validity. • A major threat to internal validity is that we have not

assessed the reliability of our measures (e.g. degree of involvement)

Future studies:• Should examine the enabling features of formal routines in

much more detail; both cross-sectional and longitudinal studies.

• Should include a multiple respondent approach to cover all major occupational cultures.

• Should do supplementary, ethnographic studies on how developers really work and how their work relate to formal routines

NIK’01, Tromsø, 26-28 Nov. 2001 17

Conclusions

• Developers in general are positive to experience transfer and guidelines.

• Formal routines alone is not an efficient way to transfer knowledge and experience.

• Large differences between managers and developers regarding the perception of formal routines to transfer knowledge and experience.

• In spite of a small sample, we think that the results are representative for a large class of software companies.

• A more cooperative and open work atmosphere, with strong developer participation is needed.

• The major and most difficult work remains non-technical, that is, to build a learning software organization.

NIK’01, Tromsø, 26-28 Nov. 2001 18

Postscript• Quality assurance: under-taught in CS curricula.• Timothy C. Lethbridge:“The Relevance of Software

Education: A Survey and Some Recommendations”, Annals of Software Engineering, No. 6, 1998 identifies three under-taught topics:– Testing and Quality Assurance– Configuration Management– Process Standards

• Tor Stålhane, IDI: further interviews last summer in 15 Norwegian software companies on attitudes to formal QA systems:– Same pattern: disrespect for formal routines.– Quote:”Oh, quality assurance – that is what those

people in that other department is doing”.• So you will here more from us!