The Influence of Experience and Information Search

7/28/2019 The Influence of Experience and Information Search

1/12

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 54, NO. 2, MAY 2007 315

The Influence of Experience and Information SearchStyles on Project Risk Identification Performance

Eunice Maytorena, Graham M. Winch, Jim Freeman, and Tom Kiely

AbstractThe management of risks in projects is a growingarea of concern. Both the identification and analysis phases of therisk management process are considered the most important, forthey can have a big effect on the precision of the risk assessmentexercise. Currently, it is assumed that project managers relylargely on experience to identify project risks. These decisions,influenced by individual perception and attitudes, are madeprimarily under conditions of uncertainty. Understanding howindividuals respond to uncertain situations, therefore, requires anunderstanding of how individuals intuitively assess the situationthey perceive, before expressing a response. The Project RiskIdentification (Pro-RIde) project interviewed 51 project managersusing active information search (AIS) as a data collection method

and cognitive mapping as a data-capturing tool. Our resultssuggest that the role of experience in the risk identification processis much less significant than it is commonly assumed to be. Bycontrast, information search style, level of education and riskmanagement training do play a significant role in risk identifica-tion performance. These findings suggest the potential for a morethorough approach to risk identification.

Index TermsActive information search, cognitive mapping,project risk identification, project risk management.

I. INTRODUCTION

PROJECT risk management has become an important areaof research in project management over the past decade.

Interest in risk management has increased as the size and com-

plexity of projects have grown and as competition between firms

has intensified. As a result, numerous best practice standards,

guides, and specialist tools and techniques have been developed

focusing on a more effective project risk management process. It

is widely held that both the identification and analysis phases of

the risk management process are the most important ones as they

can have the biggest effect on the precision of the risk assess-

ment exercise [1], [2], [3]. However, the vast bulk of research

Manuscript received June 20, 2005; revised November 1, 2005 and February1, 2006. This work was supported in part by the United Kingdom Engineeringand Physical Sciences Research Council (EPSRC) under Grant N51452/01. Re-view of this manuscript was arranged by Department Editor J. K. Pinto.

E. Maytorena is with the Center for Research in the Management of Projects,Manchester Business School, University of Manchester, Booth Street West,Manchester, M15 6PB, U.K. (e-mail: [email protected]).

G. M. Winch is with the Center for Research in the Management of Projects,Manchester Business School, Manchester, M15 6PB, U.K. (e-mail: [email protected]).

J. Freeman is with the Manchester Business School, Manchester, M15 6PB,U.K. (e-mail: [email protected]).

T. Kiely was with the Miller Construction, Edinburgh, EH12 9HD, U.K.and the Pro-RIde research project, Centre for Research in the Management ofProjects, Manchester Business School, Manchester, M15 6PB, U.K. (e-mail:[email protected]).

Digital Object Identifier 10.1109/TEM.2007.893993

to date has focused on the analysis phase, while the identifica-

tion phase and its techniques have had little rigorous evaluation

[1] and development [4]. Yet we would argue that the analysis

phase is completely dependent upon possible risk events being

accurately identified in the first instance. The consensus on this

phase, we suggest, is that experienced projects managers en-

gage in brainstorming, develop risk registers, conduct risk in-

terviews to identify risks, which can be taken forward for anal-

ysis and subsequent action through the project life cycle. This

leaves open such questions as which strategies are used to gather

information, how much information is required and who is best

placed to carry out identification in order for a judgment to be

made on what is a risk. The aim of this research is to provide

a better understanding of the individual process of risk iden-

tification by focusing on the individual information gathering

process. Thus, this paper focuses on two different but ultimately

connected aspects of the way project managers identify risks.

First, we explore the role of individual project management ex-

perience on risk identification performance (RIP). And second,

we explore the role of information search styles on RIP. We

thereby hope to provide the basis for a more rigorous approach

to the identification phase of project risk management.

II. PROJECT RISK IDENTIFICATION

A. Project Risk Identification in Context

Since the 1980s the development of project risk management

has shifted from progressing the quantitative aspect towards im-

proving the understanding of the risk management process [5].

Greater understanding of this process is important for project

risk management practice, for as Royer suggests unmanaged

or unmitigated risks are one of the primary causes of project

failure [6, p. 6].

Project risk management is a process that aims to system-

atically identify, evaluate and manage project related risks to

improve project performance. The basic idea is to make the ex-

ercise more objective. The general consensus from the variousguides [7][10] and risk management literature [11][15] is that

the risk management process can be synthesized in four basic

sub processes, illustrated in Fig. 1. These are located in the con-

text of the project mission and clearly defined project objectives,

which are looped through the project lifecycle: identify and clas-

sify the risks, analyze the risks, respond to the risks and monitor

the risks.

The identification and analysis subprocesses are considered

the most important [1][3]. However, we argue that analysis is

dependent on risks being accurately identified in the first place.

If risks are not identified they cannot be analyzed and man-

aged. Hence, the risk identification phase, where the questions

0018-9391/$25.00 2007 IEEE

Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on September 24, 2009 at 20:44 from IEEE Xplore. Restrictions apply.


2/12

316 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 54, NO. 2, MAY 2007

Fig. 1. A generic model of the risk management process [16]. Our researchfocuses on the first sub process, the identification and classification of risks.

addressed are what might happen and how, is the focus of this

research.

B. Project Risk Identification: A Literature Review

The risk identification phase has been the subject of relatively

little research. Williams [4] points out that there has been little

structured work on identification. We discuss here some of the

exceptions to this assessment. Noonan and Thamhain [17] pro-

posed a risk factor framework that can aid in the categorization

and analysis of project risks. Ashley and Avots [18] proposed

using influence diagramming as a way of simplifying and struc-

turing the sensitive issues in a project. Charette [19] proposed

a framework for risk categorization based on risk causes and

level of predictability. He identified a two-step risk identifica-

tion process: information gathering and risk categorization. In

order to understand the causes of risks one must gather back-ground information from various sources. The sources of infor-

mation, he notes, may come from traditional knowledge (per-

ceived information generally taken as fact), historical data from

other projects, personal judgment, experiments and tests (sim-

ulations, modeling), and statistical surveys. Charette [19] ar-

gues information based on past history is probably the best

primary source for identifying risks [19, p. 106]. As a second

step, categorization is used to help structure the risks based

on their causes and level of predictability. Carr et al. [20] de-

veloped a taxonomy-based risk identification method for soft-

ware engineering projects. The taxonomy offers a framework

for structuring the issues and the elicitation of risks is carriedout based on a questionnaire. Most recently, Chapman [21] has

looked at the steps a design team undertakes in the risk identi-

fication and assessment stages. The steps identified are: knowl-

edge acquisition, selection of core design team, presentation of

the process, identification, encoding, and verification. The un-

derstanding of the risk identification and assessment stages is

through description of tasks, activities and techniques used by

the design team during each of the steps. For the identification

step, Chapman [21] reviews techniques that can be used, such as

brainstorming, Nominal Group Technique (NGT), Delphi and

historic records. Empirical research on risk management prac-

tice [1], [2], [22][26] indicates that over the past decade: check-

lists, brainstorming, and interview sessions have been the mostcommonly used risk identification tools.

Thus the focus of the related literature is on the tools and tech-

niques used for assisting in risk identification, such as risk reg-

isters, risk breakdown structures (RBS) and brainstorming, but

these are not unproblematic. The widely used risk register is

simply a list of all the risks that have been previously identified;

its development is typically ad-hoc. For this to be of practical

use, the register has to be filtered for a particular project underscrutiny and the results prioritized. However, it is not clear how

this is done, by whom, and how reliable the results are [27].

There appears to be a complete lack of connection with the liter-

ature on knowledge management as a tool for capturing organi-

zational learning from projects [28]. RBSare more sophisticated

risk registers which provide a hierarchical structure of potential

risk sources [29] from which a list of risks can be drawn through

a brainstorming session. The issues related to risk registers and

brainstorming also apply to the RBS technique. Brainstorming

[11] is project specific and requires a group of experienced prac-

titioners to consider creatively possible risk sources. This list is

then more analytically considered and key risks identified. Dif-

ficulties with brainstorming include the selection of the appro-priate experts, the number required, and bringing them together

frequently enough to be of use to a dynamic project lifecycle,

and the avoidance ofgroupthink dynamics.

Chapman [30] and Al-Tabtabai and Diekmann [31] argue that

the identification of risks relies on the individual judgment and

insight of the various actors involved in a project, which is de-

pendent on their knowledge, professional training, role, level

of responsibility and length of exposure to the project sector in

which they are working.

Although this research provides a general understanding of

the risk identification process, it leaves unclear how individual

project managers search and gather information from the men-tioned sources in order to make a judgment on what is a risk,

and how, if in any way, this influences risk identification effec-

tiveness. Crutcher [32] states the importance of obtaining this

type of information in order to identify its role on performance.

In addition, the related literature highlights the importance of

the individuals understanding of the project, its development

process and risk sources as fundamental for the effectiveness of

risk identification. Risk identification relies on individual judg-

ment. The untested premise of both research and practice in

project risk management is that experience is the key to effective

risk identification, and that the deployment of this experience in

the risk identification process is largely unproblematic.

III. HYPOTHESES

From these considerations, we can derive hypotheses for

testing. We have identified the widely accepted, but untested,

assertion in the literature that experience matters in project RIP.

H1: There is no association between a project managers

years of experience and their level of project RIP.

We have also indicated that the information gathering stage as

a first step to risk identification required research attention. We

believe that information search styles, those that project man-agers use to gather information, may play a role on project RIP:



3/12

MAYTORENA et al.: INFLUENCE OF EXPERIENCE AND INFORMATION SEARCH STYLES ON PROJECT RIP 317

H2: There is no difference in the styles of information

search used by project managers and their level of RIP.

IV. THE METHOD

A. Theoretical Underpinning

Project risk management is part of the more general problem

of judgment under uncertainty [33]. To address this problem

in a project context, we draw upon a critique of the predom-

inant perspective in this area Expected Utility Theory [34].

Here, the decision-maker rationally evaluates the probabilities

against a final asset position before choosing a course of ac-

tion. However, this theory has been criticized for its assumption

that rationality is possible under such conditions, because evi-

dence has been found that decision-makers use flawed heuris-

tics in decision-making, which are subject to systematic biases

[35], [36]. Within this perspective, Kahneman and Tversky [37]

proposed their prospect theory, in which decision-makers as-

sign values to gains and losses rather than to final assets andto decision weights rather than to probabilities. This produces

the distinctive s-curve value function of the theory. While there

have been important debates within the heuristics and biases lit-

erature [38] this probabilistic approach to decision-making has

been widely accepted. However, the heuristic and biases critique

of expected utility theory has been criticized on methodological

grounds due to the artificial nature of the decision problems re-

searched [39]. Generally in decision-making studies, decision

makers are presented with well-defined problems with all re-

quired probability distributions available. In practice, an active

information search (AIS) is required by decision makers to tease

out the nature of the problem situation and assign the appro-priate decision weights to the data. This naturalistic approach is

much closer to the sort of situation facing project risk decision

makers than those of perfect information envisaged by expected

utility theory and bounded rationality envisioned by prospect

theory. The research methodology used in this research is based

on AIS.

B. AIS and Cognitive Mapping

AIS was developed to study judgment and decision making in

naturalistic tasks. These are ill structured problems in knowl-

edge-rich domains, where causal relations and attributions and

the decision makers control belief are relevant [39, p. 15].At its core AIS is a process tracing technique of information

search and collection, carried out in the context of an interview

where the interviewee is presented with a scenario of a problem.

After the review of the scenario the interviewee asks the facili-

tator questions in order to obtain information. These questions

are recorded and answers are provided in printed form. Huber s

[39] model of how individuals reach a decision in a naturalistic

situation assumes that the decision maker constructs a simple

mental representation of the situation and alternatives, which

can change in the course of the decision process. In this re-

search we utilized this technique with the developments pro-

posed by Ranyard et al. [40], Williamson and Ranyard [41],

and Williamson et al. [42]. The developments consist in pro-viding spoken rather than written answers to questions and by

including think aloud instructions, so that a conversational ap-

proach is adopted. The use of think aloud instructions is useful

for the provision of processing information data, but leaves open

the question of how these processes are to be recorded for sub-

sequent analysis. For this reason, we turned to cognitive map-

ping as a data recording and analysis tool. Cognitive mapping

[43], [44] is an interactive decision support tool used to analyzecomplex or messy processes through which decisions emerge.

A cognitive map is a graphical model which structures the way

an individual makes sense of their experiences. The map is rep-

resented by concepts (distinct phrases) and links between con-

cepts, creating a network, which communicates the nature of a

problem. Although cognitive mapping has been used in the area

of risk management [45][47] and in other fields that involve

risk[44], [48][51] its application to the problem of how project

managers specifically identify risks in projects combined with

an AIS methodology is novel.

C. Participants

We were very keen to work with practicing managers in a do-

main with which they were familiar, as this yields the most ap-

propriate context for research on naturalistic decision-making

[52]. We, therefore, chose to collaborate with a group of four

U.K.-based construction firms from whom we sampled middle

managers. These firms comprised: two large international con-

struction firms, one large U.K. national construction firm, and

one medium-sized London-based construction firm. The selec-

tion criteria were that potential participants had a minimum

of two years experience in a project management position and

could potentially take over a project at short notice. As the in-

terviews progressed, each firm provided our research team with

a list of 1220 potential participants. Our research team used ajudgment sampling [53] based on professional role to select the

interviewees.

We interviewed 51 (4 female, 47 male) practicing construc-

tion project managers. Their ages ranged from 28 to 62 years

( ), with average experience of 17.5

years in a management role, and average experience of

years in current job title. 78% of interviewees

had formal risk management training (22% had no formal risk

management training), and 51% of interviewees had a grad-

uate-level education (49% had a nongraduate level education).

The first four interviews constituted the face validity exercise

and two additional interviews could not be conducted properly

due to time constraints; we have excluded these data from the

analysis. We used for this analysis the data from 45 of our inter-

views. Agresti and Finlay state that a sample size of 25 or 30

is adequate for a good approximation of a normal distribution

[54, p. 104]. Although our sample is not random, we believe that

it is reasonable to suggest that our findings are representative of

middle-level project managers in the U.K.. We have no reason

to believe that the sample is systematically biased in any partic-

ular way. Clearly, however, a larger sample would be required

to draw general conclusions.

D. AIS Procedure

Participants were interviewed individually in their place ofwork. Interruptions and distractions were kept to a minimum.



4/12


Fig. 2. Research method process.

The ProRIde AIS interview procedure lasted between one and a

half and two hours and was structured in three stages:

1) introduction and warm-up;2) AIS exercise;

3) summary and questionnaire.

The introduction informed the interviewee of the aim of the

research project, the structure of the interview process and what

was expected of the interviewee. The aim was to clarify the ex-

ercise to the interviewee, but at the same time information was

kept to a minimum so as not to influence the outcome of the AIS

exercise. The warm-up exercise aimed to clarify the dynamics

of the main exercise (AIS), such as thinking aloud and using

questions and answers.

The aim of the AIS exercise was to produce a response from

the practicing managers that would be as real as possible. Thepiloted scenario [55], based on a real construction project,

was developed by the research team in collaboration with the

project manager of the real project. The scenario (included in

Appendix A) described a building project under a design and

build contract that was currently in progress; participants were

given limited information about its location, team, cost, client

and project status with a focus on schedule and budget risks.

The limited information meant that the potential of the scenario

to shape the interviewees responses was kept to a minimum

and would compel the interviewee to request additional infor-

mation from the facilitator. This process needed to occur in

order for the AIS method to work. Each interviewee was asked

to assume that they were part of the project team and that theyhad to take over the project at short notice. Each interviewee

then went through the AIS process described in Section IV-B

with the aim of identifying the risks in the project.

The objective of the summary exercise was to obtain a retro-spective view of how decisions were made. Interviewees were

asked to summarize the risks that they had identified and the

reason why the interviewee considered them risks. In addition,

the facilitator could also ask why certain questions were asked

or not. This type of report was used to review the consistency of

the data elicited [41]. In addition, demographic data were col-

lected through a questionnaire.

V. DATA ANALYSIS

The process of preparing and analyzing the data consisted of

three main stages: data mapping, data coding, and analysis. In

some instances these were carried out simultaneously. Fig. 2illustrates the method from data collection to data output.

A. Data Mapping

Both the scenario and summary stages of the AIS procedure

were tape-recorded and transcripts produced. The verbal reports

(sequential transcripts) contain data in sequence on the lines

of reasoning and type of information searched for and used

during the scenario exercise. Due to the volume of data gathered

(1520 pages per transcript) we recognized that we needed to do

more than analyze the content. Therefore, we used Decision Ex-

plorer (cognitive mapping software) to represent graphically

the AIS data. This type of graphical representation can be con-

sidered a cognitive map because it represents people in relationto their information environment [56, p. 267]. For the purpose



5/12


Fig. 3. Informationsearch cluster styles. (a) Linear cluster style, (b) Feedback cluster style.

of clarity we will refer to these as information search maps (Bin Fig. 2).

We built the information search maps by transcribing the data

directly into Decision Explorer. Starting at the beginning of

the tape, we entered sequentially numbered concepts into De-

cision Explorer and linked these to represent a chronological

relation (conceptsfollowing in time). A conceptcould be a ques-

tion or statement from the interviewee or an answer from the fa-

cilitator. The sequence of concepts and links was broken when

a new question was asked about a new or different topic. The

new concept then marked the start of a new line of inquiry.

B. Data Coding

During the data mapping stage, and in order to analyze the

information search maps, we developed a coding framework for

three distinct variables:

concept variable;

outcome variable;

process variable.

The concept variable, a distinct phrase, coded at concept level

(A in Fig. 2), was coded as answers (facilitators input), ques-

tions, and statements. The coding of concepts was based on

how to code guidelines developed by the coding team.

The outcome variable is the risks identified by the intervie-

wees and this was also coded at concept level (C in Fig. 2). The

risks identified form the base for the development of a RIP mea-sure. This is explained in Section V-D.

Each coded information search map contained between 200and 600 concepts. To help manage the data we used Decision

Explorers cluster analysis option (D in Fig. 2), in which each

information search map was segmented into groups of concepts

called clusters. Clusters were created based on the strength of

linkage between concepts. The process variable, coded at cluster

level (E in Fig. 2), indicates the approach taken by the project

managers to search and collect information; this could be in a

linearor feedbackstyle.

Fig. 3 illustrates these two types of cluster styles. A linear

cluster style was evident when the interviewees asked single

independent questions without follow-up. As can be seen in

Fig. 3(a) two closed questions were asked about risk assess-ments but no additional detail was requested.

A feedback cluster style was evident when the interviewees

asked a series of related questions in an investigative manner.

As can be seen in Fig. 3(b) five questions were asked to obtain

more detail about the cladding packages, and the program and

drawings were reviewed before an assessment was made. Each

cluster, therefore, describes the sequence and style of informa-

tion search an interviewee went through during a particular topic

of the scenario. In this sense, each cluster describes a specific

topic; and each was given a title to capture its contents.

To improve coding reliability, two coders independently ex-

amined the information search maps. The comparison between

coded maps indicated a high percentage of agreementbetween the two coders.



6/12


Fig. 4. Summary information search map: Linear style.

C. Summary Information Search Map Analysis

Our data cannot claim to have captured every nuance of the

interviewees thought processes. Nonetheless they do give an in-

sight into what information was sought, in what order, what de-

cisions were based on prior experience, which were based on

information collected during the decision exercise, and what in-

formation search strategies were used.

In order for the information search maps to be compared in

terms of content and structure, it was necessary to summarizethem further at cluster level. Therefore, a summary information

search map (F in Fig. 2) was created for each interviewee using

VISIO (graphics software package).

These summary maps contain data about the information that

was sought, the sequence of the information search, the number

of questions asked per topic, the style of information search

used, which could be in a linear or feedback style, the risks

identified, and feedback loops. By calculating the percentage of

linear and feedback clusters in the summary information search

map we obtain a measure for the process variable. That is, we

have a measure of both linearand feedbackinformation search

styles, which is used in the subsequent analysis. Therefore, we

can also determine the predominant style used. A predominantlylinear style is defined if more that 50% of the clusters were linear

[as shown in Fig. 3(a)]; a predominantly feedback style is de-

fined if more than 50% of clusters were feedback style [as shown

in Fig. 3(b)]. Fig. 4 illustrates a summary information search

map, with a predominantly linear style.

D. Risk Identification Performance (RIP)

The risks identified by the interviewees were of varied scope

and we needed to develop a measure that would capture this

variation. A frequency count of the risks identified on its own

was inappropriate, as this did not take into account the qualityof the risks identified. We initially considered using the concept

of impact and probability to develop a measure. However, due to

the unavoidable use of hindsight knowledge in our assessment

of the risks identified, it was not believed to be appropriate to

allocate a probability of each risk event occurring. On the other

hand, we did believe it to be appropriate to use a measure of

impact (severity) as the basis for developing a RIP measure, as

a measure of individual risk identification effectiveness.

The project manager who managed the real-life construction

project on which the scenario was based is the project manager

of the research team. We were able, therefore, to benefit fromhis

detailed knowledge of the scenarios background to assess the

impact each of the risks identified by the interviewees wouldhave on the scenario project should it have occurred. In other



7/12


TABLE I

CORRELATIONS BETWEEN RIP AND EXPERIENCE

words, we took advantage of this hindsight knowledge to estab-

lish the potential impact of each risk identified by the intervie-

wees. All identified risks were, therefore, entered into a matrix

and rated individually on a 1 (very low) to 5 (very high) impact

scale (G in Fig. 2). Two other members of the research team in-

dependently reviewed the rated matrix for consistency. The total

number of risks identified by each interviewee weighted by their

impact rating and expressed in suitably standardized form [57]

gave us the RIP measure for each interviewee (H in Fig. 2). In

other words, the RIP is constructed from partial averages of

ordinal observations which are assumed to be

independent identically distributed observations with expected

value and variance . Thus

and

If we convert the means to values where

`` '' , then the variance of

becomes whatever the value of k (number of risks). In this

case the RIP measure is actually equivalent to .

Using the law of large numbers and the central limit theoremwe deduce, for sufficiently large , that RIP is approximately

normally distributed with constant varianceas required for the

subsequent multiple regression modeling.

It is important to note that the RIP measure is not an indi-

cation of absolute level of performance, but a relative measure,

constrained by our choice of scenario. The RIP forms our de-

pendent variable in the subsequent quantitative analysis of the

summary information search maps.

VI. QUANTITATIVE RESULTS

The development of an RIP measure and the identification of

two information search styles: linear and feedback, and demo-

graphic data collected allows us to test the specified hypotheses.

A. Testing the Hypotheses

The premise of both research and practice in project risk man-

agement is that experience is the key to effective risk identifica-

tion. Here, we test hypothesis 1. The following analysis takes

the RIP measure as the dependent variable and years in man-

agement role, years in current job title and age (all proxies for

experience) as the independent variables. Table I summarizes

the results. As can be seen we cannot reject the null hypothesis

at the 5% level.

We stated that the way in which project managers search andgather information may play a role in their level of project RIP.

TABLE II

DIFFERENCE IN PREDOMINANT STYLE AND RIP

Here, we test hypothesis 2. Table II summarizes the results. As

can be seen we can reject the null hypothesis at the 5% level.

A descriptive analysis showed that project managers with a pre-

dominant use of feedback style had a higher average RIP score

than those managers who had a predominant linear style.

B. Regression Analysis

Having tested the specified hypotheses we wanted to explore

further other determinants of project managers RIP. Therefore,

multiple regression analysis was used to establish empirically,

the determinants of managers RIP.

In this case the RIP measure we adopted allowed for tra-ditional regression modeling. Before analysis began the stan-

dard validation tests [58] were carried out, these are included

in Appendix B. Only one outlier with a standardized residual

of 2.02 was found for the model (all other standardized resid-

uals were within the to range). See the relevant plots for

the model in Appendix B, which show the normality and ho-

moscedasticity assumptions for the response are upheld.

A standard multiple regression analysis was carried out using

the RIP measure as the dependent variable and, age, linear style,

feedback style, whether or not the interviewee had had formal

risk management training (dummy variable) and whether they

had a graduate or nongraduate degree (dummy variable), as po-tential predictor variables. Previous analysis showed that years

in a management role and years in current job title were not sig-

nificant predictor variables and were not included in this anal-

ysis. Descriptive statistics and correlations between the vari-

ables entered into the model are presented in Table III, and the

results of the regression analysis are shown in Table IV. Table III

shows that the linear and feedback styles of information search

and educational attainment have strong correlations with the

RIP measure, but that the correlation between the RIP measure

and other predicator variables is weak. The results of the mul-

tiple regression analysis (Table IV) show that over a third (36%)

of the variation in the RIP measure can be explained by three

predictorsin order of importance- education attainment, theuse of feedback style followed by risk management training.

C. Orphan Risk Analysis

Some of the risks identified were not linked directly to any

cluster in the summary information search map. This indicates

that the risk was identified without any prior information ac-

quisition or follow-up. We infer from this that previous expe-

rience; training or company procedures were used to identify

these risks. These types of risks we have called orphan risks.

Therefore, the following analysis takes the orphan risks (mea-

sured as percentage of total risks identified) as the dependent

variable and the demographic factors as the independent vari-able with a null hypothesis that there is no association between



8/12


TABLE III

CORRELATIONS BETWEEN RIP AND PREDICTOR VARIABLES

TABLE IV

REGRESSION OF RIP ON PREDICTOR VARIABLES

TABLE VCORRELATION BETWEEN ORPHAN RISKS AND DEMOGRAPHIC VARIABLES

the two. Table V summarizes the results. As can be seen we

cannot reject the hypothesis at 5% level except for role (com-

mercially or production orientated) with production orientated

managers tending to identify more orphan risks; years in man-agement role (positively associated) and years in job title (posi-

tively associated). It appears that more experienced project man-

agers are less likely to approach the scenario with a questioning

mind, and seem to rely upon procedures and their prior experi-

ence in identifying project risks.

The following analysis takes orphan risks (measured as per-

centage of total risks identified) as the dependent variable and

cluster style ratio (ratio of linear cluster style/feedback cluster

style) as the independent variable with a null hypothesis that

there is no association. Table VI summarizes the results. As can

be seen we can reject the null hypothesis at 1% level. These re-

sults suggest the higher the style ratio, that is, the more linear

cluster style of information search used, the higher the per-centage of orphan risks identifiedin other words those who

TABLE VI

CORRELATION BETWEEN ORPHAN RISKS AND CLUSTER STYLE RATIO

prefer a linear style of information search for identifying project

risks are also more likely to identify orphans risks, that is iden-

tify risks without any prior enquiry.

D. Results Overview

From these data, therefore, we can conclude that there is no

significant association between managers age, years in man-

agement role and years in job title (proxies for experience) and

their RIP measure. Hence, we cannot reject , in other words,

having more years of project management experience does not

lead to a higher RIP measure. There is a significant difference

in the styles of information search used by project managers

and their RIP measure. Hence, cannot be rejected, in other

words, those managers who used a feedback style more fre-

quently had a higher RIP measure and those who used a linear

style more had a lower RIP measure. Further exploration of

the data showed that the linear and feedback styles of informa-

tion search, and educational attainment have a strong correlation

with the RIP measure. And the variation in the RIP measure can

be explained by education attainment, the use of feedback style

and risk management training. In other words the way in which

managers search for information plays a role in their level of

RIP.

The link between style of information search and level of ed-

ucation is difficult to discern with our sample size because it isnot simply a question of graduates using feedback, and nongrad-

uates using linear styles of information search. Graduates tend

to use both styles as appropriate, while the nongraduates are less

likely to use a feedback style. These conclusions are reinforced

by the analysis of orphan risks, defined as those risks identified

without enquiry. While these are a minority of the risks identi-

fied, more experienced project managers (older and with more

years in a management role) and those who preferred a linear

style tended to identify more orphan risks. Exploration of these

risks indicated that they tended to be the lower impact risks, for

example, logistic risks. These we infer might be more readily

identified using prior experience, training or company proce-dures.

These findings are both counter-intuitive and interesting.

They are counter-intuitive, because they suggest that experi-

ence plays no direct role in project RIP. They are interesting

because they suggest that educational attainment and training

can improve project RIPyears of experience is not subject

to managerial intervention to improve performance, but it is

possible to train staff. In addition, we find that the process style

in which information is gathered also contributes to the RIP.

In this case, a feedback style, that is, an iterative, investiga-

tive approach to gathering information contributes to a better

RIP, that is, identification of a greater number of high impact

risks. Again, this suggests the potential for staff training. Wealso suggest that a linear style of enquiry can be considered



9/12


to be a proxy for a checklist mentality. Those educated at

nongraduate level and more senior managers tended to rely on

past experiencewhether codified or not- rather than taking

an open-minded look at potential risks on the project they are

being asked to manage.

VII. CONCLUSION

This paper describes a method for studying how project man-

agers search and gather information in order to make a judg-

ment of what is a risk in a project. The review of the related

literature indicated that the risk identification phase, although

one of the most important has received little research attention,

with the literature focusing on tools and techniques used to aid

this process. Understanding how managers identify risks, that

is, the means by which they use their knowledge, expertise and

information, placed the enquiry in the area of judgment under

uncertainty. The review of the development and critiques of key

decision making theories pointed towards the importance of the

use of AIS for teasing out the nature of a problem situation. As

a result the methodology used to study the first step of risk iden-

tification process information search and gathering is a conver-

sation-based AIS combined with cognitive mapping, which was

used to capture the AIS data for subsequent coding and cluster

analysis. The results show that:

the style of information search plays an important part in

RIP;

there is no significant correlation between the RIP measure

and age, years in management, years in job title, which are

our proxies for project management experience;

risk management training contributes to improving RIP; graduate level of educational attainment seems to con-

tribute to a better RIP;

role, years in management role and years in current job

title are significantly correlated with the identification of

orphan risks and the use of a checklist approach.

Having looked at the data in different ways, feedback style,

risk management training, and educational attainment have been

highlighted as significant. In sum, the results show that intervie-

wees with a high use of feedback style of information search per-

formed better at identifying more high impact risks. The iden-

tification of risks without any information search tends to be a

common strategy used by those with more project managementexperience and with a nongraduate level of education.

This paper has provided some insights and better un-

derstanding into how project managers search and gather

information in order to identify project risks. Therefore, con-

tributing to a body of knowledge which has not been subject

of much structured research. The AIS method has allowed us

to capture the initial thoughts and information search process.

The use of Decision Explorer to capture and analyze this

process at cluster level has been extremely beneficial for the

identification of information search styles, in addition the use

of a graphic software, in this case VISIO, has beneficial for

visualizing the process dynamics.

These results report on the first stage of longer-term researchinto how project managers cope with risk and uncertainty on

their projects. We have demonstrated empirically that the re-

liance on the project management experience alone in the identi-

fication of project risk is inadequate. Indeed, it may be counter-

productive because it seems to encourage a check-list men-

tality. In broad terms, our findings suggest the importance of

what Schn [59] has called reflective practice for the practice

of risk management. Risk registers and brainstorming by expe-rienced people may not be adequate for effective risk identifica-

tion, and this has strong implications for effective risk manage-

ment practice.

The next stages in our research focus on translating our find-

ings into practice by reviewing risk management training pro-

grams of study and developing recommendations for improving

the risk identification phase. In parallel, we are looking at risk

identification at group level focusing on the group dynamic as-

pects. We will be reporting on these in due course.

APPENDIX A

You are a project manager working for a main contracting

organization. The company operates nationally and specializes

in the mid range size of building and civil engineering projects

within the range 520 million. For the purposes of this exer-

cise, assume that this fictitious company operates the same pro-

cedures and corporate policies as the one that currently employs

you.

You have just completed your last project. The project you

will now be involved with represents an important potential

business opportunity for your company, as the client is an im-

portant property developer who is keen to exploit further sites

for development.

As a consequence of the illness of the original project man-ager, you have been asked to take over this project at very short

notice. The sudden departure of the previous project manager

has allowed little time for a formal hand-over. Therefore, you

will have to quickly review the situation finding the necessary

information from the site files and the rest of the project manage-

ment team. The team consists of an assistant project manager,

two section managers, and a graduate engineer.

You will be responsible for all operations particularly, the en-

velope of the building, including the cladding and windows sub-

contract, the brickwork associated to the external finishing and

roofing.

The first brief you receive can be summarized as follows. Theproject involves demolishing an existing building, replacing it

with an 8-storey, high-quality, prestige, office block with retail

on basement, ground and first floors. The total value of the con-

struction contract is about 8m. The site is located in the heart

of Birmingham in a very popular, busy and congested mixed use

area, controlled by a vigilant Local Authority which requires the

site to be operated in accordance with all statutes and local by-

laws.

The contract is a novated design and build. The architect had

been working on the design for some time, taking advice from

at least one trade contractor whom you will be responsible for.

Your company, as main contractor, has responsibility for the

completion of the design through to hand-over. The design teamcomprises consultants who have worked together previously, but



10/12


neither your company nor you personally have worked with any

of them before.

The whole system of cladding and fenestration is the product

of a preliminary design, which the architect carried out during

the early stages, with the help of a specialist company. The prin-

cipal part of the envelope, the fenestration package, is valued at

approximately 1.5 m. Other packages you will be responsible

for are as follows.

Roofing package (valued at about 500 000).

Reconstructed Stone package (valued at about 100 000

supply and fix).

Facing Brickwork package (valued at about 250 000).The current status of the project is as follows.

The demolition works have started.

The design status is:

structural design inc. reinforcement detailing complete;

architectural G.A.s complete;

architectural and specialist detail design at Design Princi-

ples stage.

Some subcontracts have been awarded:

demolition;

groundworks;

R.C. frame.

Others are in negotiation: reconstructed stonework (supply and fix);

facing brickwork;

membrane insulated roof construction;

windows, curtain wall and cladding;

The management team is concerned about some specific as-

pects of the project.

At this stage you need to take control of the project. We would

like you to verbally describe your thoughts and concerns about

it. Thismaybe intheform ofa short listof whatyouthinkarethe

principal risks involved. To do that, feel free to ask any questions

you need to help in your judgments. The facilitator will try to

answer with realistic answers. If he/she is not able to answer

the question you will have to make your own assumptions as towhat possibilities there may be.

APPENDIX B

Obs FEEDB CL RIP Fit SE Fit Residual St Resid.

20 62.0 5.000 9.613 0.842 .

R denotes an observation with a large standardized

residual.

Durbin-Watson statistic .

No evidence of lack of fit .

The latter MINITAB output also shows there are no problems

with first-order serial correlation of errors or model fit.

REFERENCES

[1] R. Chapman, The effectiveness of working group risk identificationand assessment techniques, Int. J. Project Manage., vol. 16, pp.333343, 1998.

[2] D. Bajaj, J. Oluwoye, and D. Lenard, An analysis of contractors ap-proaches to risk identification in New South Wales, Australia, Con-struction Manage, Econ., vol. 15, pp. 363369, 1997.

[3] D. F. Cooper and C. B. Chapman, Risk Analysis for Large Projects:Models, Methods and Cases. New York: Wiley, 1987.

[4] T. Williams, A classified bibliography of recent research relating toproject risk management, Eur. J. Oper. Res., vol. 85, pp. 1838, 1995.

[5] K. A. Artto, , K. Kahkonen and K. A. Artto, Eds., Fifteen years ofproject risk management applications where are we going?, in Man-

aging Risks in Projects. London: E & FN SPON, 1997, pp. 314.[6] P. S. Royer, Risk management: The undiscovered dimension ofproject management, Project Manage. J., vol. 31, pp. 613, 2000.



11/12


[7] Guide to Project Management Book of Knowledge. Newton Square:PMI, 2000.

[8] J. Bartlett, C. Chapman, P. Close, K. Davey, P. Desai, H. Groom, D.Hillson, M. Hopkinson, R. Gerdes, E. Major, K. Newland, S. Simister,M. Greenwood, P. Campbell, and T. Williams, Project Risk Analysisand Management (PRAM) Guide, 2nd ed. Norwich, U.K.: APM,2004.

[9] C. Lewin, RAMP: Risk Analysis and Management for Projects.London, U.K.: Thomas Telford, 2002.

[10] BS 6079-1:2002 Project ManagementGuide to ProjectManagement,, BSI, 2002, U.K.: British Standard Institute.

[11] C. Chapman and S. Ward, Project Risk Management Processes, Tech-niques and Insights, 2nd ed. Chichester, U.K.: Wiley, 2003.

[12] C. Pritchard, Risk Management: Concepts and Guidance, 2nd ed.Arlington, VA: ESI International, 2001.

[13] J. Raftery, Risk Analysis in Project Management. London: E & FNSpon, 1999.

[14] E. M. Hall, Managing Risk: Methods for Software Systems Develop-ment. , USA: Addison-Wesley, 1998.

[15] D. F. Cooper, S. Grey, G. Raymond, and P. Walker, Project Risk Man-agement Guidelines. Chichester, U.K.: Wiley, 2005.

[16] G. M. Winch, Managing Construction Projects: An Information Pro-cessing Approach. Oxford, U.K.: Blackwell, 2002.

[17] F. Noonan and H. Thamhain, Identifying and assessing risks forprojects, presented at the Measuring Success, 18th Annu. Sem.

Project Management Institute, Montreal, QC, Canada, 1986.[18] D. B. Ashley and I. Avots, Influence diagramming for analysis ofproject risk, Project Manage. J., vol. 15, pp. 5662, 1984.

[19] R. Charette, Software Engineering Risk Analysis and Management.New York: McGraw-Hill, 1989.

[20] J. M. Carr, S. L. Konda, I. Monarch, F. C. Ulrich, and C. F. Walker,Taxonomy Based Risk Identification 1993, Software Engineer Insti-tute, Carnegie Melon University, Pittsburgh, PA CMU/SE1-93-TR6.

[21] R. Chapman, Thecontrolling influences of effective risk identificationand assessment for construction design management, Int. J. Project

Manage., vol. 19, pp. 147160, 2001.[22] S. Simister, Usage and benefits of project risk analysis and manage-

ment, Int. J. Project Manage., vol. 12, pp. 58, 1994.[23] A. Akintoye and M. MacLeod, Risk analysis and management in con-

struction, Int. J. Project Manage., vol. 15, pp. 3138, 1997.[24] C. Egbu and Z. Serafinska, Attitudes to risk management in diverse

project environments, presented at the COBRA Conf., Greenwich,

U.K., 2000.[25] T. Raz and B. Michael, Use and benefit of tools for project risk man-

agement, Int. J. Project Manage., vol. 19, pp. 917, 2001.[26] T. Lyons and M. Skitmore, Project risk management in the Queens-

land engineering construction industry: a survey, Int. J. ProjectManage., vol. 22, pp. 5161, 2004.

[27] F. D. Patterson and K. Neailey, A risk register database system to aidthe management of project risk, Int. J. Project Manage., vol. 20, pp.365374, 2002.

[28] K. Ayas and N. Zeniuk, Project-based learning: Building communi-ties of reflective practitioners, Manage. Learning, vol. 32, pp. 6176,2001.

[29] D. Hillson, The risk breakdown structure (RBS) as an aid to effectiverisk management, presented at the 5th European Project ManagementConf., Cannes, France, 2002.

[30] C. Chapman, A risk engineering approach to project risk manage-ment, Int. J. Project Manage., vol. 8, pp. 516, 1990.

[31] H. Al-Tabtabai and J. E. Diekmann, Judgemental forecasting in con-struction, Construction Manage, Econ., vol. 10, pp. 1930, 1992.

[32] R. J. Crutcher, Telling what we know: The use of verbal reportmethodologies in psychological research, Psychological Sci., vol. 5,pp. 241244, 1994.

[33] J. Rosenhead and J. Mingers, Rational Analysis for a ProblematicWorld Revisited: Problems Structuring Methods for Complexity,Uncertainty and Conflict, 2nd ed. Chichester, U.K.: Wiley, 2001.

[34] J. Schoemaker, The Expected Utility Model: Its Variants, purposes,evidence and limitations, J. Econ. Literature, vol. 20, pp. 529563,1982.

[35] D. Kahneman, P. Slovic, and A. Tversky, Judgement under Uncer-tainty: Heuristics and Biases. Cambridge, U.K.: Cambridge Univ.Press, 1982.

[36] T. Gilovich, D. Griffin, and D. Kahneman, Heuristics and Biases: ThePsychology of Intuitive Judgement. Cambridge, U.K.: Cambridge

Univ. Press, 2002.[37] D. Kahneman and A. Tversky, Prospect theory: An analysis of deci-sion under risk, Econometrica, vol. 47, pp. 263291, 1979.

[38] G. Gigerenzer and P. Todd, Eds., Simple Heuristics that Make usSmart. New York: Oxford Univ. Press/ABC Res. Group, 1999.

[39] O. Huber, R. Wider, and O. W. Huber, Active information search andcomplete information presentation in naturalistic decision tasks, ActaPsychologica, vol. 95, pp. 1529, 1997.

[40] R. Ranyard, J. Williamson, L. Cuthbert, and E. Hill, Inconsistentdecisions involving risk and uncertainty: Tracing peoples thoughts,

EPSRC research programme on Risk and Human Behaviour, vol. 5,

pp. 711, 1999.[41] J. Williamson and R. Ranyard, A conversation-based process tracingmethod for use with naturalistic decisions: An evaluation study, Br. J.Psychol., vol. 91, pp. 203221, 2000a.

[42] J. Williamson, R. Ranyard, and L. Cuthbert, Risk management in ev-eryday insurance decisions: Evidence from a process tracing study,

Risk Decision Policy, vol. 5, pp. 1938, 2000b.[43] C. Eden, Cognitive mapping: A review, Eur. J. Oper. Res., vol. 36,

pp. 113, 1988.[44] C. Eden and F. Ackermann, Making Strategy: The journey of Strategic

Management. London, U.K.: Sage., 1998.[45] C. Eden, T. Williams, F. Ackermann,and S. Howick, The role of feed-

back dynamics in disruption and delay (D&D) in major projects, J.Oper. Res. Soc., vol. 51, pp. 291300, 2000.

[46] T. Williams, C. Eden, F. Ackermann,and A. Tait, Theeffectsof designchanges and delays on project costs, J. Oper. Res. Soc., vol. 46, pp.809818, 1995.

[47] T. M. Williams, F. Ackermann, and C. Eden, , K. Kahkonen and K.A. Artto, Eds., Project risk: Systemicity, cause mapping and scenarioapproach, in Managing Risks in Projects. London, U.K.: E & FNSPON, 1997, pp. 343352.

[48] G. P. Hodgkinson, Images of Competitive Space: A study in Manage-rial and Organizational Strategic Cognition. Basingstoke, U.K.: Pal-grave Macmillan, 2005.

[49] R. Axelrod, Structure of Decisions, the Cognitive Maps of PoliticalElites. Guildford, U.K.: Princeton Univ. Press, 1976.

[50] J. A. Maule, G. P. Hodgkinson, and N. Brown, , D. Hardman andL. Macchi, Eds., Cognitive mapping of causal reasoning in strategicdecision making, in Thinking: Psychological Perspectives on Rea-soning, Judgement and Decision Making. West Sussex, U.K.: Wi-

leyLtd, 2003, pp. 253272.[51] J. Swan, Usingcognitivemapping in management research: Decisions

about technical innovation, Br. J. Manage., vol. 8, pp. 183198, 1997.[52] J.M. Meziasand W. H.Starbuck, Studying the accuracy of managers

perceptions: A research odyssey, Br. J. Manage., vol. 14, pp. 317,2003.

[53] M. Berenson, D. M. Levine, and T. C. Krehbiel, Basic Business Sta-tistics Concepts and Applications, 9th ed. Upper Saddle River, NJ:Pearson Education, 2004.

[54] A. Agresti and B. Finlay, Statistical Methods for the Social Sciences,3rd ed. Upper Saddle River, NJ: Prentice-Hall, 1997.

[55] A. Edkins, G. Millan, and E. Maytorena, Construction Risk Identifica-

tion: A Methodological Proposal. London, U.K.: Bartlett School ofGraduate Studies, UCL, No. 17, 2003.

[56] C. M. Fiol and A. S. Huff, Maps for managers: Where are we? wheredo we go from here, J. Manage. Studies, vol. 29, pp. 26785, 1992.

[57] W. Feller, An Introduction to Probability Theory and its Applications,2nd ed. New York: Wiley, 1971, vol. 2.

[58] J. Johston, Econometric Methods, 3rd ed. New York: McGraw Hill,1984.

[59] D. Schon, The Reflective Practitioner: How Professional Think in Ac-tion. New York: Basic Books, 1983.

Eunice Maytorena received the B.A. degree inarchitecture from the Universidad Autonoma de

Baja California, Mexico and the Ph.D. degree fromthe Bartlett School of Graduate Studies, UniversityCollege London, U.K.

Her work experience includes architectural designand consultancy, research in various aspects ofthe built environment and lecturing in project riskmanagement. She has served as a Research Asso-ciate on ProRIde: Project Risk Identification. Atpresent, she is a Research Associate at Manchester

Business School, Manchester, U.K. Her current research interests are project

management, risk management and organizational cognition.Dr. Maytorena is a member of Project Management Institute (PMI) and thePMI Risk Special Interest Group.



12/12


Graham M. Winch is Professor of Project Manage-ment in Manchester Business School, the University

of Manchester, Manchester, U.K., and Director of theCentre for Research in the Management of Projects.A social scientist by training, he has run constructionprojects and researched various aspects of innovationand project management across a wide variety of en-gineering sectors. He is the author ofManaging Pro-

duction : Engineering Change and Stability (OxfordUniversity Press, 1992), a study of the implementa-tion of CAD/CAM Systems, Innovation and Man-

agement Control (Cambridge University Press, 1985), a study of new productdevelopment in the car industry, and most recently, Managing ConstructionProjects, an Information Processing Approach (Blackwell, 2002). These arecomplemented by over 30 refereed journal articles, complemented by numerousbook chapters, conference papers, and research reports. Professor Winch has

held numerousESRC andEPSRCawards, andis currently PrincipalInvestigatoron ProRIde : Project Risk Identification and a co-investigator on RethinkingProject Management : Developing a New Research Agenda.

Jim Freeman was educated at the universities ofWales, Bath, and Salford where he received the

B.Sc.degree in pure mathematics and the M.Sc. andPh.D. degrees in applied statistics, respectively.

His work experience includes programming andstatistical lecturing/consultancy. He is currently Di-rector of MBSs M.Sc. degree program in operationsmanagement. In the past, he has been employed as

a Statistician, a Training Adviser and in 1992 wasappointed Visiting Professor at the University of Al-berta,Edmonton, AB,Canada. At present, he lectures

in statistics at MBS where hisresearch interestsinclude gaming, simulation, andadvanced modeling applications.

Dr. Freeman is a member of the Operational Research Society and a Fellowof the Royal Statistical Society.

Tom Kiely He is a project manager by training with a U.K. Higher NationalDiploma (HND). He has 40 years management experience in the constructionindustry. He was Project Manager for the ProRIde: Project Risk Identificationresearch project.

Mr. Kiely is a member of the Chartered Institute of Building.

The Influence of Experience and Information Search

Documents

Transcript of The Influence of Experience and Information Search