Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring...

21
Brugha, C. (1998), "Structuring and Weighting Criteria in Multi Criteria Decision Making (MCDM)", Trends in Multicriteria Decision Making: Proceedings of the 13th International Conference on Multiple Criteria Decision Making, Stewart, T. J. and Van den Honert, R.C. (eds.): Springer-Verlag, p. 229-242. [MCDA] Structuring and Weighting Criteria in Multi Criteria Decision Making (MCDM) Cathal M. Brugha, University College Dublin. Abstract: The implications of qualitative distinctions between multiple criteria are considered. Some contributions to theory about the Analytical Hierarchy Process (AHP) are challenged. Experiments on alternative criteria structures are reported. These suggest that confusing structures are bad, but good structures are better than none. Guidelines on how to develop a structure are given for a well known case of the purchase of a house. It is suggested that differences between decision alternatives should provide a first phase basis for discovering criteria. A criteria tree should be structured ‘top down’ as a second phase by clustering criteria on the basis of qualitative difference. On any level the differences between criteria should follow relatively simple patterns. The rules used suggest the relevance of work on the structure of qualitative decision-making which is determined by Nomology, the science of the laws of the mind. Implications are considered for weighting trade-offs between homogeneous clusters of criteria. This should be done as a later ‘bottom up’ phase. The AHP scoring system is challenged. Some tests of alternative scoring methods are reported. Keywords. Analytic Hierarchy Process, Decision theory, Nomology, Qualitative structuring

Transcript of Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring...

Page 1: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

Brugha, C. (1998), "Structuring and Weighting Criteria in Multi Criteria Decision

Making (MCDM)", Trends in Multicriteria Decision Making: Proceedings of the

13th International Conference on Multiple Criteria Decision Making, Stewart, T.

J. and Van den Honert, R.C. (eds.): Springer-Verlag, p. 229-242. [MCDA]

Structuring and Weighting Criteria in Multi Criteria

Decision Making (MCDM)

Cathal M. Brugha, University College Dublin.

Abstract: The implications of qualitative distinctions between multiple criteria

are considered. Some contributions to theory about the Analytical Hierarchy

Process (AHP) are challenged. Experiments on alternative criteria structures are

reported. These suggest that confusing structures are bad, but good structures are

better than none. Guidelines on how to develop a structure are given for a well

known case of the purchase of a house. It is suggested that differences between

decision alternatives should provide a first phase basis for discovering criteria. A

criteria tree should be structured ‘top down’ as a second phase by clustering

criteria on the basis of qualitative difference. On any level the differences

between criteria should follow relatively simple patterns. The rules used suggest

the relevance of work on the structure of qualitative decision-making which is

determined by Nomology, the science of the laws of the mind. Implications are

considered for weighting trade-offs between homogeneous clusters of criteria.

This should be done as a later ‘bottom up’ phase. The AHP scoring system is

challenged. Some tests of alternative scoring methods are reported.

Keywords. Analytic Hierarchy Process, Decision theory, Nomology, Qualitative

structuring

Page 2: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

1. Introduction

This article develops from work by the author on the structure of qualitative

decision-making (Brugha 1998a, 1998b and 1998c) which is determined by

Nomology (Hamilton, 1877), the science of the laws of the mind. It provides a

basis for modelling the way people might differentiate multiple criteria. Also

relevant to this paper is psychophysics (Gescheider, 1985) which has provided an

empirical basis for the comparison of objects by means of relative measurement,

and is the basis of the Analytical Hierarchy Process (AHP) (Saaty, 1996).

The AHP has been the subject of much controversy, particularly to do with the

question of it apparently being the cause of rank reversal in some circumstances.

One problem that has concerned this author is the use by the AHP of the Right

Eigenvector method to synthesise multiple scores of relatively measured objects.

Crawford and Williams (1985), Barzilai et al (1987, 1992, 1994), Holder (1990)

and Lootsma (1993, 1996) have considered this at length. Lootsma’s proposed

variation, Geometric AHP, is now widely accepted and so this issue will not be

considered here. Two other issues which have concerned this author will be

considered in this article. The first is what this author sees as the inadequate

treatment of qualitative difference between criteria particularly in some

applications of the AHP. There seems to be some fundamental confusion about

what is meant by having multiple criteria. In the next section we illustrate some of

these confusions. We then review some criteria structuring experiments which

were carried out in University College Dublin. We then propose some guidelines

for structuring criteria using a well-known case for illustration.

The other issue considered here is the calculation of the weights of relative

importance of criteria and, in particular, the 1 to 9 semantic differential scoring

system used in the AHP (Saaty, 1980, 1990a). A semantic differential is used to

get a consensus on agreement about the importance of one item compared to

another, or in the AHP case agreement on how different two things are from each

other. This author has been concerned about this strength-of-agreement figure

being used to score the actual weight of that relative importance. Some

Page 3: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

suggestions about the calculation of weights are given. Also some tests of

alternative weighting systems are reported, leading to some further guidelines.

2. Criteria Structuring

The AHP grew out of a need to accommodate qualitative differences between

criteria. Some difficulties with AHP have focused on the quantitative issue of

rank reversal. Saaty (1994), for example, introduced a new form of normalisation,

the ‘Ideal’ form, to avoid rank reversal being caused by the entry of irrelevant

alternatives or numerous copies of one alternative. He also used an argument

about the introduction of copies of a hat, for example, to justify the change in rank

of existing alternatives caused by the AHP model (Saaty, 1994). For instance, if

A is preferred to B and B is preferred to C, possibly the introduction of multiple

copies of C could lead to B being preferred to A, hence rank reversal.

There seems to have been a confusion here about the interaction that occurs

between alternatives and criteria. The above problem arose only indirectly

because of the multiple copies. Clearly the introduction into the mind of the

decision maker of the possibility of copies revealed a previously hidden criterion

based on the issue of uniqueness. We would suggest that, if the other criteria have

been modelled correctly, such a new criterion should be easy to fit into the

existing structure, and the weighting of that issue should be possible to do as an

extension of the existing system of comparisons of criteria.

In general, criteria should be included in a goal modelling process on the basis

of their relevance to choices between the existing alternatives. This can mean that

new criteria can be included or old ones dropped as new alternatives are

considered or excluded. The process is essentially about the interaction between

objectives and alternatives, i.e. between criteria and attributes. The proper

structuring of criteria facilitates their change as the necessity arises.

Page 4: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

2.1 Car choice example

Schoner and Wedley (1989) have proposed ‘Referenced AHP’ and ‘B-G modified

AHP’ to deal with quantitative differences in attributes such as in the case of the

purchase of a car. The attributes which are relevant to the decision to choose

which car to buy were related quantitatively. From this author’s point of view

there was only one criterion, cost. Consequently the AHP should not have been

used. The car was expected to have a life of five years. Hence five years

maintenance was factored in with the price. The cars were expected to do 10,000

miles each year at an estimated cost of $1.50 per gallon. This was also factored

in. Each car had a different price, maintenance cost and fuel usage in gallons per

mile. For any car it is easy to calculate its estimated cost over the five years.

Thus, for any two cars it is easy to calculate their relative costs directly without

reference to the other cars. Applying the AHP in the usual way leads to rank

reversal when a new car alternative is introduced. ‘Referenced AHP’ reinstates

the correct relative relationship between alternatives. Before the new car was

introduced the following were the alternative cars and their attributes:

Car Price ($) Maintenance ($/yr.) Fuel (gal./mi.)

1 14,000 2,000 0.05

2 5,000 4,000 0.03

3 6,000 4,000 0.05

Total 25,000 10,000 0.13

The scale factors are q1 = 1 for price, q2 = 5 for maintenance and q3 = 10,000 × 5 ×

1.5 = 75,000 for fuel costs over the five years. The ratio of cost of car 1 to car 2 is

R1 214 000 5 2 000 75 000 055 000 5 4 000 75 000 03

27 75027

1 018,, , , ., , , .

,,250

.= + ∗ + ∗+ ∗ + ∗

= =

Page 5: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

The vectors of local priorities were produced using the AHP procedure, but

clearly can be viewed from the data. The three cars score as follows on price,

maintenance and fuel:

1425

525

625

15

25

25

513

313

513

���

���

���

���

���

���

The vectors of the weights of importance of the three attributes are calculated

similarly using the AHP and can be seen to be derived from the total costs from all

three attributes over all three cars: $25,000 plus 5 × $10,000 plus $75,000 × .13

equals $84,750. Proportionately these are 0.295, 0.590 and 0.115. In AHP these

weights should remain the same. So, when a new car is included and these

weights are used a conflict can lead to a rank reversal between cars 1 and 2.

‘Referenced AHP’ prevents this conflict happening by restoring the true weights.

The strength of an AHP type procedure is that it can be used to synthesise scores

on qualitatively distinct criteria which might be measured most easily in terms

relative to one another. Not only was there no need to apply the somewhat

complicated AHP procedure to measure a relationship that was already known

quantitatively, it was not appropriate to apply it in this situation because all three

attributes were based on cost. The implication that the AHP procedure is usable in

every multi-attribute situation is more than just wrong, it distracts from its

contribution as a method for synthesising criteria whose multiplicity arises in the

mind of the decision maker.

‘Referenced AHP’ produces scaling factors that convert quantitatively related

attribute scores from AHP tables back into a common score. Because the AHP

should not be used to synthesise qualitatively similar attributes we believe there is

no need to use Schoner and Wedley’s (1989) proposed ‘Referenced AHP’ or ‘B-G

modified AHP’ which they adapted from Belton and Gear (1983, 1985).

Page 6: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

2.2 Farmer’s field example

Schenkerman (1994) shows similar difficulties when the AHP is applied to a field

where the goal is the perimeter and the attributes are the distances on the

rectangular sides North-South (N) and East-West (E).

Unnormalised Normalised

Field N E Perimeter Priority N E Synthesised

A 100 800 1800 0.450 15

815

0.367

B 200 100 600 0.150 25

115

0.233

C 200 600 1600 0.400 25

615

0.400

Total 500 1500 4000 1.000 55

1515

1.000

Prioritisation on the basis of the perimeter using the AHP puts field A first.

Synthesisation using AHP normalisation puts field C first. Schenkerman shows

that the normalisation method causes a difficulty with absolute measurement (such

as this case), and that Referenced AHP, B-G modified AHP, the linking-pin

method and the supermatrix method simply undo the eigenvector normalisation.

In this case, the N and E attributes are brought back to the same dimension by re-

introducing weights of 5 and 15. The resolution to this problem is to recognise

that lengths and breadths of fields are not qualitatively distinct criteria for any

decision maker’s goal. Consequently the conflict implied by the illustration does

not apply.

Page 7: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

2.3 Experiments With Alternative Trees

As part of their course assignments some alternative trees were tested by several

groups of final year students of the Bachelor of Commerce Degree in University

College Dublin. The decisions modelled mainly had to do with the choice they

had to make about what to do the following year. Part of their task was to

compare Naive AHP with Structured AHP, and then to compare both with

SMART. Naive AHP corresponds to the usual approach as in Saaty’s (1990b)

house example (below), which means pairwise comparisons of each criterion with

every other criteria. Structured AHP meant clustering the criteria into a hierarchy

based on qualitative similarities.

One group carried out the process with final year students of engineering on the

issue "What to do next year?" after graduating. In the first phase they identified

the following alternative activities the graduating engineers would consider: get a

job (Job), do a post-graduate course in Ireland (P/G Here), do a post-graduate

course abroad (P/G Abroad), or take a year off (Bum). They also identified their

main objectives as make money (Cash), develop work experience (Work Exp), get

better qualifications (B/Q), an issue to do with pressure and free time (Pres F/T),

and one to do with personal development (Per Dev). They then applied Naive

AHP to those five issues with one group of eight students, and Structured AHP to

another group of eight students, both randomly chosen and tested individually.

The latter was structured with Cash, Work Exp and B/Q clustered together and

called Economic (Figure 1). This author suggested that comparisons on any level

should be between things of a corresponding level of generality. Consequently

PRES F/T was called Personal Feelings and Per Dev was called Personal Growth.

The AHP package Expert Choice was used to synthesise the results. The most

striking outcome was that Pres F/T had a global weight of 0.128 using Naive AHP

for one group and 0.474 using Structured AHP under the heading of Personal

Feelings for the other group. It was felt that there may have been some other sub-

criteria under Personal Feelings which had been left out. Hence a possible benefit

of the structured approach would be that general categories such as Personal

Page 8: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

Feelings might subsume other important but unarticulated feelings. The overall

inconsistency score was 0.19 for the naive approach and 0.07 for the structured

approach. The better score for the latter may have been partly due to having fewer

comparisons, but was more likely due to the structuring.

What To Do Next Year? Economic Personal Personal Feelings Growth Cash Work Exp B/Q Pres F/T Per Dev

Figure 1. Graduating Engineers' Choices

The following is a summary of the comments from all the groups. The Naive

AHP caused a problem when there were many criteria. This led to an excessive

number of questions, a drop off in interest towards the end of the questioning

process and higher inconsistencies. The Structured AHP led to higher

consistency scores. The grouping of criteria helped respondents to become aware

of criteria or attributes that they might have forgotten. The questions were more

specific and more easily understood. However, if the structuring was poorly done

there was a danger of confusion. It required a greater understanding of the issues

in order to do the structuring. Also, the added levels might lead to distortions in

the global weightings.

Page 9: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

2.3.1 Comparison of Three Criteria Tree Structures

In a subsequent running of the course the above test was extended to a comparison

of three alternative structures. Several class groups were given the following task

specification. In the first phase of the study they had to find a homogeneous group

of similar respondents who had a real choice to make, interview them, determine

their three most preferred alternatives, and their four most important criteria. The

rest of this section describes one of the groups which based its project on the

choice a final year college student faces in the year following their departure from

college. The first phase determined the three alternatives to be Employment,

Travel and Postgraduate Study. The criteria were Income, Challenge, Education

and Experience. Three alternative trees were then constructed. The first tree was

comprised of the four criteria in Naive AHP form. Each of the respondents was

asked to group the four criteria into two pairs based on which were closest. Some

formed clusters of Income / Education and Challenge / Experience, others had

clusters of Income / Challenge and Education / Experience. This provided the

second tree. Each was then asked was there any criterion in one of the pairs which

seemed closer to the combined other pair. This provided the basis of the third

tree, as in Figure 2.

Page 10: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

Criterion Tree (i) Income Challenge Education Experience .283 .314 .237 .166 Criterion Tree (ii) .767 1.304 .894 1.118 1.304 .767 Income Challenge Education Experience .190 .237 .361 .212 Criterion Tree (iii) 1.360 .735 1.140 .877 1.304 .767 Income Challenge Education Experience .329 .236 .274 .161

Figure 2. Alternative Criteria Trees

The respondents were asked to rate the different trees, to comment on the

process, particularly the ease of understanding it, and to evaluate the exercise. Of

the nine respondents in this case only one liked the first tree, two liked the second,

and six liked the third. Most of this group isolated Income as the most different as

in Criterion Tree (iii). These results fit some theoretical ideas developed by this

author from his work in applying Nomology to MCDM, which are discussed in

the next section.

Page 11: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

The numbers in Figure 2 come from one of the respondents in this group.

Geometric AHP was used to synthesise the data. The below the line figures are

normalised to total one so as to allow comparison. The figures on the branches of

the trees multiply to one. Weights are distributed downwards so that they multiply

to one at any level. Thus, for Criterion Tree (ii) the global geometric weight for

Income is 0.783 which is got by multiplying 0.894 by the square root of 0.767.

It is apparent from the test that different criteria trees have a considerable effect

on criteria weights. This was true for all nine respondents in this group.

Generally the weights using Criterion Tree (ii) were more out of line than the

other two, as would be expected from the comments. This result confirms what

was discovered in the first test, which is that a poorly structured tree causes

confusion. The next section presents some suggestions on proper structuring.

2.4 Structuring Criteria Trees

A formalisation of some ideas on criteria structuring is now presented using

Saaty’s well known house example (Saaty, 1990b) in which the criteria were size

of house, location to bus lines, neighbourhood, age of house, yard space, modern

facilities, general condition, and financing available. The main idea is that proper

structuring of criteria should take into account any qualitative distinctions which

the decision-maker has identified. The proposed structure is presented in Figure 3.

In this case the financing issue is quite distinct from all the others. Without

finance you cannot get the house. The difference between finance and the other

issues corresponds to the conflict between what is needed to get each house and

the preferences for each one. Some applications of MCDM treat this as a separate

issue to the rest of the problem and use benefit-cost analysis as a second stage

process. This becomes unnecessary with the method described here.

Page 12: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

Within the remaining issues, location to bus lines (transportation),

neighbourhood and yard space form a comparable subgroup; which might be

called surroundings. The other subgroup corresponds to more specifically house

issues. Within the house cluster the size of the house is very different to issues to

do with age of house, general condition and modern facilities.

Generally this qualitative clustering and ordering of the criteria is done in a ‘top

down’ manner after the main differentiating attributes between the alternatives

have been discovered. By distinguishing out the criteria which are very different,

such as financing in this case, one can help to clarify the decision maker’s

thinking particularly. This helps later with trade-offs. A frequent comment from

the students was that having to do this forced them to clarify their thoughts.

Presenting the issues on any level on a qualitatively logical order may also help.

Thus financing comes before new home. Likewise within surroundings one

should begin with the more technical transportation, then neighbourhood which

has more to do with other people and lastly yard space which is more a

description of the situation of the house. The latter two could be interchanged if

yard space meant a play area for children and neighbourhood corresponded to

ambience.

Page 13: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

Satisfaction With House Financing New Home House Surroundings Size Of Maintenance Trans- Neighbour- Yard House And Value Portation Hood Space Age Of General Modern House Condition Facilities

Figure 3. Hierarchy for satisfaction with house

2.5 Relevance of Nomology

Nomology (Brugha 1998a, 1998b and 1998c) offers a basis for understanding the

structures which underlie qualitative difference. It has three main dimensions:

Adjusting, Convincing and Committing. The kinds of problems considered here,

course choice and house purchase, involve a combination of convincing oneself

about various issues and committing oneself to making a choice. Committing has

three levels: need, preference and value. Convincing has three levels: technical or

self, other people and situational. These are presented in Figure 4 along with a

revision of Maslow’s Hierarchy of Needs to show the generic nature of this

structure. Proper structuring of a multi criteria problem should take account of the

nomological rules based on which the criteria clusters are formed. In this case it

provides the basis for differentiating the various levels.

Page 14: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

Consider the structuring that was done for the house choice. The hierarchy in

Figure 3 shows that the biggest qualitative distinction is at the top of the hierarchy

with the issue of financing determining whether or not one can afford the house or

not. The major trade-off is about commitment and the financing the buyers have

against the new home they would prefer. At the second level there is a trade-off

to be made between the house itself and the situational aspect of the surroundings.

At the third level down there is the issue about the size of house one will have

versus the work one has to do in terms of maintenance and adding value to it.

This is a question of commitment of one’s work to the house.

Convincing oneself about the house’s maintenance and value is a question of

age of house having more of a technical aspect, the general condition likely to

have more impact on the people in it, and the modern facilities determining the

type of situation that the people will be getting. Convincing oneself about the

house’s surroundings is a question of trading transportation which is more

technical, the neighbourhood and the people in it, and the yard space which is

situational.

Levels of Convincing Technical Others Situational Self End-users Business / Environment Somatic Physical Political Economic Have / Need Levels of Psychic Social Cultural Emotional Committing Do / Preference Pneumatic Artistic Religious Mystical Are / Value

Figure 4. Structure of Development Activities

Page 15: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

3. Criteria Weighting Criteria arise from the differences between the alternatives. We would propose

that the third phase of the process should start with the comparative evaluation of

the alternatives with respect to the lowest level criteria. Intuitively, it makes sense

to not mix evaluations which are qualitatively quite different and distinct from

each other, for instance, in the case of the house above, to compare general

condition with neighbourhood, or yard space with financing available. We

would propose that the identification of criterion weights within each category and

on each level that is relevant to a choice process should be carried out at the one

time, working from the bottom up, if possible. Then the more macro weighting

process could be done between the synthesised sets. Occasionally this could lead

to only two items being compared at one level. For example, in the house case

there would be only one comparison between house and surroundings, and then

one between financing and new home. This has the benefit of requiring the

decision-maker to answer fewer questions. It might create anxiety that the result

would be over-dependent on single judgements such as on the financing versus

new home question. In our opinion this problem can be dealt with most easily by

using sensitivity analysis. As it was, in the actual application (Saaty 1990b, p.17)

there was some surprise about the emergence of the least desirable house with

respect to financing as most preferred. It is possible that our suggested approach

might have helped in this case.

The benefit of working from the bottom up is that, through working with the

alternatives and the criteria, the decision-maker learns more about what they mean

and so scores them accurately. We would also suggest that it makes more sense to

have decision-makers work firstly with criteria which are qualitatively close and

finish up with those which are very different from each other. These latter are the

more difficult judgements, and the trade-offs most crucial to the decision.

Decision makers should be able to see the synthesised scores of the alternatives

for each criteria cluster as these scores are synthesised. Working interactively can

allow for ‘outranking’ or elimination of alternatives which scored very poorly on

some criterion. It also facilitates sensitivity and benefit cost analyses.

Page 16: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

A possible alternative scoring approach is to compare the importance of the

group of surroundings issues with one another and then extend the comparison to

the individual house issues. This might be difficult because of mixing different

qualitative changes from criteria to criteria. It also makes more demands on the

patience of the decision-maker because of the increased number of questions.

A variation on this would be to order the criteria qualitatively within any level

and carry out comparisons between levels using the criteria that are qualitatively

nearest to each other. For instance, higher criteria within an one category could be

compared with lower criteria within a higher category, hopefully producing a

seamless join. Within surroundings transportation would be lower than

neighbourhood or yard space and so closer to some of the house criteria. Within

maintenance and value modern facilities is a higher criterion than age or general

condition. Thus one would expect it would be relatively easy for a decision-maker

to compare the relative importance of the modern facilities and transportation

criteria. In fact the first is about the convenience within the home and the second

is about convenience of travel to and from the home, two qualitatively similar

issues. Likewise size of house is very close to age of house. From Figure 3 one

can see that a ‘qualitative frontier’ has been created. If one was uncomfortable

about developing clusters of criteria one could compare criteria which were

nearby on the frontier.

Structuring criteria by working from the bottom up the tree in clusters, the first

and recommended method, has operational advantages. The number of

comparisons are few because the clusters are usually small. This facilitates

information gathering and reduces inconsistency. The challenge it presents the

decision advisor particularly is to analyse the structure of the decision maker’s

criteria before gathering the data. The increase in the complexity of the analysis

offsets the operational advantages. The benefits from qualitatively structuring the

objectives come more from the validity of the process, how it is truer to the

processes of the decision maker’s mind.

Page 17: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

Bottom-up evaluation of alternatives may convert the weighting process into

two phases. It may be necessary to convert quantitative scores on attributes of

alternatives into scores which match some criterion. If one of the houses is very

much bigger than the others it might be appropriate to use a utility function to take

account of diminishing returns from house size. Where the yard space of one of

the houses was very inconvenient or wasteful of space one might convert its space

into terms which were expressed in terms of the yard space of another house. The

many modelling possibilities for scoring alternatives on particular attributes

should not influence how one should synthesise the relative importance of

multiple criteria, that is where the distinctions incorporate a qualitative aspect.

In the second set of student tests the criteria were compared using AHP type

relative measurement, while the attributes of the alternatives with regard to each

lowest level criterion were scored using a SMART type utility scale. If adopted

generally, this approach would have the effect of introducing compatibility

between AHP and multi-attribute utility theory (MAUT), as mentioned by Dyer

(1990).

3.1 Naming Clusters

In the first test (Figure 1) and in the house purchase example (Figure 3) the criteria

clusters were given new names. The group running the second test preferred not

to do this because of the fear that it would introduce new issues into the decision.

It is most important that the respondents understanding of the issues be used when

naming clusters. Respondents generally seem to be more comfortable if clusters

are presented in the language they have been using in the process. The greatest

difficulties arose when comparing clusters whose meanings were not

homogeneous and consequently did not facilitate the trade-offs. For example with

Criterion Tree (ii) in Figure 2, comparing Income / Challenge against Education /

Experience was seen as difficult. On the other hand comparing Income against

Challenge / Education / Experience was seen as easier.

Page 18: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

3.2 Experiments With Alternative Forms of Weighting

Some of the tests carried out in University College Dublin involved comparing the

AHP with SMART. AHP’s 1 to 9 scale caused difficulties leading respondents to

re-consider some of their first answers. SMART’s visible scale was liked.

An second experiment was carried out whereby respondents were asked to rank

their criteria from the least preferred to the most preferred. Then they were asked

to fill in an AHP type table with numbers greater than one corresponding to how

much more they preferred one to another, either directly or in percentage terms.

Respondents found this difficult.

This experiment was refined by giving respondents scale bars to look at with the

lower one given a “length” of 100; they were required to fill in the value, over

100, for the one being compared with it. This was welcomed. This author had the

suspicion that it might work well for relative scores in the order of 1 to 1.5 times

the lower value, but might break down for higher values.

The current belief is that scoring the relative importance of criteria from the

highest downwards might more easily deal with large variations in relative scores.

Page 19: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

4. Conclusion

The research described here focused on personally relevant multi criteria

decisions. By emphasising respect for the decision processes within the mind of

decision makers some developments have been made and tested with regard to the

procedures used for constructing criteria trees, for synthesising the weights of

criteria, and for choosing preferred alternatives. Some small tests mainly amongst

college students indicate a good response and enthusiasm for the improvements

from both respondents and students who carried out the tests as part of their course

assignments. Criteria trees that are structured to reflect the various small and large

qualitative distinctions in the minds of the decision makers help decision makers

to clarify their thoughts. They also make scoring trade-offs easier and reduce

inconsistency. The use of visible bands or bars to help score trade-offs was also

seen as an improvement on using numbers alone, including the AHP 1 to 9 scale.

The suggestions proposed here can be summarised as follows. Criteria and

alternatives should be seen as intertwined. Differences between the alternatives

generate the relevant criteria. Consequently including or excluding an alternative

may have a consequence for the criteria. The initial list of criteria generated by

the alternatives should be structured on a ‘top down’ basis from the most

qualitatively distinct to the least. Alternatives should be scored on the lower level

criteria. Then trade-off scores should be calculated on the criteria moving ‘bottom

up’ towards the overall goal. Decision makers should be made aware of the entire

process as it happens. In this way they can revise their judgements where

necessary. Also, occasionally it may be possible to ‘outrank’ or eliminate an

alternative which scores poorly on some clustered criterion. At the end of the

process, if the major issue is a trade-off of benefits against costs, the procedure

takes on the appearance of a benefit-cost analysis.

It is believed that these conclusions apply where the multiplicity of criteria in a

decision is based on qualitative difference, particularly to models such as the AHP

whose foundations are based in psychophysics. The research also indicates the

relevance of Nomology for guiding the qualitative structuring of a criteria tree.

Page 20: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

References Barzilai, J.; Cook, W. D. and Golany, B. (1987) Consistent Weights for

Judgements Matrices of the Relative Importance of Alternatives, Operations

Research Letters, 6 (3), 131-134.

Barzilai, J.; Cook, W. D. and Golany, B. (1992) “The Analytic Hierarchy Process:

Structure of the Problem and its Solutions” in Systems and Management Science

by Extremal Methods, Phillips, F. Y. and Rousseau, J. J. (eds.), Kluwer

Academic Publishers, 361-371.

Barzilai, J. and Golany, B. (1994), AHP Rank Reversal, Normalisation and

Aggregation Rules, INFOR, 32 (2), 5 -64.

Belton, V. and Gear, A. E. (1983), On the Shortcoming of Saaty’s method of

Analytic Hierarchies, Omega, 11, 228-230.

Belton, V. and Gear, A. E. (1985) The legitimacy of Rank Reversal - A comment,

Omega, 13, 143-144.

Brugha, C. (1998a), The structure of qualitative decision making, European

Journal of Operational Research, 104 (1), pp 46-62.

Brugha, C. (1998b), The structure of adjustment decision making, European

Journal of Operational Research, 104 (1), pp 63-76.

Brugha, C. (1998c), The structure of development decision making, European

Journal of Operational Research, 104 (1), pp 77-92.

Crawford, G. and Williams, C. (1985) A Note on the Analysis of Subjective

Judgement Matrices, Journal of Mathematical Psychology, 29, 387-405

Dyer, J.S. (1990), Remarks on the Analytic Hierarchy Process, Management

Science, 36 (March), 249-258.

Gescheider, G.A. (1985) Psychophysics Method, Theory, and Application,

Lawrence Erlbaum Associates, Publishers, New Jersey.

Hamilton, W. (1877), Lectures on Metaphysics, Vols. 1 and 2, 6th Ed., in

Lectures on Metaphysics and Logic, London: William Blackwood and Sons.

Holder, R. D. (1990), Some comments on the Analytical Hierarchy Process, J.

Opl. Res. Soc. 41 (11), 1073-1076.

Page 21: Brugha Structuring and Weighting Criteria MCDM98mis.ucd.ie/Members/cbrugha/pubs/Brugha Structuring and Weighting... · 13th International Conference on Multiple Criteria ... [MCDA]

Lootsma, F. A. (1993), Scale sensitivity in the Multiplicative AHP and SMART,

Journal of Multi-Criteria Decision Analysis, 2, 87-110.

Lootsma, F.A. (1996), “A model of the relative importance of the criteria in the

Multiplicative AHP and SMART”, European Journal of Operational Research,

94, 467-476.

Saaty, T.L. (1980), The Analytic Hierarchy Process, McGraw-Hill, New York.

Saaty, T.L. (1990a), Multicriteria Decision-Making: the Analytic Hierarchy

Process, The Analytic Hierarchy Process Series Vol. 1, RWS Publications.

Saaty, T.L. (1990b), How to make a decision: the Analytic Hierarchy Process,

European Journal of Operational Research, 48, 9-26.

Saaty, T.L. (1994), Highlights and critical points in the theory and application of

the Analytic Hierarchy Process, European Journal of Operational Research, 74,

426-447.

Saaty, T.L. (1996), "Ratio Scales are Fundamental in Decision Making", ISAHP

1996 Proceedings, Vancouver, Canada, July 12-15, 146-156.

Schenkerman, Stan (1994), Avoiding rank reversal in AHP decision-support

models, European Journal of Operational Research, 74, 407-419.

Schoner, B. and Wedley, W.C. (1989), Ambiguous criteria weights in AHP:

consequences and solutions, Decision Sciences, 20, 462-475.