S ystemsAnalysis LaboratoryHelsinki University of Technology
Observations from computer-
supported Even Swaps experiments
using the Smart-Swaps software
Jyri MustajokiRaimo P. Hämäläinen
Petri LievonenSystems Analysis Laboratory
Helsinki University of Technologywww.sal.hut.fi
S ystemsAnalysis LaboratoryHelsinki University of Technology
Introduction
Even Swaps method• Hammond, Keeney and Raiffa (1998, 1999)• Easy-to-use multi-criteria decision analytical (MCDA)
method based on value tradeoffs
Web-based Smart-Swaps software• Procedural support to carry out even swaps
In this study, observations of students using the method with the help of the software
• Do they like and understand it?• How laborious it is felt to be?• Role and importance of the computer support?
S ystemsAnalysis LaboratoryHelsinki University of Technology
Structure of this presentation
Introduction to the Even Swaps process and the Smart-Swaps software
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
Results and evidence
Conclusions and discussion
S ystemsAnalysis LaboratoryHelsinki University of Technology
PrOACT-modelSmart Choices (1999)
Define your decision problem tosolve the right problem.
Clarify what you’re really trying to achieve with your decision.
Make smarter choices by creating better alternatives to choose from.
Describe how well each alternative meets your objectives.
Make tough compromises when you can’t achieve all your objectives at once.
Problem
Objectives
Alternatives
Consequences
TradeoffsSmart-Swapssoftware -this experiment
Introduction to the Even Swaps process and the Smart-Swaps software
+ Uncertainty, Risk profiles, Linked decisions
S ystemsAnalysis LaboratoryHelsinki University of Technology
Even Swaps elimination process
Carry out even swaps that makea) Alternatives dominated (attribute-wise)
≡ There is another alternative, which is equal or better than this in every attribute, and better at least in one attribute
b) Attributes irrelevant≡ Each alternative has the same value on this attribute
» These can be eliminated
Process continues until one alternative, i.e. the best one, remains
Introduction to the Even Swaps process and the Smart-Swaps software
S ystemsAnalysis LaboratoryHelsinki University of Technology
Smart-Swaps softwarewww.smart-swaps.hut.fi
Introduction to the Even Swaps process and the Smart-Swaps software
S ystemsAnalysis LaboratoryHelsinki University of Technology
Example
Office selection problem (Hammond et al. 1999)
Dominatedby
Lombard
Practicallydominated
byMontana
(Slightly better in Monthly Cost, but equal or worse in all other attributes)
78
25
An even swap
Commute time removed as irrelevant
Introduction to the Even Swaps process and the Smart-Swaps software
S ystemsAnalysis LaboratoryHelsinki University of Technology
Supporting Even Swaps with Preference Programming
Support for1. Finding candidates for the next even swap
2. Identifying practically (i.e. almost) dominated alternatives
Both tasks need comprehensive technical screening
Idea: supporting the process – not automating it
Introduction to the Even Swaps process and the Smart-Swaps software
S ystemsAnalysis LaboratoryHelsinki University of Technology
Decision support in Smart-Swaps
More than oneremaining alternative
The most preferred alternative is found
Trade-off information
Even swap suggestions
Practical dominance candidates
Initial statements about the attributesProblem initialization
Updating of
the model
Make an even swap
Eliminate irrelevant attributes
Eliminate dominated alternatives
Yes
No
Even Swaps Preference Programming
Introduction to the Even Swaps process and the Smart-Swaps software
S ystemsAnalysis LaboratoryHelsinki University of Technology
Previous studies of Even SwapsComparison of Even Swaps and MAVT
• Belton et al. (2005): “MCDA in E-democracy. Why weight? Comparing Even Swaps and MAVT.”, TED Workshop, May 19-22, Helsinki
Environmental planning• Gregory et al. (2001): “Bringing stakeholder values into
environmental policy choices: a community-based estuary case study.” Ecological Economics 39, 37-52.
Strategy selection in a rural enterprise• Kajanus et al. (2001): “Application of even swaps for
strategy selection in a rural enterprise.” Management Decision 39(5), 394-402.
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
S ystemsAnalysis LaboratoryHelsinki University of Technology
Thoughts about Even Swaps process
From a cognitive point of view, Even Swaps process has several characteristics that are of a special interest, e.g.
• What kind of swaps the DMs tend to carry out?• How the DMs understand the alternatives with the
revised consequences?• Does the DM end up with the same result following
different paths of even swaps?
Smart-Swaps software available» Easy to study how the DMs carry out the process in
practice
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
S ystemsAnalysis LaboratoryHelsinki University of Technology
Questions of interest in our experiment
In this study, the focus is on:
1) Issues related to carrying out the process with the Smart-Swaps software or (manually) with Microsoft Excel?
2) Issues related to the size of the problem?
3) What are the benefits of using the Preference Programming functionality of Smart-Swaps software, if any?
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
S ystemsAnalysis LaboratoryHelsinki University of Technology
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
Material and methods
• Subjects consisting of engineering studentsas DMs
• Controlled experiments included in courses1) Assignments in applied mathematics
2) Decision making and problem solving
• Questionnaires with open and scaled questions
• Decision process logs
S ystemsAnalysis LaboratoryHelsinki University of Technology
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
Experimental procedure After a brief introduction to the Even Swaps process each DM carried out two decision analytical assignments:
1) The Even Swaps process on a small introductory problem. On this assignment,
A. half of the DMs used Excel manually
B. the other half conducted the process with the Smart-Swaps software
2) The Even Swaps process on a much larger problem. Every DM used the Smart-Swaps software but
A. half of the DMs were instructed to ignore the Preference Programming functionality (‘recommender’)
B. the other half was instructed to utilize it
S ystemsAnalysis LaboratoryHelsinki University of Technology
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
1st: a small introductory problem
You are choosing an apartment. After preliminary elimination there are three alternatives: Lombard, Baranov and Montana. You are in a happy position, as all want you as a tenant. It all boils down to what do you want. You have ended up with four criteria to base your decision on. You gather up a consequences table:
Lombard Baranov Montana
Distance to school
25 min 20 min 45 min
Condition Poor Good Excellent
Size 35 m2 42 m2 23 m2
Rent 350 €/month 450 €/month 250 €/month
S ystemsAnalysis LaboratoryHelsinki University of Technology
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
2nd: a large primary problem
Your company is about to choose the facility for a new office.
After serious thinking you consider eight attributes relevant: a1: size of the officea2: rental costsa3: renovation need a4: car park opportunitiesa5: means of commuting a6: distance to city center a7: other facilities in the neighborhood a8: habitability
After searching for a while you find 12 alternatives to choose from (X1-X12). The consequences are given in a table.
S ystemsAnalysis LaboratoryHelsinki University of Technology
1st
study2nd
studyboth
1. Smallproblem
A (Excel) 10 DMs 6 DMs 16 pers.
B (Smart-Swaps) 10 DMs 14 DMs 24 pers.
2. Large problem
A (without recomm.) 10 DMs 9 DMs 19 pers.
B (with recommender) 10 DMs 11 DMs 21 pers.
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
Overall sample sizes
Small but sufficient for our purposes to get preliminary results.
S ystemsAnalysis LaboratoryHelsinki University of Technology
Hypotheses on the experiment
H1.1: The DMs using the Smart-Swaps software evaluate the results more positively than the DMs using Excel.
H1.2: The DMs using the Smart-Swaps require less time to get the result than the DMs using Excel.
H2.1: The support by Smart-Swaps is evaluated to be more useful in large problems than in small ones.
H2.2: The DMs tend to eliminate rather dominated alternatives than irrelevant attributes.
H3.1: The even swap recommender is evaluated as useful.H3.2: Utilizing the recommender results in fewer swaps.H3.3: Practical dominance propositions are seldom
neglected.
Observations from computer-supported Even Swaps experiments:
Research questions and experimental procedure
S ystemsAnalysis LaboratoryHelsinki University of Technology
DMs arrived to various decisions
Results and evidence
Decisions in the 1st (small) and 2nd (large) problems. Areas indicatefrequencies of subjects that chose certain alternative.
1 2 7 2 9 1 3 5 1
X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12
All DMs (N=31, NA=9)
5 6 29 All DMs (N=40)
Montana BaranovLombard
S ystemsAnalysis LaboratoryHelsinki University of Technology
H1: Smart-Swaps software vs. Excel?
Results and evidence
• The DMs using Smart-Swaps were faster • No significant difference between no. of swaps
• Note that the problem was small
• No significant difference between decisions made
• No significant difference between the opinions on the result
S ystemsAnalysis LaboratoryHelsinki University of Technology
The DMs using Smart-Swaps were faster
Results and evidence
Decision time comparison in the 1st (small) problem. Areas indicate frequencies of subjects that completed the task in certain time.
Smart-Swaps software vs. Excel:
4 1 1 2 7 2 5 1 1 7 3 1 1 1 1 1 1
1 3 1 4 2 1 1 1 1 1
3 1 1 2 4 2 5 1 3 1 1
0 min 5 min 10 min 15 min 20 min 25 min 30 min 35 min
Median
All DMs (N=40)
DMs using MS Excel(N=16)
DMs using Smart-Swaps(N=24)
S ystemsAnalysis LaboratoryHelsinki University of Technology
No significant difference between no. of swaps Number of swaps on the 1st problem. Areas indicate frequencies of
subjects that completed the task with certain amount of swaps.
Results and evidence
Smart-Swaps software vs. Excel:
1 1 1 14 12 1 8 1 1
4 7 1 2 1 1
1 1 1 10 5 6
0 swaps 1 swaps 2 swaps 3 swaps 4 swaps 5 swaps 6 swaps 7 swaps 8 swaps 9 swapsMedian
All DMs (N=40)
DMs using MS Excel (N=16)
DMs using Smart-Swaps (N=24)
S ystemsAnalysis LaboratoryHelsinki University of Technology
H2: Issues related to the size of the problem
• In large problems, it may be very difficult to manually carry out the even swaps process• Smart-Swaps is regarded as helpful in
applying the Even Swaps process also in large problems
• DMs tend to eliminate more dominated alternatives than irrelevant attributes
Results and evidence
S ystemsAnalysis LaboratoryHelsinki University of Technology
The Even Swaps process is seen to apply best to small problems
Results and evidence
Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.
Size of the problem:
2 5 4 3 2
6 6 1 1 1 1
2 1 3 3 6
3 1 2 2 4 4
0 1 2 3 4 5 6 7 Median
The apartment selection problem couldbe solved applying Even Swaps processwith pen and paper (weakly-strongly)(N=16, N/A=4)
The office selection problem could besolved applying Even Swaps processwith pen and paper (weakly-strongly)(N=16, N/A=4)
Smart-Swaps helps applying EvenSwaps process to small problems (e.g.apartment selection problem) (weakly-strongly) (N=15, N/A=5)
Smart-Swaps helps applying EvenSwaps process to large problems (e.g.office selection problem) (weakly-strongly) (N=16, N/A=4)
S ystemsAnalysis LaboratoryHelsinki University of Technology
Elimination by dominance was used more than irrelevance
Results and evidence
Number of all eliminations in the 2nd problem studies are presented in the table.
Size of the problem:
No proposer With proposer
ALL ALL-% A A-% B B-%
Removal 66 11% 53 21% 13 4%
Dominance 345 59% 130 52% 215 64%
Irrelevance 163 28% 58 23% 105 31%
Practical dom. 11 2% 7 3% 4 1%
585 100% 248 100% 337 100%
S ystemsAnalysis LaboratoryHelsinki University of Technology
H3: Benefits of using Smart-Swaps
• The DMs using the Preference Programming functionality (the recommender) were faster
• The DMs using the recommender made fewer swaps
• The recommender was rated high• The DMs using the recommender gave
more positive opinions on the method
Results and evidence
S ystemsAnalysis LaboratoryHelsinki University of Technology
DMs using the proposer were faster
Results and evidence
Decision time comparison in the 2nd (large) problem. Areas indicate frequencies of subjects that completed the task in certain time.
Benefits of Smart-Swaps:
2 1 1 3 6 1 3 1 1 1 3 2 1 2 1 1
2 1 1 1 2 1 1 1
2 1 1 3 4 2 1 1 3 1 1
0 min 5 min 10 min 15 min 20 min 25 min 30 min 35 min 40 min 45 min Median
All DMs (N=30, N/A=10)
DMs without proposer(N=10, N/A=9)
DMs using proposer(N=20, N/A=1)
S ystemsAnalysis LaboratoryHelsinki University of Technology
DMs using the proposer made fewer swaps
Results and evidence
Number of swaps on the 2nd problem. Areas indicate frequencies of subjects that completed the task with certain amount of swaps.
Benefits of Smart-Swaps:
2 1 1 4 3 4 2 1 1 1 1 1 2 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
2 1 1 3 3 4 2 1 1 1 1
0 swaps 10 swaps 20 swaps 30 swaps 40 swaps 50 swaps 60 swapsMedian
All DMs (N=31, N/A=9)
DMs without proposer(N=11, N/A=8)
DMs using proposer(N=20, N/A=1)
S ystemsAnalysis LaboratoryHelsinki University of Technology
Preference programming functionality was rated high
Results and evidence
Benefits of Smart-Swaps:
1 4 7 9
2 4 8 3 2
1 1 3 5
2 2 7
2 4 5
3 4 3
1 1 1 1 6
0 1 2 3 4 5 6 7
Median
I regard proposer as (cumbersome-convenient) (N=21)
I regard proposer as (unreliable-reliable)(N=19, N/A=2)
I regard proposer as (difficult-easy) (N=10)
I used the proposer (not at all - much)(N=11)
I did the swap the proposer suggested(never - always) (N=11)
I used proposer more for (irrelevance-dominance) (N=10, N/A=1)
I accepted practical dominancesuggestions (never - always) (N=10, N/A=1)
Opinions on the recommender functionality of the Smart-Swaps software.Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.
S ystemsAnalysis LaboratoryHelsinki University of Technology
DMs without proposer vs. DMs using proposer
Results and evidence
Benefits of Smart-Swaps:
Differences in opinions between DMs ignoring the recommender and DMs utilizing it.Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.
2 5 3 4 12 7 4
2 5 2 1 5 2
1 3 7 5 4
2 4 3 4 11 7 8
2 3 2 2 6 3
1 1 2 5 4 8
0 1 2 3 4 5 6 7
Median
The method is suitable for problems thathave a large number of attributes - allopinions (weakly-strongly) (N=37, N/A=3)
Opinions of DMs ignoring the proposer(N=17, N/A=2)
Opinions of DMs utilizing the proposer(N=20, N/A=1)
I regard the method as (easy-difficult) touse - all opinions (N=39, N/A=1)
Opinions of DMs ignoring the proposer(N=18, N/A=1)
Opinions of DMs utilizing the proposer(N=21)
S ystemsAnalysis LaboratoryHelsinki University of Technology
DMs without proposer vs. DMs using proposer
Results and evidence
Benefits of Smart-Swaps:
1 1 3 8 1 4
1 1 3 3
5 1 4
2 1 6 7 12 9 1
2 1 4 4 5 2
2 3 7 7 1
2 5 9 10 10 1 1
2 2 7 3 3
3 2 7 7 1 1
0 1 2 3 4 5 6 7 Median
Software helps justifying decisions toothers - all opinions (weakly-strongly)(N=18, N/A=2)Opinions of DMs ignoring the proposer(N=8, N/A=2)
Opinions of DMs utilizing the proposer(N=10)
Method improves my decision making - allopinions (weakly-strongly) (N=38, N/A=2)
Opinions of DMs ignoring the proposer(N=18, N/A=1)
Opinions of DMs utilizing the proposer(N=20, N/A=1)
I trust the method (mildly-strongly) - allopinions (N=38, N/A=2)
Opinions of DMs ignoring the proposer(N=17, N/A=2)
Opinions of DMs utilizing the proposer(N=21)
S ystemsAnalysis LaboratoryHelsinki University of Technology
Conclusions
Our first experiments on the Smart-Swaps software suggest:
• The support provided by the Smart-Swaps software is perceived to be useful
• Preference Programming approach provided considerable help
• A significant difference between the opinions of the group utilizing the recommender and the group instructed to ignore it
» Further studies needed to get a deeper insight of how the DMs perceive the approach in practice
» In addition, we got a lot of usability feedback to help improve the Smart-Swaps software
Conclusions and discussion
S ystemsAnalysis LaboratoryHelsinki University of Technology
ReferencesV. Belton, G. Wright and G. Montibeller (2005). “MCDA in E-democracy. Why weight?
Comparing Even Swaps and MAVT.” Presentation at the TED Workshop on e-Participation in Environmental Decision Making, May 2005, Helsinki. (Downloadable at http://www.ted.tkk.fi/presentations/Belton-TED.ppt)
Hammond, J.S., Keeney, R.L., Raiffa, H., 1998. “Even swaps: A rational method for making trade-offs.” Harvard Business Review 76(2), 137-149.
Hammond, J.S., Keeney, R.L., Raiffa, H., 1999. Smart choices. A practical guide to making better decisions. Harvard Business School Press, Boston.
Mustajoki, J., Hämäläinen, R.P., 2005. “A Preference Programming Approach to Make the Even Swaps Method Even Easier.” Decision Analysis. (to appear) (Downloadable at www.sal.hut.fi/Publications/pdf-files/mmus04.pdf)
Salo, A., Hämäläinen, R.P., 1992. “Preference assessment by imprecise ratio statements.” Operations Research 40(6), 1053-1061.
A. Salo and R.P. Hämäläinen (1995). ”Preference programming through approximate ratio comparisons.” European Journal of Operational Research 82(3), 458-475.
Applications of Even Swaps:Gregory, R., Wellman, K., 2001. “Bringing stakeholder values into environmental policy
choices: a community-based estuary case study.” Ecological Economics 39, 37-52.Kajanus, M., Ahola, J., Kurttila, M., Pesonen, M., 2001. “Application of even swaps for
strategy selection in a rural enterprise.” Management Decision 39(5), 394-402.
Top Related