I • •:-Decision AnalysisTechnical Report No.82-2

1

Decision Analysis: State of the Fieldby

Ralph L. Koeney

March 1982

n-1I

Woodward.Clyde ConsultantsThree Emharcadern Centei SuitP 700 San F r,mir r 0 (,A q4 111

ULAN",

N~~~i Decision Analysis

Technical Report No.82-2

CI

yby

A Ralph L. KeeneyMarch 1982

Woodward-Cld Coslat

T h re e ~ 3ma c d r e t r u t 0 0 a t n i ~ , C 4 1

DECISION ANALYSIS: STATE OF THE FIELD

Ralph L. KeeneyDecision Analysis Group

Woo1'Jward-Clde ConsultanmsThree Embarcadero Center. Suite 7(1(,

San Francisco, California 941 i I

This article, written for the non-decision analyst, describes what uccisimi analysis is. what itcan and cannot do, v" hy one should care to do thi,,. and htow, one does it. In the process. we al'ohope to dispel some myths:

0 decision analysis is a tool of operations research and management science,

a some analvses are objective and value-free.

• it would be de',;rable to have "objective, value-free" anal.ses,

* decision analysis solves decision problems.

0 sonic decision problems are too difficult for decision analvsi!.,

0 decision analvsis and decision theory are the same thing.

To accomplish these purposes, we set the stage b. describing the decision environment. I hen thearticle presents an overview of decision anal\%is and provides additional sources for its fo.undatiions.procedures. history, and applications.

1. THE D)ECISION ENVIRONMENT

The compl,.,xity of the decision environment is greater today than ever. Governmental regu-lations, such as the National Environmental Policy Act and Occupational Safet\ and Health Act.require corporations and governmental agencies to consider and justify the impacts, Informed con-sumers. employees, and shareholders demand greater public consciousness. responsibility, and ac-countability from corporate and governmental decision makers. For example. executives evaluatingpotential mergers or acquisitions mumt consider antitrust and other legal matters. social impacts.and political issues in addition to fin:,ncial . -ects. In appraising potential public programs or theelimination of existing programs. a governmcntal agency i.hould consider not only the multifacetedcosts; and benefits of its options but also the diversity of the population and its sometimes conflictingviewpoints and political concerns. The bottom line is that today's decision problems are character-ized by the following:

High stakes. The difference in perceived desirability between alternatives is enormous. It mayinvolve mia;Ions of dollar,. or severe environmental damage, for instance.

Complicated structure. Numerous complexities (discussed below) make it extremely difficultto appraise alternatives informally in a responsible manner.

No overall experts. Because of the breadth of concerns involved in most important decisionproblems, there are no overall expertf. Different individuals, however, have expertise in dis-ciplines such as economics, engineeri'ig, and other professions which should be incorporatedinto the decision process.

Need to justify decisions. Decisions may need to be justified to regulatory authorities, share-holders. bosses, or the public.

t ICollectively, these characteristics describe many complex decision problems. although the fea-

tures causing the complexity in specific problems may differ. Because the purpose of analysis is toilluminate complexity and provide insight, it is worthwhile to summarize these features. Althoughthese features are intrinsically intertwined, a categorization is as follows:

I. Multiple objectives. It is desirable to achieve several objectives at once. In evaluatingroutes for proposed pipelines, one wishes simultaneously to minimize environmental im-pact, minimize health and safety hazards, maximize economic benefits, maximize positivesocial ir pact, and please all groups of intersted citizens. Unfortunately, all this cannotbe done. but it is important to appraise the degree to which each objective is achieved bythe competing alternatives.

2. Difficultv of identifvi-ig good alternatives. Becaus., many factors affect the desirability ofan alternative, the generation of good alternatives for caieful analysis involves substantialcreativity, In some problems, a good deal of soul-searching is required to identify even asingle alternative which seems possible. let alone reasonable, for achieving the objectivsof the problem.

3. Intangibles, How should onc assess goodwill of a client, morale of a work force, distressat increasing bureaucracy and governmental regulations. or the aesthetic disruption of atelecommunications tower? Although it is difficult to measure such intangibles, they areoften critical factors in a decision,

4. Long-time hori:ons. The consequences of many decisions are not all felt immediately, butoften cover (by intention or otherwise) a long time period, For example. the projectedlifetime for most major facilities is from 25 to 1() years a:',( research and developmentprojects routinely require 5 to 201 years. Future implications of alternatives now beingconsidered should be accounted for in the decision-making process,

5. Man' impacted groups. Major decisions, such as constructing canals for crop irrigation orlegislation regarding abortions, often affect several groups of people, The impacts to thesegroups and their attitudes and values differ greatly, Because of these differences, concernfor equity contributes to the complexity of a problem,

6. Risk and uncertainty. With es:rentially all problems. it is not possible to predict preciselythe consequences of each alternative, Each involves risks and uncertaintie:--an advertis-ing campaign may fail. a large reservoir may break, a government reorganization mayresult in an unwieldy bureaucracy, or a new product 'wuld turn out to be an Edsel, Themaj.or reasons for the existence and persistence of the!,. uncertainties include: (1) little orno datw can be gathered for some Lvents, (2) some data are very expensive or time con-suming to obtain. (3) natural phenomena such as earthquakes and droughts affect impacts.(4) population shifts affect future impacts. (0) priorities, and hence perceived impacts.change over time, and (6) actions of other influential parties, such as government orcompetitors, ar'! uncertain.

7. Risks to life and limb, A general class of critical uncertainties concerns the risks to lifeand limb. Numeroas personal and organizational decisions affect the likelihood that ac-cidents or "exposure" result in fatalities or morbidity. Examples include decisions abouthighway maintenance, foods and drugs, toxic or hazardous materials, birth control. len-iency toward criminals, and whether to walk or drive somewhere, It is not an easy task toinclude such dire consequences in an analysis, but it is certainly a part of many decisionproblems.

8. Interdisciplinary substance. The president of a multinational firm can not be professionallyqualified in all aspects of irnternational law, tax matters, accounting, marketing, produc-tion, and so on. Qualified rprofessionals should supply the relevant inputs on these keyfactors in a major decision,

2

F

9, Several decision makers. ('.ne player rarely holds, all the cards with resppect to a majorddecision. Several plaver,, r ho ma. or may not be on the sain1 lam. contrrol crucialaspects in the overall decis,.on-making proce., Tl, begin production and marketang opcr-ations in a new geographical area, corporate managemiet rnaN require appros ol fromstockholders, several rcgulatory agencies, cornmunit. zoning boards, and perhaps e'enthe courts, The potential actions of other players must be considered when a corporationevaluates its strategic policy,

10. Value tradeoffs, Imporant decisions involve critical value trad',.1,its to mndicate the relati\ edesirability between environmental impacts and economic costs today, immediate socialcosts versus future social benefits, negative impacts to it small group verus smaller posi-tive impacts to a larger group, and sormetimes the value of a human life versus the benefit',generated by a hazardous technology.

11. Risk attitude. A firm operating with the status quo strategy ma' forecast small and dccli'h-ing profits in the next few years. Changing to an inno\ati\e strategy mam has\e a chanceof resulting in substantially\ hih'r ,rofits, but have a risk of Ioses or cvcn bankruptc\.Even if the likelihoods of the various consequences are known, crucial value judgment"about an attitude toard risk are essential to appraise the ap!-,priiteness of acceptingrisks necessarily accompanying each aliernalise.

12. Sequential naturC of decisions. Rarely is one decision completcl. uncoupled from otherdecisions. Choices today affect both the alternatives availabec in the future and the desir-ability of those alternatives. Indeed, many of our present •.•,iices are important becatueof the options the\ open or close or the information the\ provide rather than because oftheir direc! consequences.

Complexity cannot be avoided in making decisions. It is part of the problems. not onl. partc~f the solution process, There are, ho" ever, options concerning the degree of formnality used toaddress the complexity. Near one extreme, this ma\ be done ;ntuiti\el.\ in a rather informal manner.Near the other extreme, formal models can be used to capture as, much of the complexity as possiblc.In any case, the process of obtaining and combining the available inftormation is a difficult task thatrequires balancing all the pros and cons aw well as recognizing the uncertainties for each alternati\c .

2. WHAT IS DECISION ANALYSIS

Decision analysis can be defined on different l.h'vels. Intuitivchl, I think ot decisioon analysis as"a formalization of common sense for decision problems which are too comple., for informal use ofcommon sense." A more technical definition of decision analysis is "a philosophy, articulated b\ aset of Ikgical axioms, and a methodology and collection of s.\stematic procedure.ý, based upon thoseaxioms, for responsibly analyzing the complexities inherent in decision problems."

The foundations of decision an.,.iysis are provided by a set of axioms stated alternativel\ invon Neumann and Morgenstern 11947). Savage 11954], and Pratt, Raiffa. and Schlaifer 11964]. andthe Appendix of this article. These axioms, which provide principles for analyzing decision proh-lems, imply that the attractiveness of alternati\,..:, should depend on ( 1) the likelihoods of the pos-sible consequences of each alternative, and (2) the preferences of the iecision makers for thoseconsequences. The philosophical implications of the axioms arc that all decisions require subjectivejudgments and that the lik.lihoods of consequerices anJ their desirability should be separatelyestimated using probabilities and utilities respectively. The technical implications of the axioms irethat the probabilities and utilities can be used to calculate the expected utility of each alternativeand that alternativcs with higher expected utilities should be preferred. The practical implication ofthe decision anaiysis axioms is the provision of a sound basis and general approach for includingjudgments and values In an analysis of decision alternatives. This permits systematic analysis in adefensible manner of a vast range of decision problems.

3

)Decision analysis focuses otn aspects fundamental to all deciion probleni., nancl, ,, ..

A~~~, fIjctv s

I, a perceived need to accomplish some objectives, TIC2, several alternaiv.s, one of which must be selected.

3. the consequences associated with alternatives ire different, • , .

"4, uncertainty usually about the consequences ol each alternative

5. the possible consequences are not all equally valued... [The decision problem is decomposed into parts, which are separately analyved and integrated withthe logic of the decision analysis axioms to suggest which alternative should be chosen. This "diihdeand conquer" orientation is almost essential for addressing interdisciplinary problem.b. The met h-odology of decision analysis provides a framework to combine traditional techniques o. operation,research, management science, and systems ami lysis with professional judgments and values in a

unified analysis to support decision making, With the procedures of decision analyis. models (e.g.,economic, scientific. operations research). available data, information from samples and tests. andexperts' knowledge are used to quantify likelihoods of various consequences of alternatives in termsof probabilities, Utility theory is used to quantify the values of decision makers for these conse-quences.

Decision anal'.sis is not a tool. Decision analysis is not simply a tool or a niethodolog., al-though it is both sometimes used and perceived as such. Tools of disciplines such as operationsresearch and management science often assume that both the alternatives and objectives of a prob-lem are exogenously provided to the study, Queuing theory, inventor\ theory, and mathematicalprogramming, for example, concentrate much more on analyzing given alternatives with prescribedobjectives, This is of course critically important. However, for less structured decision problems,

more time is required to create the alternatives and articulate the objectives before analysis canoccur, Decision analysis is designed for such problems, Another important difference which distin-guishes tools from decision analysis is the existence of fundamental axioms to provide a philosoph-icallv and theoretically sound foundation from which its resulting methodology is developed,

Objective, value.free anialysis is not possible or desirable. A comment sometimes heard fromanalysts. governmental authorities, and managers of organizations is that what is rcally needed tohelp decision makers is objective, value-free analysis, Simply stated, there is no such thing as ob-jective or value-free analysis. Furthermore, anyone who purports to conduct such an analysis isprofessionally "cry naive, stretching the truth, or using definitions of objective and salue-free whichare quite different from those commonly used. Professional judgments and value judgments areabsolutely necessary in essentially every step of analysis in order to address the complexities ofdecision problems. Objective, value-free analysis would be undesirable because it would simplyavoid the problem. What is needed is logical, systematic analysis that makes the necessary profes.sional and v..lue judgments explicit and combines these with the "objective" data for the problem.The resulting analysis should be responsive to the decision maker's needs and justifiable to others,

Decision anal' sis is uniquely a methodology which provides for such analysis.Decision analaysis is prescriptive in nature. Prescriptive decision analyses are conducted to in-

dicate which alternative should be chosen to be consistent with the information about the problemand the values of decision makers. This can be contrasted with descriptive studies which attempt todescribe how and perhaps why a particular decision was or will be made. Descriptive studies provideuseful information (e.g.. about cognitive processes on how a competitor might behave) for prescrip-tive analyses, but by themselves are not prescriptive decision analyses.

It has been clearly demonstrated that individuals often do not make decisions in a mannerconsistent with the decision analysis axioms (see. for example, Kahneman and Tversky 119791).However, many of those same individuals find the axioms compelling for prescribing their evalua-tiuln of alternatives. The fact is that in complex decision environments, many decision makers preferto act in accord with the decision analysis axioms and yet seriously violate them in selecting alter-natives without the benefit of a decision analysis. This is a strong motivation for the prescriptiveappeal of the approach.

Decision o'nah'isis doe.P not solve probh'm., Decision analysis will not solve a decision problem,nor is it intended to. Its purpose is to produce insight and promote creativity to help decision makersmake better decisions. It does this by providing a methodology and procedures t) decompose theproblem into parts that can be meaningfully analyzed, a logic to integrate the parts. and documen-tation for supporting a decision to others. No analysis includes everything of importance in a deci.sion problem. In selecting an alternative the decision makers should jointly %cigh the implicationsof an analysis together with other factors not in the analysis.

This odientation simukaneously implies that decision analysis is not up to the task of solvingany decision problem, but that it is appropriate to all, Of course it is not worth analyzing ever\problem. More difficult decisi,,, problems are naturally more difficult to analyze. This is true re-gardle.s of the degree to which formal analysis (i.e. use of models as a decision aid) or intuitiveappraisal (i.e., in one's head) is used. However, as complexity increases, the cfficac\ of the intuitiveappraisal decreases at a more rapid rate than that of formal anal.,sis. Thus, roughly speaking. itmay' be more useful to analyze 6(0 percent of a difficult problem than 90 percent of a simplerprioblem.

There is another critical factor which relates to the complexilt, of the decision problem. Ascomplexity increases, the percentage of the problem which can be captured by "hard data" de-creases. Simultaneously, the role that values, professional judgment, and experienlc must neces-sarily play in the decision process increases. We do not have data bases for the possible conse-quences of a particular merger, the overall impacts of "rescuing an industry." the "true" probabilitvof low probability-high consequence events, the price and availahilit\ of oil in 199(0. or the value ofthe environmental, economic, and social consequences of an oil shale program. Yet decision, in-volsing such factors will necessarily continue and are crucial to everyone. Of all analytical meth.odologies, only decision analysis provides the theory and procedures to address these directl\ andincorporate them into a decision problem. It does not provide "the answers" or "the solution ," butit does address the right questions.

Decision analysi,% and deciion theorv art not it'e san•e. Broadly interpreted, decision theory isthe logical foundations of decision analysis and the technical implications wkhich follow. Decisiontheory does not include the techniques or skills for structuring decision problems or assessment ofprobabilities or utilities. The more common interpretation of decision theory is a sampling theor.involving statistical problems (see Waild 119501, Savage 119541. and Raiffa and Schlailer [1I•96)).This narrow focus of decision theory plus the common misunderstanding that decision analysis anddecision theory are essentially the same has led to a misinterpretation about the breadth of problems

to which decision analysis is relevant,

3. THE METHODOLOGY OF DECISION ANALYSIS AThis section presents an overview of the methodology of decision analysis. It is clearly not

possible to delve into too much detail. Books by Raiff, 11968). Schlaifer [i196). Tribus [1969]. iWinkler [19721, Brown et a), [1974], Keenev and Raiffa 11976). Moore and Thomas [1976), Kaul-man and Thomas 11977), LaValle 11978]. and Holloway 11979) provide more details on variousaspects of the methodology. Our purpose is to indicate its general thrust, with emphasis on thoscaspects unique to decision analysis.

For discussion purposes, the methodology of decision analysis will be decomposed into foursteps:

I 1. structure the decision problem.

2, assess possible impacts of each alternative,

3. determine preferences (values) of decision makers. and

4. evaluate and compare alternatives.

Figure I illustrates the interdependencies of the steps and indicates where the components of corm-plexity introduced in Section 1 are addressed. To interpret the implications of these steps, it is

Upeiyaeses

!5

I

important to keep two facts in mind. First. one iterates among the various steps. Not only whatshould be done in one step but how it should be done can be affected by preliminary results fromanother step. Second, decision analyses concentrating on some steps almost to the exclusion ofothers are often appropriate and useful. Such considerations are mentiond in more detail in Section4 on the practice of decision analysis.

St,:p I-Structure the Decision Problem

Structuring the decision problem includes the generation of alternatives and the specificationof objectives. The creativity required for these tasks is promoted by the systetmatic thought pro-cesses of decision analysis.

Decision analysis captures the dynamic nature of decision proccsses. It prescribes a decisionstrategy that indicates what action should be chosen initially and what further aciions should beselected for each subsequent event that could occur, For instance, a decision stralegy might suggestan initial test market for a new product and then, based on the results. either cancel the product.initiate further testing, or begin a full scale marketing and sales effort, Thus. in describing thealternatives, one must simultaneously specify the decision points, events that may occur betweenthem, and the information that can be learned in the process. This dynamic structure can conven-iently be reiesented as a decision tree (Raiffa [1968]).

Two major problems are associated with generating alternatives. First. there may be a largenumber of potential alternatives, many of whic', are not particularly good. However. early in theinvestigation of the decision problem, it may be difficult to differentiate between the good alter-natives and those which are eventually found to be clearlv inferior. In such circumstances, inferioroptions can be identified by screening models which use assumptions too crude for a final evaluationbut sensitive enough to weed out the "bad" alternatives. These models analyze a simplified decisionproblem by using deterministic rather than probabilistic impacts. dominance or "almost dominance"rather than a complete objective function. and constraints. This has the effect of eliminating alter-natives so the deci-ion tree is pruned to a manageable size. Then. more time and effort can beexpended to carefully appraise the remaining viable alternatives,

Step 1 Structure the Step 2 A•sLss Possible Step 3 Determine Preler Step 4 Evaluatp arnd CoirDecision Problem Impacti of ences of D.cisiOn Lrari Altetnatrve

Alternai-rves Makers

t Determine Magnitude

Generate Proposed and Likehhood ofAlternatives Impacts of Proposed

Alternatives

"fva;uate ProposedSpecify Objectives Structure and .'J Alternatives anti

Ouantify Va.rjes of Conduct Sensitivityan d A t t r ib u t e , [ D e c is io n M a k e r s J A n al y s i s

COMPLEXITY COMPLEXITY COMPLEXITY

a Multiple Objectives a Long-Time Horizons o Several Decision Makers

a Difficulty in Identifying * R and Uncertainty a Value TradeoffsGood Alternatives a Risk to Life and Limb i Rislk Attitude

a Intangibles e Interdisciplinary Sub.

* Many Impacted Groups stance

a Sequential Nature ofDecisions

a Figure 1. SCHEMATIC REPRESENTATION OF THE STEPSOF DECISION ANALYSIS

6

A second major problem associated with generating alternatives is that sometimes there seemsto be a complete lack of reasonable alternatives. In this case. it is often worthwhile to utilize theobjectives of the problem to stimulate creativity. Basically. if the objectives are clearly specified.one can describe possible consequences of the problem which seem particularly desirable. Then c ,eworks backward and asks what types of alternatives might achieve such consequences. The processof quantifying the objectives with an object~ve function (i.e., a utility function as discussed inStep 3) promotes additional thinking about worthwh' , alternatives. The result of such a process isoften a broadening of alternatives, which is actuallý d broadening of the decision problem. Forinstance, a significant increase in local crime may result in a "premature" decision that more policeare needed. An analysis may then be initiated of alternatives differing only in the number of addi-tional police. However, the problem is presumably much broader. The objective would likely be tominimize crime or to minimize specific impacts of crime. From this perspective, one may createalternatives involving additional police equipment (e.g.. cars, communications), different operatingpolicies with existing personnel and equipment, community action programs to report "suspicious"activity, or the reduction of early release programs for hard-core criminals in jails. A critical changeis often the introduction of dynamic alternatives rather than reliance on static alternatives alone.The difference is that a dynamic alternative is designed to be adapted over time based on externalcircumstances that occur and information that is learned.

The starting point for specifying objectives is the creation of a rather unstructured list ofconcern-, indicating anything of interest about possible consequences of the alternatives. These needto be organized into a set of general concerns. For instance, with many problems involving sitinglarge-scale facilities, the general concerns may be environmental impact, economics. sociocco-nomics, health and safety, and public attitude,;. To determine specific objectives, the question is.for example. what are the environmental impacts of concern for a particular problem. The proccsvof answering such questions is essentially a creative task. However. previous studies on rel!atedtopics and legal and regulatory guidelines should b..! ot significant help in articulating nhji'.c..Also. for problems which require external revi,-". the Ppotentii d rc itwers (i.c., intcrvcnors. shart-holders, or concerned citizens) may conuibutc useful ideas for objecti'ves.

From all of this infoim:ation, an .;hjt'fi1,c icrrci; :.houid enierge Nith hiboad oc',jiA%.!pertaining to general concerns at the top of the hierarch\ and more detailt'd Ob.lCtiVCS '-U "l-c.

down. Tile lower-level objectives e,,entially define Ow nuii-tnn, .- ! higherIc', .e ;x'ti',,. ihclower-level objective- ar1 On',;;. -, liighel-Ie0ei !-11. 1o0es in the hicrir li~ can, tit: identih'o.and filled by follov. ini' ".aii-emMd, reli.tionship';

For cael of the towet!-lcvel objectives in tht' hierarhy. -,;e need it) identil attributes it,meamure the degree to which the' ob ective i,; achieved Sometimes this is easy. For example. anobvious attribute for the objective -maximize profits" is milliuns ot dollars (why not think big?).However, it is more difficult to determine an attribute for ;a . ., . like "minimize visual deg-radation." This often requires constrLcting an attribute to measuie Oith o jective using proceduressuch as those in Keeney [1981].

Let us now introduce notation to concisely describe our problem structure. We have generateda number of alternatives A,, j = I J...., and an objectives hierarchy with n lowest-level objectivesO,i i = ... n, where n may be one. With these lowest-level objectives would be associated attri-butes X, i I..... n. Furthermore, define x, to be a specific level ef X,, so the possible impact ofselecting an alternative can be characterized by the consequence x U (x,.x, ..... x,). An example ofan objective 0, is "maximize the local economic benefit" and an associ•'ed attribute X, may be"annual local tax paid," A level x, could then be $29 million.

The first step of decision analysis addresses several complexities discussed in Section 1. Themultiple objective feature is addressed by specifying O1 to 0. Some of these objectives concernthe implications to various impacted groups so this feature is also considered. The intangibles areincluded by using objectives such as "minimize aesthetic disruption" and. of course, significanteffort is focused on the complexity of generating viable dynamic alternatives.

7 ~ tr~~r--

Step 2-Assess Possible Impacts of Alternatives

In this step of decision analysis, we wish to determine the impacts of each alternatike, If itwere possible to precisely forecast these impacts. we could associate one consequence with eaLhalternative. Then the evaluation of alternatives would boil down to a choice of the best conse-quence. Unfortunately, the problem is usually not so easy because of uncertainties about the even-tual consequences. Therefore. for each possible alternative, it is desirable to determine the set ofpossible consequences and the probabilities of each occurring. This can be done forinall\ by deter-mining a probability distribution function P1(x) over the set of attributes for each alternative A,. Insome cases the uncertainty associated with in alternative may he small. Then, an appropriate sim-plification is to omit the uncertainty for that alternative. Because one can treat p, in general toinclude cases with no uncertainty (where pj(x) assigns a probability one to a particular x and zeroto all others). we will use p, throughout.

When feasible, meaning that both general knowledge about tne prol~lem structure and thescope of the project allows it. it is desirable to determine probabilities of possible consequenceswith the development and use of formal models. These models typicall. utilize the traditionalmethodologies of operations research, management science, systems analysis. simulation. planning.and the sciences and engineering. Complex models canl often be constructed to have several com-ponents, each pertaining to knowledge associated with a single discipline or organizational unit.For instance, in a decision analvsis by Smallwood and Morris [1980] to examine whether to build anew manuficturing facility, a model had components concerning the market for the proposed prod-;ict n-imin,'nannc. production. capital costs, the competition. and the financial ir.mpact to the corn-pany. Expcl i•: cach of these substantive areas could then provide information on their respectiveplil.r o! the problem. Hence. these models allow one to break the assessment into manageable partsand combine the parts to determine p*

When a model is utilized. cithcr deterministic or probabilistic information is required to spec-ify model inputs in order to determine appropriate probability distributions over model outputs(i.e.. consequences). When a model is not appropriate, information is necessary to directly deter-mine possible consequences. In both cases, such information must be based on the analysis ofexisting data. data collected specifically for the decision problem, or professional judgment. D'ataanalysis is common to many disciplines other than decision analysis so, although it is important. itwill be passed over here. The quantitative assessment of professional judgments or probabilities isa unique aspect of decision analysis discussed below.

There are several methods for quantifying probabilities (see Winkler 11967a]. Spetzler andStadl von Holstein [1975], and StaIl von H-ilstein and Matheson [19"9]). One method is to use astandard probability distribution function and assess parameters for that function. For example, theparameters of a normal distribution can be the mean and standard deviation. Another technique.referred to as a fractile method, involves directly assessing points on the cumulative probabilitydensity function. Suppose y is the single dimensional parameter of interest and we wish to assessthe probability density function p(y). One is asked fot a level y' such that the probability is p' thatthe actual level is less than y'. This questioning is repeated for several probabilities such as p =0.05, 0.25, 0.5, 0.75, and 0.95, Alternatively one can ask for a probability p" that the y-level is lessthan y", By fitting a common probability distribution to the assessed data. one obtains p(y). A thirdprocedure for assessment is appropriate when the possible impact is categorized into a number ofdistinct levels. The professional familiar with the subject is asked to specify directly the probabilityof each level, These assessments may' sound easy, but in practice they are involved processes withmany potential sources for error (see. for example. Tversky and Kahneman [1974. 1981]). However,recent experience suggests that professionals with training can formulate probabilistic forecasts in areliable manner (see Murphy and Winkler 119771).

A factor which can increase the complexity of impact assessments is probabilistic dependenciesamong attributes for given alternatives. If two attributes are probabilistically dependent. the impactspecified for one will affect the assessed impact on the other, When there are such conditionaldependencies, it is almost essential to either model these dependencies and develop probabilisticassessments using t~e output of the model or to bound the possible probability distributions utilizinglogic and understanding of the problem (see. for example. Sarin [1978] and Kirkwood and Pollack

8

[1980]). Then one can investigate whether and how the dependencies influence the evaluation ofalternatives. When such dependencies turn out to be important, additional effort to better charac-terize them may be appropriate.

A host of additional difficulties can occur when more than one expert is asked for professionaljudgments about the same events. These experts may have different opinions, and yet it may bealmost impossible to find out the reasons for the differences. And the experts likely formulate theirjudgments based in part on the same experiments and data sources. so they are not independent,Still, the decision maker may desire a single coherent representation of the uncertainty in theproblem. Recent contributions by Morris [1977] and Winkler [1981] address this problem, which isone area of current research in decision analysis.

Specifying probability distributions addresses the risk and uncertainty aspects of the decisionproblem. in describing the possible impacts. the time in which consequences might occur should beindicated. Thus. the feature of long-time horizons is addressed in this step. The interdisciplinar.substance is also included by utilizing the skills of the various discipline-. to develop and structuremodels, provide information and professional judgments relevant to the discipline, and appraise theresults of the model about possible consequences concerning the disciplinary substance.

"r..n 3--Determine Preferences (Values) to Decision Makers

1, would likely be impossible to achiee the best level with respect to each objective in aderision problem The question is. "How much should be given up with regard to one objective toachie-e a specified improvement on another?" The issue is one of value tradeoffs. For decisionproblcr-s with either single or multiple objectives, it is rarel\ the case (except in simple problems)that oni ,.'ernative is guaranteed to yield the best available consequence. There are usually circum-stanc,.s that could lead to undesirable consequences with any given alternative. The question is,"Are the r,'tential benefits of having things go right worth the risks if things go wrong?" This issue, about &ilk attitudes. Both value tradeoffs and risk attitudes are particularl. complicated becausetoerh are no riht or wrong values. 3assally, what is needed is an objective function which aggre-;,,ties all ýih , in'iividuai objectives and an attitude toward risk. In decision analysis, such an objectiveiunction is referred to as a utility function, symbolically written u, Then u(x), the utility of theconsequence x, indicates the desirability of x relative to all other consequences. As mentioned inSection 2, following directly from the axioms of decision analysis, alternatives with higher expected(i.e., average) utilities should be preferred to those with lower expected utilities.

This step. rather unique to decision analysis, involves the creation of a model of values toevaluate the alternatives. This is done in a structured discussion with the decision makers to quantifyvalue judgments about possible consequences in the problem. The procedure systematically elicitsrelevant information about value tradeoffs, equity concerns, and risk attitudes with provision forconsistency checks. In addition to the obvious advantage of providing a theoretically sound mannerto evaluate alternatives, the explicit development of a valtue model offers several other advantages,including indicating which information is of interest in the problem. suggesting alternatives that may

have been overlooked, providing a means to calculate the value of obtaining additional information,and facilitating concise communication about objectives among interested parties. In addition, asensitivity analysis of the value judgments can be conducted to appraise their importance for theoverall decision.

The process of determining the utility function can be broken into five steps: (1) introducingthe terminology and ideas, (2) determining the general preference structure, (3) assessing single-attribute utility functions, (4) evaluating scaling constants, and (5) checking for consistency andreiterating. For decision problems with a single objective, only Steps 1. 3, and 5 are relevant. Inpractice there is considerable interaction among the steps although each will be separately discussed.

Introducing the Terminology and Ideas. rhe basic purpose of this step is to develop a rapportand an ability to communicate with the decision maker or decision makers. It should be stated thatthe goal of the assessment process is to end up with a consistent representation of preferences forevaluating alternatives. "he analyst should make sure that the decision makers are comfortablewith the assessment procedure and understand the meaning of each attribute and the objective it is

9

meant to measure. If the decision makers have not been closely inolved in defining the attributesor describing the impacts of alternatives, this phase of communication is particularly important.The decision makers shoold understand that there are no correct or incorrect preferences and thatexpressed preferences can be altered at any, time.

Determnining The General Preference Structure. Here, one structuies preferences with a modelindicating the general functional form of the utility function u(x1 ..... xn). To obtain the structure formultiple objectives, on,' uses value independence concepts in the same way that probabilistic in-dependence is utilized in structuring models of impacts. Most of the independence concepts concernrelative values for consequences with levels of a subset of the attributes fixed. The independenceconcepts ar. used to derive a simple function f such as

U(Xl .....X ) --" f[U X ).....un(Xn),k .... k ..... k ]where the u, are single-attribute utility functions and the km 'ire scaling constants. Specific functionalforms following from various assumptions are found in Fishburn [1964, 1965, 1970]. Meyer [1970],Farquhar [1975]. Keeney and Raiffa [1976], Bell [1977b, 1979b], Tamura and Nakamura [1978]. andFarquhar and Fishburn [1981]. Using (1). the overall utility function is determined by assessing thevingle-attribute utility functions and the scaling constants which weight various combinations ofsingle-attribute functions.

A related approach to model values for multiple objectives involves building a value functionv(x1..... ,x,), which assigns higher numbers (ie,, values) to preferred consequences, This is done ina spirit akin to (1) using either single-attribute value functions or indifference curves together withscaling constants, Then a utility function is assessed over value providing u[v(x)] which incorporatesvalue tradeoffs in v and an attitude toward risk in u, Models of value functions addressing multipleobject.ves are found in Debreu [1960], Koopmans [1960]. Luce and Tukey [196A]. Krantz [1964].Krantz et al. [1971], Dyer and Sarin [1979]. Kirkwood and Sarin [1980]. and Keelin [1981], Acommonly used value function is discounting of cash flows over time at a fixed rate. Boyd [1973]and Keeney and Raiffa [1976] discuss procedures to obtain both v(x) and u(v).

Assessing Single-Attribute Utility Functions. Procedures for assessing single-attribute utilityfunctions are well developed. In summary, one wishes to first determine the appropi, :- risk atti-tude. For instance, for consequences involving profits. one is said to be risk averse i' irofit level(xi + x.)/2 is always preferred to a lottery yielding either x, or x. each with a probability of 0.5, Inthis case, one prefers the average of the profits x, and x2 for sure rather than risk a half chance ofthe higher and a half chance of the lower. When one is risk averse, it must be the case that thecorresponding single-attribute utility function is concave, As discussed in Pratt [1964], special riskattitudes restrict the functional form of single-attribute utility functions. A common utility functionis the exponential utility function

u'kx) =- a + b-" p2

where a, b>0, c>0 are scaling constants. This utility function is referred it. as constantly risk aversesince it is the only one consistent with the following property. If x3 for sure is indifferent to a 0.5chance at either x, or x,, then x3+e must be indifferent to 0.5 chances at either x, + ( or x. +F forall possible r

To specify the scaling constants a and b in (2). one arbitrarily sets the utility corresponding totwo consequences, This is similar to defining a temperature scale by selecting a boiling and a freez-ing point. The utilities of all other consequei..es are relative to the two chosen for the scale. Tospecify the appropriate numerical value for a constant c in (2). one can identify both a lottery anda consequence which are equally preferred by the decision maker. For instance, suppose the decisionmaker is indifferent between the certain consequence x3 and a lottery yielding either x, or x. withequal chances of 0.5. Then, to be consistent with the axioms of decision analysis. the utility of x3must be set equal to the expected utility of the lottery. Hence,

u(xi) = 0.5u(xi) + 0.5u(x2). (3)Substituting (2) into (3) and solving gives us the value for parameter c.

Evaluating Scaling Constants. With multiple objectives, the same concept is utilized to deter-mine scaling constants, which relaie to the relative desirability of specified changes of differentattribute levels. To illustrate this in a simple case, consider the additive utility function

u( x. ) k,ujx,), (4)i-i

10

7T"- -- "-;,

where k,, i= 1 ,n are scaling cons:ants. For this additive utility function, th'e values of the k,indicate the relative importance of changing each attribute from its least desirable to its "iost desir-able level. To assejs these scaling constants, one generates data reprLsenting stated value judgmentsof the decision rraker. For instance, the decision maker may be indifferfent between (x1 ..... x.) and(x1,....x). Then the utility of these two consequences, since they are indifferent, must be equal.They are set equal using (4) which yields an equation with the scaling factors as unknowns, Usingsuch indifferences, one generates a set of n independent equations which is solved to detwrminevalues for the n unknown scaling factors. The equations can be generated by sequentially consid-ering consequences which differ in terms of the levels of only two attributes. This significantlysimplifies the comparison task required of the decision makers. More details about the assessmentof utility functions can be found in Fishbuia [1967j, Huber [1974], Keeney and Raiffa [1976]. Bell[1979a]. and many other sources.

Checking consistency. It has been my experience that invariably there are inconsistencies inthe initial assessments. In fact, this is one of the main reasons for the procedure, because once theseinconsistencies are identified, decision makers upon reflection can alter somL of their responses toreach consistency and better reflect their basic values, Furthermore, they seem to f.el better afterhaving straightened out their value structure in their own mind, Thus, it is essential to ask questionsin different ways and carefully reiterate through aspects of the assessment procedure until a consis-tent representation of the decision maker's values is achieved. Conducting sensitivity analysis of the

evaluation of alternatives (Step 4 of decision anaiysis) may suggest if the utility function is a goodenough representation of decision maker values.

With multiple decision makers, as discussed in Harsanyi !1955], Fishburn [IQ73], or Keeneyand Raiffa [1976], additional value judgments -,,re required to address the relative importance ofthe different decision makers and the relative intensity of the potential impact to each in urder todetermine an overall utility function. Alternately. the decision problem can be analyzed from :heviewpoints of the different decision makers by using their own utility functions. It may be that thesame alternative is preferred by each decision maker. possibly for different reasons, In any case. itmight be helpful to eliminate dominated alternatives. identify the basis for conflicts, and sugges:tmechanisms for resolution.

This tiwird stzp of decision analysis uses value judgments to address the complexi ;es concern-ing value tradeoffs, and a risk attitude outlined in Section 1. The value judgments are made explicitin assessing u for each decision maker. This process of building a model of values corrcspondsprecisely with that used for an\ model, We gather tome data (the decision maker's judgments), anduse the data in a generic model (the utility function u) to calculate its paramaters (e.g., the kn's in(1) and c in (2)). Additional value judgments are ne-essary to structure values of multiple decisionmakers into one coherer* utility function.

Step 4-Evaluate and Compare Alternatives.

Once a decision problem is structured, the magnitude and associated likelihoods of cor,nse-quences determined, and the preference structure established, the infu, marion must be synthe',izedin a logical manner to evaluate the alternatives. :t follows from the axioms of decision analysis thaithe basis for this evaluation is the expected utility Ej(u) for each Alternative Ai. wnich is

Ej(u) = fpj(x)u(x)dx. (5)The higher Ej(u) is. the more desirable the alternative. Thus the magnitudes of Ej(u) can be usedto establish a ranking that indicates the decik,n maker's preferences for the alternatives. It shouldbe remembered that the expected utility associated with an alternativ' i: directly related to theobjectives originally chosen to guide the decision and reflects the degree of achievement of theobjectives. One can transform the Ej(u) numbers back into equivalent consequences to obtainSinformation about how much one alternative is preferred over another.

It is extremely important to examine the sensitivity of the decision to different views aboutthe uncertainties associated with the various consequences and to different value structures. This isconceptually easy with decision analysis, since the impacts and values are explicitly quantified withprobability distributions and the utility function, respectively. Without quantification it would be

11!

difficult to conduct a thorough sens: ity analysis. A useful way of presenting the results of asensitivity analysis is to identify sets of conditions, in terms of uncertainties and preferences, underwhich various options should be preferred.

4. PRACTICE OF DECISION ANALYSIS

The ultimate purpose of decision anilysis is to help decision makers make better decisions.The foundations, provided by the axioms do not "assume the problem away). Even though thetheory and procedures are straight-forward, a price isIpaid for attempting to address the complexi-ties of a decision problem explicitly. The implementation phase, that is putting the methodologyinto practice, is more involved compared to other forms of analysis. A significantly greater portionof the overall effort in decision analysis is spent generating alternatives, specifying objectives, elic-iting professional and value judgmei,'1. and interpreting implications of the analysis. Each of theserequires interaction with the decision makers and individuals knowledgeable about the problemsubstance. Structured creative thinking is demanded and sensitive information is elicited.

In this section, we suggest how to conduct a decision analysis and the art cf interaction nec-essary to elicit information. Several uses of decision analysis in addition to evaluating alternativesare indicated. Finaidy some key potential pitfalls are identified.

Conducting a Decision Analysis

A careful definition of the decision problem is essential. For complex problems. an adequatcdefinition is rarely available at the time the analysis is to begin. Yet, it is tempting to begin analyzingthe problem immediately. What is available at the beginning is a somewhat vaguely perceived notionof problem objectives and possible alternatives. Defining a problem means the following: generatingspecific objectives with appropriate attributes and articulating dynamic alternatives including pos-sible information to be learned in the decision process. The attributes indicate what information iswanted about the alternatives, namely the degree to which the alternatives measure up in terms ofthe attributes.

If the utility function is assessed to quantify ihe decision maker's values, this will indicate therelative importance of gathering different information. That is, Step 3 of a decision analysis canproceed before Step 2 (see Figure 1). This is often useful because structuring a decisinn problemand assessing values require only personal interaction which is much less expensive than field tests,equipment, and surveys often necessary to quantify the impacts of the alternatives. Knowing whatinformation to collect may reduce this burden or at least focus it on the information desired. Oneother point is worth mentioning in this regard. There is one value structure for a decision problemsince each alternative is to achieve the same objectives. There are possible impacts to be assessedfor each alternative. Thus, concentrating on the values first and thoroughly may save time. effort,and money on a decision analysis, and result in more useful insights for the problem.

Once the decision problem is well-structured, the collection of information should proceed asindicated in Step 2 of Section 3. The procc:-s may be complicated because of problem substance orrequired personal interaction. The former situation is not unique to decision analysis and will notbe discussed further.

The Art of Decision Analysis Interaction

A key to successful decision analysis is the interaction of decision analysts with thL, !ecisionmakers and other professionals working on the project. As with all forms of personal interaction,there is a great deal of art and skill required. Most of the skills required to be a successful memberof any group are also necessary to be a successful member of the team analyzing the decisionprocess. However, because of the nature of dcc'sion analysis, we will note a few special aspectsrelated to that interaction process.

12

Decision analysts obtain clearly articulated (often quantitative) information about the prob-lem structure, possible impacts. paranmeters for a model, and value judgments, In addition to thecomplexity of the problem substance, the practice of obtaining such information is sometimes dif-ficult because:

1. the information may be sensitive,

2. the natural procedures to process the information in one's mind often result in biasedjudgments,

3. the respondent may have a vested interest in misrepresenting information.

The decision anaiyst should be aware of any of these three possibilities.In a recent article, Fischhoff [1980] draws an analogy between decision analysis and psycho-

therapy. Decision analysts try to formalize the thinking and feelings that the decision maker wishesto use on the problem. By clarifying and even quantifying the process, these thoughts and feelingsare potentially opened for review by others (e.g., bosses, regulators. courts). In any assessmentprocess, one should take the time and use any available devic.zs to establish a rapport with therespondent and to make him or her feel comfortable. I always point out that the reason for theanalysis is that the problems are too difficult tn informally analyze consistent!v. Hence. a majorpurpose of these processes is to identify inconsistencies in the unassisted thinking of the respondent.It is critical to assure these individuals that they will have a first right to adequately review yourwork. Furthermore, they should have the option of changing their responses. This helps to ensurethat no misrepresentation of their judgments occurs. What this boils down to is the need to buildtrust between the decision analyst and al! respondents working on a decision problem, The estab-lishment of this trust must be the responsibility of the decision analyst.

Tversky and Kahneman [1974, 1981] have identified many biases that indi.'iduals may inad-vertently utilize in providing professional or value judgments. It is probably safe to say that thesebiases occur with any procedure, tormal or informal, to assist in the decision making process, Withdecision analysis which focuses on such issues. reasonable procedures have been developed withenough consistency checks to avoid or at least identify the major biases which may be influencingthe particular analysis. Many professionals. including and Winkler [1967b]. Slovic and Lichtenstein[1971], Hogarth [1075], Spetzler and Stadl von Holstein [1975]. Fischer [1976. 1979]. Seaver. Ed-wards, and von Winterfeldt [1978]. and Alpert and Raiffa [1981]. have compared various ap-proaches to examine their strengths , nd weaknesses for such assessments.

A more difficult issue for the analyst might be that of potential confic*. A decision makerwho wishes that a particular product be produced and marketed may be motivated to overestimateits potential sales. A product manager being evaluated on meeting a specific goal may care tounderestimate the potential sales during the goal setting process. To assist in identifying such con-flicts, aside from one's knowledge of the position of individuals with respect to the problem, severaltechniques are used to reduce the conflicts.

Effects due the sensitive nature of decisior information, inherent conflicts, and unconsciousbiases, can be reduced by using four devices: iteration with consistency checks, assessments withdifferent individuals, decomposition, and sensitivity analysis. Information should be gathered usingredundant lines of questioning, and resulting inconsistencies should be investigated until consistencyis achieved. Then, there is some comfort that the major discrepancies are eliminated. Use of judg-ments about the same factor obtained from different qualified individuals has obvious virtues. De-composit'on involves dividing the assessment into component parts and obtaining judgments on thecomponents. For instance, in addition to asking the product manager about profit from the product,ask component judgments about product manufacturing costs, distribution costs, potential sales atvarious prices, pricing policy, and competitor actions. Different individuals should provide theseinputs which would then be utilized to provide estimates of profit. Sensitivity analysis can identifyoroblem elements which are crucial for the evaluation of the alternatives. It is only for these thatsignificant effort is necessary to appraise the recommendations of the analysis,

I

S • •- " • .. . • • =••' "i. •''' t• •' " •" '•2 ',., -,i,•.,.£-., " ••.., 2• ',.7•. ..'" • " •:• • " "'!13,,

It

Uses of Decision Analysis

As previously mentioned, the overall use of decision analysis is to provide insight to improvedecision making One key manner of deriving this insight is to evaluate the alternatives. This is ofcourse common to most prescriptive analytical approaches. However, decision analysis has othercrucial uses to prov.de insight.

A strength of decision analysis is that one can readily calculate the value of additional infor-mation (see LaVale [1968] and Merkhofer 119771). This is done by defining and evaluating alter-natives which include the costs of gathering specific information and the likelihoods of what thatinformation will be. For example, a test market for a proposed new product may cost one milliondollars and the results may indicate potential annual sales anywhere between 20,O() and 5(XlX)()sales per year. If the "test market" alternative has a higher expected utility than the "no test market"alternative, it is worthwhile. By raising the cost of the test market, we can find the cost where thesetwo alternatives art indifferent. This cost is referred to as the value of the test market informationand indicates the rmaximum one should pay for that information. Using this basic idea. Gilbert andRichels [1981] analyze the value of uranium resource information for U.S. energy policy decisions.

Because of the focus on problem complexities, there are many useful byproducts of decisionanalysis. The framework of decision analysis promotes honesty by providing the opportunity forvarious independent checks and centers communication on crucial problem features. For instance,one often develops a clear understanding of the substantive issues of a problem in the process ofstructuring the objectives hierarchy. This also has the effect of sensitizing different individuals tothe issues and perhaps bringing about a commonality of understanding of what the problem is or atleast a common set of terms to discus the pro.lem. Also, creative alternatives can be generated bystimulating thinking based on the problem objectives.

Finally, decision analysis can be very important in conflict identification and risolution. Itshould indicate whether conflicts among various individuals conctrn the possible impacts or thevalues for these impacts. Furthermore, conflicts may only involve certain objectives in either case.Once conflicts are identified, attention can be concentrated on their resolution by examining thebases for judgments of each individual concerned. It may be that only parts of the individuals' basesdiffer and these parts are the reason for the conflict. Information might be gathered which wouldresolve such a conflict. However, there are irresolvable conflicts, such as justifiable differences invalues. For these cases. identification of the basis for the conflict may in itself be an importantcontribution to- ard a more responsible decision.

Man), decision analyses oo not need to be complete. Partial decision analyses which givecursor)y qualitative attention to some steps in Section 3 are definitely appropriate for many decisionproblems. These partial analyses should focus on the aspects of the overall pioblem where insightmight be most fruitful to the decision makers. Once the problem is structured or the impacts ofalternatives clarified or the values articulated, the rest of the analysis may be easy or even unnec-essary. In these partial analyses. the unique contribution of decision analysis is often the proc:-duresto address explicily the softer parts of the problem-its structure and professional and value judg-ments.

Pitfalls of Decision Analysis

Decision analysis is subject to the same pitfalls as other approaches designed to assist decisionmakers. One might categorize these pitfalls as follows:

41. weak or no logical or theoretical foundations.

2. lack of consideration of subjective and value components of the decision problem,

3. a claim that analysis provides a solution to the decision problem,

4. poor analysis,

5. weak personal interaction skilK.

S"14

WI

As previously stated, the foundation% of decision analysis are strong and the subjective and valueaspects of the decision problem are addressed. Hence, specific pitfalls under categories 1 and 2 arerarely the downfall of a decision analysis.

Category 3 represents a pi :fall often more common to decision analysis than other approaches.Because decision analysis do,.s try to capture a bigger share of the ' real problem," there is a ten-dency to assume the entire problem is addressed. Worse though is the misrepresentatior, that such

I, an analysis provides a solution to the decision problem. Decision atdalys:. indeed any anailsis, onlyfocuses on part of a problem 'nd this should be under,,tood.

Poor analysis or poor personal init raction can of course, render thc best conceivcd decisionanalysis as worthless. Rath,,: than repeat all the th ngs that could go wrong here, it may bc moreappropriate to refer to Keene\ and Raiffa [19721 for a short critique of decision analysis or toMajone and Q,!,mde [1980] for an entire volume on pitfalls of analysis.

S. APPLICATIONS OF DECISION ANALYSIS

Discussions of early applications of decision analysis in the oil and gas industry are describedin Grayson [1960] and Kaufman [1963], Applications also occurred in other fields, However, lorproprietary rmasons, manry of the completed decision analyses do not appear in the published liter..ature, Fortunately, the essence of sonic of these analyses do appear in the form of "fictitious"analyses or case studies. Magee [1904a.b] describes applications to capital investment decisions.Howard [19661 discusses a product introduction, and a number of cases representing experiences ofthe early 1960's are found with analyses in Schlaifer [1968]. Papers by Brown 11970] and Longbottomand Wade [1973] surveyed applications of derision analysis through the 1960's.

The 1970's saw an expansion in applications of decision analysis, The applications concernedboth privat, industry and governmentai decisions. They involved new product decisions, researchand development efforts, medical problems. energy problems., environmental alternatives, and stan-dard setting, to name a few, In this article, it would not -,, possible to survey all of ,hese appli:a-tions, Hence we will simply attempt to indicate souices of some applications which are readilyavailable. Many of these sources describe other applicatioi'.;.

There have beer, many applications of decision analysis addressing various corporate prob-lems. Although many of these are proprietary, there are some published examples of these corpo-rate decision ana~yses. Spetzler [1968] describe•s the procedure of assessing a utility function for atcorporate board, Matheson [1969] summarizes an application concerning the in'roduction of a newproduct. The book by Brown. Kahr and Peterson 11974] descrIbes several applications, Keeney11975] discusses the assessment of a multiple objective corporate utility function to examine cor-porate policies, Keefer and Kirkwood [1978] discuss an application to optimally allocat. an oper-ating budget for project engineering. A recent application described in Smallwood and Morris[1980] considers whether Xerox Corporation should construct new manufacturing facilities for anew product and when this should be done, Stillwell et al. [1980] report the evaluation of creditapplications,

Rather than discuss selected applications in medical fields, it is simple, to refer readers to arecent annotated bibliography of decision analysis applicatious b\ Krischer [1980]. The applicationsaddress such diverse problems as evaluating governmental programs to save lives, the evaluation ofnew drup,:, the selection of medical technologies for advanced mr.lical systems, analyses to selecttreatment strategies for numerous different diseases or ailments. the development of on-line com-puter systems to assist physicians in decision making. and the development of various health indus-tries.

VTher'e have been numerous applications of decision analysis to problems faced by variousbranches of government over the last decade. Examples of these include the possibility of seedinghurricanes threatening the coasts of the United States (Poward et al. 11972]). metropolitan airportdevelopment in Mexico City (de Neufville and Keeney 1i972]), protection from wildland fires (Northet al, [1975]), trajectory selection for the Mariner Jupiter/Saturn project (Dyer and Miles [1976]).and the evaluation of busing alternatives to achieve school integration (Edwards [19801), Several[I more recent applications of decision analysis to governmental problems concern selection of stan-

15I

cdards. Examples are emission control strategic- b. North and Merkhoter 1197()1. chronic oil dis-charge standards by von Winterfeldt [I19-821. and the negotiation of international oil tanker stan-dards by UW'ila and Snider [1980t].

Bv their nature, significant environmental problems concern both government and industry.In the reent past, there have been a large number of decision analyses addressing, such environ-mental problems. Examples are the work of Gardiner and Edards 119751 concerning developmentwithin the areas under the juris,diction of the California (oastal Commission. kork in~olving Bell11977a] and Htolling [19781 concerning conroI ofla 'orest pest, the anal,,i,, of marine mining optionsby Lee 11979], and thv evaluation of rcghnal n',,itonmental systemns b\ Seo and Sakawa [1979].

The area with perhaps the greawest nu~mber of applications in recent \cars has been ernerg..There have been decision analywses of the United States synthetic fuels policy (Synfuels hlntravnL-vTask Force 119751) and nuclear reacto: program (Manne and Richel, I 1I97X]1. expanion of theCalifornia electrical system capacitN (Judd 11978]), management of nuclear waste (Ladh•rop andWatson 119821). and commercialization of solar photovoltaic systems (Boyd et al. [1982]). Therehas beer considerable effort focused on) alternatives faced by the utilit.\ industry. Thes.. inclade theselection of technological alternati\hes for specific projects such as transmis,,ion coridu•Loe. (Craw-ford et al. [1978]). the examination of the implications of both over- and under-eapacit, (Cazalctet al. [1978], the siting of energy facilities (Keeney and Nair [1977], Keency [198ttb], and Sarin11980]. and the choice between coal and nuclear technology for large-scale power plants (Be e.\ etal, [19•81]),

6. HISTORY OF DECISION ANALYSIS'

It is difficult to ti-ice decision analysis from its beginning to the prc:,cnt because of the evolu-tionary nature of both its content and its name. The foundations of decision analysis are the inter-wined concepts of subjective probability and utility, and Ramsey [1931] was the first to suggest atheory of decision making based on these two ideas. Two centuries earlicer, Bernoulli [1738] wrotea remarkable paper on the moti\ ation for the concept of utility and on a possible form for a utilityfunction. For a historical discussion of the early development of subjective probability and utilitytheory, see Fellner [1965]. On the uncertainty side. DeFinetti [19371 contributed greatly to thestructure of subjective probability. Modern utility theory for decision making under unccrtaintv wasdeveloped, independently, by von Neumann and Morgenstern [1947]. They, postulated j set ofaxioms simila: to those in the Appendix (using only objective probabilities) aid demonstrated i.,:a utility could be assigned to each coosequence in a manner such that tl'e decision maker shouldprefer the alternative with the highest expected utility in order to act in accord with the axioms.This result is often referred to as the expected utility hypothesis.

Wald [1950], in his classic work on statistical decision problems, uscd theorems of game theoryto prove certain results in statistical decision theory. Although he used an expected-loss criterioninstead of utility theory, it was only a minor modification to introduce utility into the Waid frame-work. This work highlighted a critical problem. namely, how to account for irforrnal inbormationabout the states of the world in his model. The school of statisticians and deciion theorists, includ-ing J. Marschak. H. Chernoff, and H. Rubin, advocated the use of judgmental probability as onemethod of tackling the statistical decision problems proposed by Wald. The pioneering \-ork ofBlackwell and Girshick [1954] contributed to the integration of utilitiL, i'nd subjective probabilitiesinto a coherent program for handling these problems. Then Savage [1954]. in a major contribution.provided a rigorous philosophical fotndation and axiomatic framework for the approach.

Once the theory was developed, many individuals began applying it to mathematically well-structured problems involving uncertainties and possibilities for sampling or cxperimentation. Theseresults, building on the previous work of others, formcd a body of results known as Bayesian orstatistical decision theory (Schlaifer [1959], Raiffa ;'nd Schlaifer 11961]. Pratt et al. 11965]). Whenin the early 1960's these same individuals and th.ir associates. mainly at the Harvard BusinessSchool, began using these theories on real business problems involving uncertainties, whether or

'This section is liberally adapted from Keeney 119781.

16

.it sampling at. experimentation were possible, an adjective Aas added to yield applied statisticaldecision theory. Howeve., since applied statistical decision theory was rclesant to broad classes ofcomplex decision problems (see Schlaifer 119691), it was better to have a more application-orientedname, ard the term decision analysi. Appeared in the literatuie (Howard 1196•6t.

Over the past thirty years, the contributions of many people concerned with the behavioralaspects of decision making have had a significant impart on prescriptive decision analysis. Mostellerand Nogee 11951]. Friedman and Saviage 11952]. Edwards 11954]. and Davidson et al. 11957) made

g early contributions to the assessment of preferences and judgments. Excellent sources of this w-orkare Slovic and Lichtenstein 119711. I'versky and Kahneman (1974. 1981]. Hammond et al. 1198(l],and Einhorn and Hogarth 119811.

Procedures have been d-3kveloped to better account for specific oh:•racl-istics of decision prob-lems mentioned in Sec ',n I Examples include work by Pratt [1964) and Schlaifer 119091 on as-:.essing utility function.,; by Winkle; [1969, 19811, Edwards 11908). Schlaifer 11969). Spetzler andSta6l von Holstein 191•51, and Morris I1977) on assessing probability disteibutions: by Arro\ 11963],[ Haranyi [1955] and Keeney and Kirkwood 119751 on group preferences; hy Fishburn [1904, 1905.19741, Pollak [19671. Raiffa (1969], and Boyd (1973] on multiattribute preferences: by Koopmans119601, Lancaster 119631. Meyer [19771, and Bell 11977b] on preferences over time: by Rousseauand Matheson (19671, Schlaifer [1971], Keeney and Sicherman (1976], and Seo et al. 119781 onde,,eloping software systems for structural and computational assistance: by Miller et al. 11971].Jungetmann 119801], and von Winterfeldt 119S0] on structuring decision problems- and numerouscontributions of people in statistics, stochastic processes. systems analysis and computer science todevelop better probabilistic models, In many analyses concerning. for example, liquefied naturalgas, nuclear power. and hazardous wastes, a critical issue is the value of the lives which may be lostdue to either an accident or "normal" use. Sumý of the current methodological development indecision analysis concerns critical value judgments such as the value of human lite (see. for example.Bodil. (1980], Howard 11979]. Keeney 11980a], and Pliskin ct al, (1980]).

7. RESEARCH

The techniques and procedures of decision analysis are sufficiently developed to make sub-stintial contributions on many complex decision problems. Compared to other approaches. bothtormnai !!nd informal, decision analysis often has much to offer. Compared to "providing all theinsight -i decision maker could possibl\ want at a price too low to refuse," there are significantimprovements which could be made. Research on the following topics will help lead to these im-provements. This research is categorized by the steps of decision analysis outlined in Section 3,

Regarding structure of the decision problem, better approaches to develop objective hierar-chics, create alternatives, and interrelate the two are needed. The approaches should be systematicto increase the likelihood that important parts of the problem are not being omitted. Proceduresare needed to help identify relevant stakeholders-individuals or groups with an interest in theproblem-and ensure that their objectives and advocated alternatives are recognized by the anal\-sis. Likewise. more systematic procedures are needed to identify exogenous events not under thedecision makers control which could significantly influence the consequences of alternatives. Thisshould reduce the likelihood of unanticipated conusequences. Any analysis excludes many aspectsfelt to be less relevant than included ones. Yet, in1anv decision problems are interrelated eventhough at some level it is impractical to include these interrelationships in detail in an analysis.Better means to address the interrelationships are needed in decision analysis. As a specific exam-pie, a carefully defined attribute to measure flexibility in a decision problem might address thedegree to which alternatives in a given problem might foreclose options in other decision problems.

With possible impacts, two major problems deserving additional research concern probabilis-tic dependencies and multiple experts. Better methods to identify probabilistic dependencies andto elicit subjective probability distributions with probabilistic dependencies would be helpful. Onmany important problems, different experts disagree about the impacts exnected from various al-ternatives. Research is needed to prnvide methods to reduce such discrepLncies when appropriate

and to produce responsi ble represe itii tions it fhee tilt cc scudegmeit Iin eaciisc hetc discrcld-cies persist.

Mlodels of' values could bc imipro\ cd Iin thi cc signifit.ant rcspccts. I irsi. models to hctti Owl-haacicri/c the valIues oft at .rou p arc nccedcd. Thc pr otCss iequ ires JUdgn111Ilt t abti'Ut he intcnismit 5oprefere nces for group nicmbc rs and ilhc ii ic at isc milport ar.c Iin OIL: viup Ini addinon. tveItcrnod.'ls to cs aluatc morhidit\ and 1or1011,11 ncqun1\ ofICJ~CIC deiionICI11 prob~lems ssouklo he hcfpullAduitional rcscatch onl structuring and clikciting pe vicfczcnce toil i1pa1cW11 Mel 111tnw. csCýICiLl\ f'ilnornmonctar\ imlpacts. cOUld maY:kl. a subsIMtatia contribuition

progranms to assist Iin comipleting tilc tasks oft dec.isIkIIIMion I I.\' Is tin benII CII II)IC I; p ntIio n nII.IIC*I, IIrils i Indk

procedurc% ito rcalize tile full potentia~l hcncfits tif decision ,nls

8. SUMMNARY

Deccision anlakski cmbodics a philosophfi, someik vonccptS. Mnd anl 10poaf to fornrkill~ adsvstcniaticafls exmitniti a decision problem.ii It is inl not \\it\ a substliltut for crcatisv c. IOinn1aiI\cthinking. but rather it prtutcts and utIli/Cs SU;Jl Cifoits ito prosLI id impor1itant Ilsivirts !tilt I pioll-1cmi. I'llilosoph icall~ I decision anafsskis Mieis onl the basis th,11 thc' dcsir,ihilit of anl 11,ternaals c110h,11dcpcnd onl iso Concerns: thce likelihood that thlt ,1'tcrmitise ss l lead it' \ar OuIcoscqjuClcnc andthle dccW~Itionakcr's prcfcre ni.c for t10S hocCOrscqjUCnccs MI )esiOn anald\Nis Iddr csSCs 1I hcsk C0oncroIscpartiiteh, anld prosidcs tilt It.gic .111l~ techniqucs fot integlIiitin them.

Th e foundations tif decision anals \,,I arc aI sctil ilof riotn s ilch dfcinic tile baste cons~tructs,judgnicntal probabilit\ aind Utilit\. anld OLw logic of thc lpprwoacvh I-toim thcsC aimonis. ih1C fund.i-mental rcsult ofl decision maknasiS is dIcris d: the alVfteraIsL: %k ilt 111w hjiLhCst e~pVcctd 1111111\ shouldbc thc most prcferrcd, 1. anid mans oilier mdi' duals, find thesc. aiwots, voiipcflitig for anlw\/ini.u

*decision problems to presuribChill fIa tIht1% alcn ti a dccsionl nIIAke s1OUld Choose. It is im11portant tolnote that :in\ dccisi on making approach \00hih is noi consisicnit %kill. alll Of tile aimonts of dvccýisoianalksis must . b\ d(initiiitoll \st illat at least 0onc oft thcntl

Jo (inc ssho believcs inl tile aiomis. thc sitindwiJ fot corrcctncess inl formai'l anllling decisionlproblcnms miust tic decision analssis. 'This docs, not meanl other approiichcs arc not Mote aippropt 1a1icIin some eases. but that tilc additional assutlipt ions nCccssar\ for thcse othict apptoachics arc irnpottant to undcrstand and appraise. )ftcn. hitos r. othcr ccrimngf\ comipct ime appr-ioaclics arc notinl conflict \01 it ilte amioms oit decision anals \si. F~or insiancc. in cascs \klhe' c thcre are no unecriat itties in dcscribing the possible implicatiotns of cacti alitcrnatis and \\ hcrc tic ut utý f unction (i.e..objcctiise functioti INi lincar. linear programming does not \ioiatc thle axiomis of decision ilnall sis.Ill such ca!.cs. it could lie coniside red at tool of decciioti anal~ sis. s itih the higl ads antage of ctfcct isc cies aluating anl infinite niumber of alternatis cs and sclece'ing thc best tinc.

Aunique aSPeCt Of iedecision anaisksis formnulat ion is the theory and proccdures, des elopedto frnilk ntro~c nd roces sh~jc61\ .;~dgllclts ll he es all mu of a tcitt ylenles. Profcs

ait Iid's afue jud~gmetslll are clcairlý an important parltiof thle ma ijOr prolem~s facinig our societ\W\ith- problems i-oncerningv abortion. the desira-ihiti, of' capital punishment. ort the treatimciii ofterrorist,,. profcssional judgments about the likelihoods of ýarious consequenices resutilng from cachalternative and thle VALuc judgments required to evaluate such alternatises must bei malide. lIn deci-sions concerning inflation or the cnergý situation of' the countrm judgtmenits musit somehoss beformulated about the likely effects of %arious policies. Value tradeoffs must bie miade beiss eninflation rates and unemployment or between the enierg\ a\ ailablc for personal use ( i.e. . cotmfortand national dependence oin foreign fuels. Ini man\ cases. ito neglect such features misses, the essenceof the problem altogether.

Experience in using decision analysis indicates that knoss edgeable professiotnals. industr\executives. and government officials are ss illing to address, the difficuli professional judgments andvalue questions necessary to focus mcaningfuflk on the characteristics of complex decision p.hlenis. 1-losever. most analyses of' important decision problems has e left the Incorporation oft judg-mients. and valuesý to informal proced~ures with unidentified assumptions atnd ito the in!uition of thedecision makers. What has been lacking is not iniormation but at f 'imnesork to articulate and inte-

Is

I

grate the values and professional judgments of decision makers and experts with the existing datato examine the overall implications of alternative courses of action, Decision analysis provides thisframework,

Acknowledgement

Da'id E. Bell, James S. Dver. ('rail, W. Kirkwood. D, Warner North. William P, Pi,:rskallk,Stephen M, Pollack. Richard G, Richels, HIoward Raiffa, Robert L Winkler, and 1)ctlof von Win-

r terfeldt made many helpful suggestions on earlier drafts of this article. This work was supported b%p The Office of Naval Research under contract NtW)014-81.C-0536.

APPENDIX. THE AXIOMS OF DECISION ANALYIS

A unique feature of decision analysis is that it i'jas an axiomatic foundation, The axiomsprovide the rationale and theoretical feasibility for the "divide and conquer" approach of decisionanalysis, In Section 3, decision analysis was decomposed into four steps:

1. Structure the decision problem.

2. Assess possible impacts of each alternative:

3. Determine preferences (values) of decision makersý and

4. Evaluate and compare alternatives,

Axioms corresponding to Steps I through 3 state conditions under which it is feasible to obtain thenecessary information for a decision analysis decomposed in this manner. Axioms corre, nding toStep 4 provide the substance for aggregating the information in the preceding steps to evaluate thealternatives,

To facilitate understanding, the axioems of decision analysis are stated here in an informal andintuitive manner, The complete sense of the axioms is preserved although they are not technicall.hprecise, A formal statement of the axioms is found in Pratt, Raiffa, and Schlaifer 119041, In thefollowing. Axioms la and b pertain to Step 1 of the decision analysis methodology, Axiom 2 pertainsto Step 2, and so on,

Axiom la-Generation of Alternatives. At least two alternativc can be specified.

For each of the alternatives, there will be a number of possible consequences which might result ifthat alternative is followed.

Axiom Itb--Identification of Consequences, Possible consequences of each alternative can beidentified,

In identifying consequences, it may be useful to generate an objectives hierarchy indicating thedomain of potential consequences in the problem, Attributes can be specified to provide evaluationscales necessary to indicate the degree to which each objective is achieved,

Axiom 2-Ouantification of Judgment, The relative likelihoods (ie.. probabilities) of eachpossible consequence that could result from each alternative can be specified.

As discussed in Section 3, there are a number of procedures to assist in specifying relative likeli-hoods. Such probabilistic estimates are based on available data. information collected, analyticalor simulation models, and assessmeni f experts' judgments.

19

Axiom 3-Qu-untification of Preference. ThL relative desirability (i.e.. utilits ) for all the pos-sible consequences of any alternative can he specified.

The preferences which should be quantified in a decision problem are those of the decision makers.It is very helpful if one can assess these preferences directly from the decision maker or decisionmakers. However. for many problems other individuals have a rcsponmibilitm for recommendinmalternatives to the decision makers. In such problenis. those individual% na, have a responihilit.for articilating an appropriate preference structure.

, Axiom 4a..-Comparison of Alternatives. If tlo alternatives would each result in the same tONOpossible consequenes, the alternative yielding tlL higher chance of the preferred Lon-sequence is preferred.

Axiom 4.-Transitivity of Preferences. If one alternative is, preferred it) a second alternativeand if the second alternative is preferred to a third alternatike, then the first diiernati\eis preferred to the third alternative,

Axiom 4c-Substitution of Consequences. It an alternative is modified b. replacing one of itsconsequences with u set of consequences and associated probabilities (i.e., a lottcr.) thatis indifferent to the consequence being replaced, then the original and the modifiedalternatives should be indifferent.

Axiom 4a is necessary to indicate how various alter: ,ti\es should be. compared, Axioms 4b and 4care often referred to as consistency axioms, Axiomn 4c alloks one to reduce comple' alternati\csinvolving a variety of possible consequences to simple alternatives r'lcrred to in Axiorm 4a, It isthen easy to compare alternatives. Axiom 4t, is necessary to include tmnparisons of more than twoalternatives.

The main result of these axioms is that the expected utility of an alhcri.-tive is the indicationof its desirability. Alternatives with higher expected utilities should be preferred to those with lo~erexpected utilities. The probabilities and utilities necessary to calculate expected utility emerge asdistinct items, !nformation aboul each must be gathered in conducting a decision analysis. However.the axioms themselves provide little guidance about how to obtain this information, This is dis-cussed in Sections 3 and 4 on the methodology and practice of decision analysis,

Decision analysis does not require either a single decision maker or identifiable decision mak-ers. It requires an orientation toward the decision to he made and individuals able and willing toprovide information essential to that decision. The essential assumptions in this regard are thosegivcn in axiore * co-i.:sponding to Steps 1, 2. and 3. What is assumed :s that the information requiredby those assumptions can be obtained in a useful manner. A decision analysis which structured andanalyzed a decision problem without any interaction with or knowledge of the "decision makers"could provide a tremendous amount of insight-the product of decision analysis-to them if theysaw the analysis. Recognition of this misconception is important because it has often been statedthat an implicit assumption of decision analysis is that the decision maker is a single individual andcan be identified. It is further claimed that this assumption is invalid for essentially all important

j decision problems. Hence. decision analysis is at best an interesting theoretical exercise, with littlepractical value, While it may be easier to structure a decision problem and provide critical infor-mation by interacting with the identified decision makers, many critical problems do not afford thisluxury, It is not essential for constructive decision analysis to occur.

II

T!_______________

~.I 1REFERENCESM. Alpert and H. Raiffa. "A Progress Report on the Training of Probabilit\ Assessors." in Jiudglnent UnderUncertainty: Heuri~stics3 and Bias~es, D. Kahneman, P. Slovic, and A. Tversky. eds,. Cambridge Uni% ersit% Press,New York, 1981.

K. J. Arrow, Social Choice and Individual Values, 2nd ed., Wilcy, New York, 1903.J. R, Beles. P, Fleischauer. R, L. Keene\. ,C. W. Kirkwood and A. Sicherman, "Decision [ramessc\ork torTcchnologý Choice-Volume 1: A C'ase Study of One Utility's (oal-Nuclear Choice." LA*2 153, Research

Proek 1431vnei eor.Eeti oe Research Institute, Palo Alito. C alitornia. 1981.D. E. Bell, "A Decision Aalysis of Objectives for a Forest Pest Problem .'' in Con flicting ()hjecv'' i t e

sion.s. D. E. Bell, R. L. Keenev and H.~I fesWly(ltrcec) e Yoik, 1977a,D. E. Bell, -A Utility Function for Time Streams Having Interperiod D~ependencies." Operatbon.% Re.%arch,215, pp. 448-458. 1977b.D. E. Bell. "Consistent Assessment Procedures Using Conditional Utility Functions. O)peration.% Rc.%varch, 217.pp. 105., -1066. 1979a,D. E. Bell. "Multiattribute Utility Functions: Decomipositions U :.i ng Inrtecrpol at ion." Alanageppie.i Scine 25,pp. 744-753, 1979h.

D. erouli."Exsionofa Ne -Theory on the Measurement of Risk." 'nnerr cdeicN lhtrnIniperwaics Pe':ropoluwion,. 5, pp. 175-192, 1738. Translated b\ L, Sommer, Econ'ome'tica, 22, pp. 23-36. 1054D. Blackwell and M,. A, Girshick. liieor 'v of Game.% and Statistical IVecision.%. Wilc\. Nes York, 1954.

S. E. Bodil. "Analysis of Risks to Life and Limb." Operation.% Rceoerch, 28. pp. 156-175, 198)).

D. Boyvd. "Interactive Preference Ascsesment lor D)ecision %hith Multiatiribui' Outcomes,' Fourth ResearchConference oin Subjectise Probabihlr, Utilt\. and Decision \taking. Rome, Ital\, 1973.D, W. Boyd. C'. E . Clark. Jr-, and D. W, North, "Analvsis oif a Gosermient Initiatis e to Accelerate Conimertcia I Accept anrce of Solar Ph ot ovoIt aic Syst enis .' Joun nal of th e Operartion al Rekecar, it Scieut)i, 32. to ap pear1981.

R. V, Brown, "Do Managers Find Decision Theor\ Useful?'. Harvard Buiott.%% Rei-'w, 4S, pp. 78-S9. 197(1.

R. V, Brown. A. S. Kahr and C. Peterson, Dlecision AnalYsi~s br the Alatiagc'i I lolt. Ness York. 1974.

E. 0. Cazalet. C. E. Clark and T, W. Ktchmn, "Costsý and Benecfitsý of1 (cr Under Capacity in Electric Posse,System Planning," EA-927. Research Project 11(07. Electric Posser Research Institute, P'alo Alto, ('tdifotnw.i1978.

D. M. Crawford. B. C. Huntzinger and C'. W, Kirkwood, "Niultiobjective Decision Analysis lor TransmissionConductor Selection." Mlanagemnent Science, 24. pp. 17001-1709. 1978.

D, Davidson, P. Suppes and S. Siegel. Decision Making: An Experimnt'nal Approach. Stanford Uni-ersinsPress, Stanford, California. 1957.

G. Debreu, ''Topological Methods in Cardinal Utility Theory." in Mahemat/',ical Methrod.% in ftire Social Sci'nce'A,19.59, K. J. Arrow. S. Karlin and P. Suppes. edls., Stanford University Press, Stanford, California. 196)).

B. DeFinciti. "La prevision: ses lois logiqlues. ses sources subjectives," Anntalv.% de llnstitut Henrri Poipruare, 7.pp.1--$, 937 Enlis trnsltio byIL L. Kyburg. Jr.. in 11. E, Kyburg. Jr., arid 11, 1L. Smokler. eds..

Studies in Subjective ProbabilitY, Wiley., New York, 1964.

R. dc Neufville and R. L. Keeney. "Use of Decision Analysis in Airport Development in Mexico Cit\," inAtial * sis of Public S *ystenis, A. W Drake. R. L, Keeneý and P. MI. Morse. eds., M.I.T. Press. Cambridge.Massachusetts. 1972.

J. S. Dyer and R, F. Miles, Jr.. "An Actual Application of Collectise Choice Theory to the Selc~tion oftTrajectories for the Mariner Jupiter.'Saturn 1977 Project." Operations Re~searci, 24, pp. ý202(1-44, 1976.

J. S. Dyer and R, K Sarin. "Measurable Multiattribute Value Functions." Operations Research, 27, pp. 8 I0-822. 1979.W. Edwards. "The Theory of Decision Making." Psychological Bulletin, 51, pp. 3801-417, 1954.

W, Edwards. "Conservatism in Human Information Processing." in Formal Represenitation.% of' Humnan Judg-mnen.,. B. Klein-Muntz. ed.. 'Wiley, New York. 1968.

21

W. Edwards, "Reflections on and Criticisms of a IhghlN Political Multiattribute Utility Analsis." in Math/-matwal Frontiers of Behavioral and Polity Sciences, L.. Cobb and R. Wk. Thrall, eds., Westvie Press. Boulder.Colorado, ;9KO.

H. J. Einhorn and R. M, Hogarth, "Behavioral Decision Theory: Processes of ludgment and Choice." AnnmualReview of P.Avchology, 32, pp. 53--88, 1981.

P. 11. Farquhar, "A Fractional Hypercube Decomposition Theorem for Multiattribute UtilitN Functions,"Overation.% Research, 23, pp. 941-967, 1975,

P. 1, Farquhar and P. C, Fishburn, "Equialences and Continuit. in Multivalcnt Preference Structures,"Operations Re.s'arch, 29, pp. 282-293, 1981.

W. Fellner, Probability and Profit: A Study of Economw Behaavior .4/hig a L'• i.in,.s, Richard 1). Irwin,Homewood, Illinois, i965.

G. W. Fischer, "Mltlidimensional Utility Models for Risky and Riskless, ChoicL ' (rgani:aticnal Bhat tor andHuman Performaace, 17, pp. 127-146, 197.,

G. WN Fischer, 'Utilitv Models for Multiple Objectise Decisions: Do They Accuratel. Represent HumanPreferences," Decision Science, 10, pp. 451-479, 1979,

B, Fischhoff, "Clinical Dr.i,,ion Analysis," Operations Research, 28, pp. 28-43, 1981),

P. C. Fishburn, Decision and Value T/'orv, 'ile.v, New York, 1964,

P. C. Fishburii. "Independence in Utility Theor. with Whole ProduLt Sets.," Operation.s Re.scarcih, 13, pp. 28-45, 19t5.

P. C. Fishburn, "Methods of Estimating Additive Utilities." Management Sciemc, 13, pp. 435-453, 1947.

P. C. Fishburn, Utility Th/ory for Decision Making. Wile%.. Nesk York, 197(0

P. C. Fishburn, The Theory of Social Choice, Princeton University Pres.. Princeton, Ne Jerc\. 19173.

P. C. Fishburn, "Von Neumann-Morgenstern Utilit. Functions on Two Attribute%,," Operation.% Rsc.'arch 22,pp, 35-45, 1974.

M. Friedman and L, J, Savage, "The Expected.Utility Hypothesis and the Measurabii of Ltilit.." Journalof Political Economy, 60, pp. 463-.474. 1952.

P. C. Gardiner and W. Edwards, "Public Values: Multi.attribute Litt. Meawurement tor Social DIcci•'ionMaking." in Human Judgmni'nt and Deci.oi Prone..se. , M. V. Kaplan and S So arts. ed., Academic Press.Ne% York, 1975,

R. J. Gilbert and R. G. Richels, "Reducing Uranium Resource Uncertaint.: Is it Worth the Cost?", Rve•o11rcesand Energy, 3, to appear, 1981.C. J. Gravson, Decision Under Uncertainty: Drilling D'cisi:mns by Oil and (ia. ' Operator.%, Division of Research,Harvard Busines School, Boston. Massachusetts, 1960,

K. R. Hammond, G, H. McClelland and J, Mumpower. Human Judgment and D)ecrsion Making, PraegerPublishers, New York, 198(0

J. C Harsanvi, "Cardinal Welfare. Individualistic Ethics, and Interpersonal Comparisons of Utility." Journalof Political Economy, 63, pp. 31(9-321, 1955,

j R. M, ttogarth, "Cognitive Processes and the Assessment of Subjective Probabilit\ Distributions," Journal ofthe American Statistical Association, 70, pp. 271-289. 1975.

C. S, Holling, ed.. Adaptive Environmental Assessment and Management, Wiley (interscience). Ne York,1978.

C. A. Holloway, Decision Making Under Uncertainty, Prentice-Hall. Englewood Cliffs, New Jersey. 1979.

R. A. Howard, "Decision Analysis: Applied Decision Theory." Proceedings of' the Fourth International Con.ference on Operational Research, Boston, Massachusetts. 1966,R. A. Howard, "Life and Death Decision Analyss.," Research Report No. EES DA-79-2, Department ofEngineering-Economic Systems, Stanford Unis, sity, Stanford. California, 1979.

R. A. Howard. J, E, Matheson and D. W, North. "The Decision to Seed Hurricanes." Science, 176, pp. 1191-1202, 1972,

G. P. 1-uber. "Methods for Quantifying Subjective Probabilities and Muhiautrib,:c Utilities." Decision Science,5, pp. 43(0-458. 1974.

22I

g II B. PR. Judd. "Decision Analysis of Electrical Capacity Expansion." Lawrence Symposium on Svstem.' andDecision Sci'm'eA 1977 Western Periodicals, North Hollywood, California, 1978,

It, Jungermann, " Structural Modeling of Decision Problems," Institute for Psychology. Tec.inical Unixersity"of Berlin, 198,0.

SD. Kahneman and A. Tversky, "Prospect Theory: An Analysis of Decision Under !'Isk," Econometrica, 47,pp, 263-291, 1979.

G. J. Kaufman, Statistical D)ecision ond Related Techniques in Oil and (;a. El•ploration, Prentice-|Hall, Englc-wood Cliffs, New Jersey, 193,

G. J. Kaufman and II. Thomas, eds., Modern De'cision Anal.-sis, Penguin Books, Nee York. 1977,

D. L. Keefer and C. W. Kirk-ood. "A Niultiobjective Decision Analysis- Budget Planning for Product Engi-neering," Joutnal of the Operational Re.warch Society, 29, pp. 435-442, 1978.

T. W. Keelin, "A Parametric representation of Additive Value Functions," Managenient Science, 27, pp. 121)0-.1208, 1981.

R. L. Keenqv. "Examining Corporate Policy Using Multiattribute Utility Analysis." Sloani Managemient Re-view, 17, pp. 63-76, 1975,

R. L. Keeney, "Decision Analhsis," in Handbook of Operations Research, J. J. Moder and S. E. Elmaghray,eds,, Van Nostrand Reinhold, New York. 1978,

R. L, Keeney., "Evaluating Alternatives Involving Potential FatL.lities," Ope'rations Re.warch, 28, pp. 188-205.1980a,

RW L, Keeney. Siting Energy Facilitie.s, Academic Press, New York. 1980b.

R. L, Keeney, "Measurement Scales for Quantifying Attributes," Behavioral Science, 26, pp. 29-36. 1981,

R. L, Keenev and C, W. Kirkwood, "Group Decision Making Using Cardinal Social Wellare F'unctions,"Management Science, 22, pp. 430-437. 1975.

R. L Keenev and K. Nair, "Selecting Nuclear PoAer Plant Sites in the PacifiL Northwest Using DecisionAnalysis." in Conflicting Obj'ctiveýs in Deci.stons. D. E. Bell. R. L, Keeney and H. Raiffa. eds.. Wiley (Inter.science). New York. 1977.

R. L. Keenev and !H. Raiffa, "A Critique of Formal Analysis in Public Decision Making." in Anal.'si.s of P•blic

System.t, A. W Drake, R. L, Keeney and P. M, Mon.... :;ds., M.I.T. Press, Cambridge. Mat.sachusotts, 1972.

R. L. Keeney and H, Raiffa. Decisions with AMtdtipe Objective.%, Wiley, New York. 1976,

R. L. Keeney and A. Sichermin, "An Interactive Conmputer Program for Assessing and Analyzing PreferencesConcerning Multiple Objectives," Behavioral Science, 21, pp. 173-182. 1976,C. W, Kirkwood and S. M. Pollock, "Bounding Probabilities for Scenarios %,ith Multiple Attributes," Depart.

ment of Industrial and Operations Engineering. University of Michigan. Ann Arbor. 1980.

C. W, Kirkwood and R. K, Sarin. "Preference Conditions for Multiattribute Value Functions," Operation.sResearch, 28, pp. 225-232, 198(0.

T. C, KooV mans, "Stationary Ordinal Utility and Impatience." Econotnetrica, 28, pp. 287-309. 1960),

D. H, Krantz, "Conjoint Measurement: The Luce-Tukey Axiomatization and Some F"tensions." Journal of'Mathematical Psycholog., 1, pp. 248-277, 1964.

D. H. Krantz. R. D. Luce, P. Suppes and A. Tversky. Foundations of Measrurement, 1. Academic Press, Ne\York, 1971.

J. P. Krischer, "An Annotated Bibliography of Decision Analytic Applications to i'ecth Care." OperationsResearch, 28, pp. 97-113, 1980.

K. J. Lancaster, "An Axiomatic Theory of Consumer Time Preference." International I "oiopnic Review, 4.pp. 221-231. 1963.

J. W. Lathrop and S. R. Watson. "Decision Analysis for the Evaluation of Risk in Nuc)':ar Waste Manage-ment." Journal of the Operational Research Society. 33, to appear. 1982.

1. H. LaValle. "On Cash Equivalents and Information Evaluation in Decisions Under Uncertainty," Journalof the American Statistical Association, 63, pp. 252-290, 1968.

1. H. LaValle. Fundamentals of Decision Analhsis, Holt. New York, 1978,

23

W. W. L. Lee, Decisions in Mlarine' Afinipt: The Role' of Preferrt',ut.s and Trade'offs. Ballinger Publishers.Cambridge, Massachusetts, 1979.D. Longbottom and G, Wade, "An Investigation into the Application of Decision Anialysis in Umited KingdomCompanies," Otnega, 1, pp. 207-215, 1973,R. D. Luce and J. W. Tukey, "Simultaneous Conjoint Measurement: A Newk Type of Fundamental Measure-ment." Journal of Mathematical Psychology, 1. pp. 1-27. 1904.

J. F. Magee. "Decision Trees for Decision Making," Harvard Busin's3 Review, 42, No. 4. pp. 120.~ 138. l1964a.

J. F. Magee, "How to Use Decision Trees in Capital Investment." Harvard Bn~tom'%. Review. 42, No. 5. pp79-96. 1904b.

G. Majone and E. W, Quade. eds.. Pitfall..ý of AnalYsi.%, Wiley. Chichester. England. 1980.

A. S. Manine and R. G. Richels, "A Decision Analysis of the U.S. Breeder Reactor Program.' Epierg.\, 3. pp747-767. 1978.

J. E. Matheson. "Decision Analysis Practice: Examples and Insights.'" Procee'ding.% of t/he Ftfth huternaniopiulConference on Operational Research, Venice. Italy. 1969.

M. W. Merkhofer, "The Value of Information Given De!cision Flexibility." Alan ogem tnt Science, 23;. pp. 710-727, 1977.

R. F. Mever. "On the Relationship Among the Utiliti, of Assets. the ,tlt of Consumption, and InsesinientStrategy in en Uncertain. but Time Invariant World.' in OR 09. lProrcedig.s o. t/he Ft.tht hnttcrnatiop (Conferemecon Operational Researchi, 1. Lawrence. ed,. Tavistock Publishing. London. 1970,

R. F. Meyer. "State-dependent Time Preference." in Conflicting Ol'jectnie.s in Decimo.'~o D. E, Bell. R, L..Keenev and H., Raiffa. eds., Wiley (lnterscicenc). Nc\% York. 1977.

A. C. Miller. M. W. Merkhofer. R, A. Howard. J. E. Matheson and T. R. Rive. "Deselopmenltiof AutomatedAids for Decision Analysis." SRI Rctort 3309, SRI International. Menlo Park. California, 19%h.

P. G, Moore and H, Thomas, Flit, Anatopmy of Dc'cision.s, Penguin Books. New York. l'0J6.

P. A. Morris. "Combining Expert Judgments: A l3.iesian Approach," Alanagent'n, Sciewnc,2. pp. 67L)-693.1977,

F. Mosteller and P. Nogee. "An Experimental Measurement of Utility." Journal of P'olitical Ernaoifi.%, !V, pIp371-404.,1951.

A. H. Murphy and R. L, Winkler. "RtliaK1.iNy of Subjective Prohabilit\ Forecast,, of Precipitation and Tern-perature." Applied Statisthus. 26. pp. 41-47. 1977

D. W. North, F. Offensend. and C. Smart. "Planning Wildfire Protection for the Santa Monica Miountains."Fire Journal. 69, No, 1, 1975.

D. W, North and M. W. Merkh(,fer. "A Methodology for Anaiv zing EmissionCri (otrol Strategies." Comnputer.%and Operations Research, 3. pp. 185-207. 1976.

J. S, Pliskin. D. S. Shepard and M, C. Weinstein. "Utility Functions for Life Y'ears and Health Sta'us."Operations Research, 28, pp. 206-224, 19801.

R. A. Pollak. "Additive von Neumanni-Morgenstern Utility Functions." Economnetrica, 35. pp. 485-494. 1907.

J. W. Pratt. "Risk Aversion in the Small an J in the Large." Econopnetrica, 32. pp. 12.2- 136. 1964.

J. W. Pratt, H. Raiffa arid R. 0. Schlaifer. "The Foundations of Decision Under Uncertainty: An ElementaryExposition." Journal of the Amnericai; Statistical Assoc-iaion. 59, pp. 353-375. 1964.J, W. Pratt. H. Raiffa and R. 0. Schlaifer, Introduction to Stuastical Decision Theory, McGta"-Ifill. Nc%%

V . H. Raiffa, Decision Analysis. Addison-Weslev, Reading. Massachusetts, 1968.H. Raiffa, "Preference for Multi-attributed Alternatives." RM-5868-DOT/RC. The Rand Corporation. SantioMonica. California, 1969.

H. Raiffa and R. 0. Schiaifer. Applied Statistical Decisiop. Theory, riarvatcd Busint 3s School. Division ofResearch. Boston. Massachusetts, 1961. Reprinted by the N1.11 1'. Press. Cpmbridge. Massachusetts. 1968.

F. P. Ramsey. "Truth anid Piobabilitv." in The Fouadations of Mathiematics and Other Logical EssaYs, bN F. P.Ramsey, Harcourt, Brace and Co., New York. 1931. Reprinted in Studiest in Subjective Probabilits. K E.Kyburg and H. E. Smokier. eds.. Wiley, New York. 1964.

24

W. R. Rousseau and J. E. Matheson, "Computer Programs for Decision Analysis." Stanford Research Insti-

tute. Menlo Park, California, 1967.

R. K. Sarin, "A Sequential Approach to Cress-Impact Analysis," Futures, 10, pp. 53-62, 1978.

R. K. Sarin, "Ranking of Multiattribute Alternatives with an Applicatioo to Coal Power Plant Siting." inMultiple Criteria Decision Making-Theory and Apt;lication, G. Fandel and "I. Gal, eds., Springer-Verlag.Berlin, 1980.

L. J. Savage, The Foundations of Statistics, Wiley, N.-w York, 1954.

R. 0. Schlaifer, Probability and Statistics for b,:iness Decisions, McGraw-Hill, New York, 1959.

R. 0. Schlaifer, Manual of Cases on Decision Under Uncertainty With Analyses and Teaching Notes, McGraw-Hill, New York. 1968,

R. 0, Schlaifer. Anaiys., of Decisions Under Uncertainty, McGraw-Hill, New York. 1969,

R. 0. Schlaifer, Computer Programs for Elementary Decision Analysis, Division of Research. Harvard Busi-ness School, Boston, Massachusetts, 1971.

D. A. Seaver. D, von Winterfeldt and W. Edwards. "Eliciting Subjective Probability Distributions on Contin-uous Variables," Organizational Behavior and Human Performance, 21, pp. 379-39"1. 1978.

F. Seo and M. Sakawa, "An Evaluation method for Environmental-systems Planning: An Alternative UtilityApproach:' Environment and Planning, 11, pp. 149-168, 1979.

F. Seo, M. Skaya. H, Takahashi, K, Nakagami and H. Horivama, "An Interactive Computer Program forMultiattribute Utility Analysis," GE18-1890-0. Tokyo Scientific Center, IBM, Tokyo. Japan. 1978,

P. Slovic and S. Lichtenstein. "Comparison of Bayesian and Regression Approaches to the Study of Informa-tion Processing in Judgment," Organizational Behavior and Human Per frniance, 6, pp. 649-744. 1971.

R, D. Smallwood a,,d P. A. Morris, "A Task Force Decision Analysis," Operations Research, 18, pp, 60-80,1980.

C. S. Spetzler, "The Development of a Corporate Risk Policy for Capital Investment Decision." IEEE SystemnsScience and Cybernetics, SSC-4, pp. 279.-30U, 1968.

C. S. Spetzler .and C-A, S. Stall von Holstein. "Probability Encoding in Decision Analysis, ManagementScience, 22, pp. 340-358, 1975.

C-A. S. Statl von Holstein and J. E. Matheson, "A Manual for Encoding Probability Distributions," SRIReport 7078, SRI International, Menlo Park. California, 1979.

W. G. Stillwell. F. H. Barron and W, Edwards. "Evaluating Credit Applications: A Validation of MultiattributeUtility Techniques Against a Reil World Criterion," SSRI Research Report 80-1. Social Science ResearchInstitute, University of Southern California, Los Angeles. California, 1980.

Synfuels Interagency Task Forcc. Recommendations for a Synthetic Fuels Commercialization Policy, U.S. Gov-ernment Printing Office, Stock No. 041-001-00111-3, 1975.H. Tamura .rd Y. Nakamura, "Decompositions of Mliliattribute Utility Functions Based on a New Conceptof Convex Depeidence," Conference of IEEE Systems, Man and Cybernetics Society, Japan, 1978,

M. Tribus. Rationc! Descriptions. Decisions, and Designs, Pergamon Press. New York. 1969.

A. Tversky and D. Kahneman, "Judgment Under Uncertainty: Heuristics and Biases." Science, 185, pp. 1124-1131. 1974,

A. Tverskv and D. Kahneman. "The Framing of Decisions .- d the Psychology of Choice." Science, 211, pp.453-458, 1981.

1. W. Ulvila and W. D. Snider "Negotiation (f International Oil Tanker Standards: An Application of Mul-tiattribute Value Theory," 0 (ations Research, 28, pp. 81--)6. 1980.

J, von Neumann and 0. Mo. genstern, Theory of Games and Economic Behavior, 2nd ed., Princeton UniversityPress, Princeton. New Jerse,. 1947.

D. von Winterfeldt, "Setting standards for Offshore Oil Discharges' A Regulatory D-cision Analysis:' Oper-ations Research, 30, to appear. 1982.

D. von Winterfeldt, "Structuring Decision Problems for Decision Analysis." Acta Psychologica, 45, pp. 71-93.1980.

A. Wald, Statistical Decision Functions, Wiley. New York. 1950.

25

4|F-

i" | R, L. Winkler, "The Assessment of Prior Distributions in Bayesian Anxl\ysis'" Journal of the American Sta-* tistical Association, 62, pp. 776-8(X), 1967a,

R. L. Winkier, "The Quantification of Judgment: Some Methodolojfical Suggestions." Journal of tlhe American"Statistical Association, 62, pp. 1105-1120, 1967b.

R. L. Winkler, "Scoring Rules and the Evaluation of Probability Assessors," Journal of the American Statistical

Association, 64, pp. 1073-1078, 196Q,

R. L. Winklei, Introduction to Bayesian Inference and Decision, Holt, New York. 1972.

R. L. Wiinl:,er, "Combining Probability Distributions from Dependent Information Sources," ManagementScience, 27, pp. 479-488. 1981,

2j

II4

26