Guidelines for Successful Simulation Studies

8/8/2019 Guidelines for Successful Simulation Studies

1/8

Proceedings of the I990 Winter Simulation ConferenceOsman Balci, Randall P. Sadowski, Richard E. Nance (eds.)

GUIDELINES FOR SUCCESSFUL SIMULATION STUDIES?

Osman BalciDepartment of Computer ScienceVirginia Polytechnic Institute and State UniversityBlacksburg, Virginia 24061-0106

ABSTRACTThe purpose of this paper is to provide guidelines for conduct-ing a successful simulation study. The guidelines are providedthroughout the entire life cycle of a simulation study that is com-posed of 10 processes, 10 phases, and 13 credibility assessmentstages. The practitioners can follow the guidelines presented hereinand significantly increase their chance of being successful in con-ducting a simulation study.

1. INTRODUCTIONIn a simulation study, we work with a model of the problemrather than directly working with the problem itself. If the modeldoes not possess a sufficiently accurate representation, we can easi-ly have junk input and junk output. It is no challenge to write acomputer program which accepts a set of inputs and produces a set

of outputs to do simulation. The challenge is to do it right.Multifaceted and multidisciplinary knowledge and experience arerequired for a successful simulation study. In order to gain the basicknowledge for using simulation correctly, Shannon [I9861 predictsthat a practitioner is required to have about 720 hours of formalclassroom instruction plus another 1440 hours of outside study(more than 1 man-year of effort).Assessing the acceptability and credibility of simulation resultsis no t something that is done after the simulation results areobtained. Assessment of accuracy (i.e., verification, validation, andtesting) must be done right after completing each phase of a simula-tion study.The key to success in a simulation study is to follow a compre-hensive life cycle in an organized and well-managed manner. Theever-increasing complexity of systems being simulated can only bemanaged by following a smctured approach to conducting thesimulation study.The purpose of this paper is to present a comprehensive lifecycle of a simulation study and guide the simulationist in conduct-ing I O processes, 10 phases, and 13 credibility assessment stages ofthe life cycle. The paper begins by defining the life cycle in Section2. This is followed, in Section 3, by guidelines for each of the 10processes and 10 phases of the life cycle. It goes on, in Section 4, todiscusseach of the 13 credibility assessment stages of the life cycleand proposes an overall evaluation scheme for assessing the accept-ability and credibility of simulation results. Concluding remarks aregiven in Section 5 .2. THE LIFE CYCLE O F A SIMULATION STUDY

The life cycle is composed of ten phases as depicted in Figure1 [Balci 1987; Nance 19811. The ten phases are shown by oval sym-bols. The dashed arrows describe the processes which relate thephases to each other. The solid arrows refer to the credibility assess-ment stages (CASs). The life cycle should not be interpreted asstrictly sequential. The sequential representation of the dashedarrows is intended to show the direction of development throughoutthe life cycle. The life cycle is iterative in nature and reverse transi-tions are expected.

7 This paper is a modified version of: Balci, 0. (1989),How toAssess the Acceptability and Credibility of SimulationResults, In Proceedings of the 1989 Winter SimulationConference,pp. 62-7 1.

3. PROCESSES OF THE LIFE CYCLEThe ten processes of the life cycle are shown by the dashedarrows in Figure 1.Although each process is executed in the orderindicated by the dashed mows, an error identified may necessitateretuming to an earlier process and starting all over again. Someguidelines are provided for each of the ten processes below.

3.1 Problem FormulationWhen a problem is recognized, a decision maker (a client orsponsor group) initiates a study by communicating the problem toan analyst (a problem-solver, or a consultant/research group). Thecommunicated problem is rarely clear, specific, or organized.Hence, an essential study to formulate the a c t w l problem usuallyfollows. Problem Formulation (problem structuring or problemdefinition) is the process by which the initially communicated prob-

lem is translated into a formulated problem sufficiently well definedto enable specific research action [Woolley and Pidd 19811.Balci and Nance [1985] present a high-level procedure that: (1)guides the analyst during problem formulation, (2) structures theverification of the formulated problem, and (3) seeks to increase thelikelihood that the study results are utilized by decision makers. Thereader is referred to [Balci and Nance 19851 for details of theprocedure.3.2 Investigation ofSolution Techniques

All alternative techniques that can be used in solving the for-mulated problem should be identified. A technique whose solutionis estimated to be too costly or is judged to be not sufficiently bene-ficial with respect to the study objectives should be disregarded.Among the qualified ones, the technique with the highest expectedbenefitdcost ratio should be selected.The statement when all else fails, use simulation is mislead-ing if not invalid. The question is not to bring a solution to the prob-lem, but to bring a sufficiently credible one which will be acceptedand used by the decision maker(s). A technique other than simula-tion may provide a less costly solution, but it may not be as useful.Sometimes, the communicated problem is formulated under theinfluence of a solution technique in mind. Occasionally, simulationis chosen without considering any other technique just because it isthe only one the anaIyst(s) can handle. Skipping the investigationprocess may result in unnecessarily expensive solutions, sometimesto the wrong problems.As a result of the investigation process, we assume that simula-tion is chosen as the most appropriate solution technique. At thispoint, the simulation project team should be activated and be maderesponsible for the verification of the formulated problem and feasi-bility assessment of simulation before proceeding in the life cycle.3.3 System Investigation

Characteristics of the system that contains the formulated prob-lem should be investigated for consideration in system definitionand modeling. Shannon [19751 indicates six major system charac-teristics: (1) change, (2) environment, (3 ) counterintuitive behavior,(4) drift to low performance, (5) interdependency, and (6) organiza-tion. Each characteristic should be examined with respect to thestudy objectives that are identified with the formulation of the p roblem.In simulation, we mostly deal with stochastic and dynamic real

25


2/8

0.Balci

fIi edefinition Model:ntationCOMMUNICATIVEIMULATION Mddel DataValidation ValidationI,,

EXPERIMENTAL Design of Experiments

Figure 1. The Life Cycle of a Simulation Study

systems that change over a period of time. How often and howmuch the system will change during the course of a simulationstudy should be estimated so that the model representation can beupdated accordingly. Changes in the system may also change thestudy objectives.A system's environment consists of all input variables that cansignificantly affect its state. The input variables are identified byassessing the significance of their influence on the system's statewith regard to the study objectives. For example, for a traffic inter-section system, the interarrival time of vehicles can be identified asan input variable making up the environment, whereas pedesmanarrivals may be omitted due to their negligible effect on the

system's state. Underestimating the influence of an input variablemay result in inaccurate environment definition.Some complex systems may show counterintuitive behaviorwhich we should try to identify for consideration in defining thesystem. However, this is not an easy task, especially for thosesystems containing many subjective elements (e.g., social systems).Cause and effect are often not closely related in time or space.Symptoms may appear long after the primary causes [Shannon19751.To be able to identify counterintuitive behavior, it is essen-tial that the simulation project employs people who have expertknowledge about the system under study.A system may show a drift to low performance due to the

26


3/8


4/8

0.Balcistudy objectives. Verification deals with building the simulationmodel m. Validation deals with building the simulationmodel. Testing is demonsmting that bugs exist or revealing theexistencc of errors (bugs).Since a d e l s an abstraction of the reality, we cannot talkabout its absolute accuracy.Credibili ty, quality, validity, and verityare measures that are assessed with respect to the study objectivesfor which the model is intended. In some cases, a 60% level ofconfidence in the credib ility of simulation results may very wellseme for the purpose; in another. 90%may be required.Thne types of errors may be committed in conducting a simu-lation study. TypeI Error is committed when the study results arerejected when in fact they are sufficient ly credible. Type 11Error iscommittedwhen the study results are accepted when in fact they arenot sufficient ly credible. The definitions of type I and type errorscan be extended to apply for every CAS . In the case of modelvalidation, we called the probability of committing type I error asmodel builders risk and the probability of wmmimng type 1 erroras model users risk [Balci and Sargent 19811. Type 111Error iscommitted when the formulated problem does not completelycontain the actual problem [Balci and Nance 19851. Committingtype IIerror corresponds to solving the wrong problem.A simulation project team should possess at least four areas ofknowledge and experience to be successful: (1) project leadership,(2) modeling. (3 ) programming, and (4) knowledge of the systemunder study [Annino and Russell 19791.Lacking the necessary levelof expertise in any area may result in a failure of the project or an

Subjectivity is and will always be part of the credibility assess-ment for a reasonably complex simulation study. The reason forsubjectivity is two-fold modeling is an art and credibili ty assess-

error of type , nor III.

ment is situation dependent.A subjective. yet quite effective method for evaluating theacceptability of simulation results is peer assessment. the assess-ment of the acceptab ility by a panel of expert peers. This panelshould be composed of: (1) people who have expert knowledge ofthe system under study, (2) expert modelers, (3) expert simulationanalysts, and (4) people with extensive experience with simulationprojects.The panel examines the overall study based upon the projectteams presentation and detailed study of documentation. Workingtogether and sharing their knowledge among each other, panelmembers measure the indicators shown by the leaves of the tree nFigue 2 which assists in explaining the hiemchy of CAS s . Abranch of the tree represents a CAS except the branch of otherindicators of experimentald e l uality.An i d c u r o r is an indirect measure of a concept. It can bedecomposed into other indicators. The ones at the base level (i.e.,the ones that are not decomposed further) ,must be directlymeasurable.The indicators (leaves) are presented in the following subsec-tions. The kth indicator out of N i j ones corresponding to the jth

branch (from the top of Figure 2) at level i i s measured with a score,s$. out of 100 and is weighted with W i j k a fractional valuebetween 0 and 1, according to its importance. The followingconsuaint must be satisfid

Level I Level 2 Level 3

of Simulation Results

Simulation Results

\Experiment DesignExperimental Model Verification

DataValidationModelValidation

PresentationVerification y (other indicators)Figure 2. A Hierarchy of Credibility Assessment Stages for Evaluating the Acceptability of Simulation Results


5/8

Thus, a score for the jth branch at level i, Sw is calculated on ascale from 0 to 100 where 0 represents not credible at all and 100means sufficiently credible.

In addition to weighting the indicators, panel members can alsoweight the branches based on experience and mining . For example,model validation branch should be given higher weight than theother branches at level 3, if it is possible to validate the modelobjectively using the real system data under all experimentalconditions of interest. On the other hand, if the model represents anonexistent system or a future-oriented situation in which the past isnot a good predictor of the future, higher weight should be given toother branches at level 3.Assume that W ;;denotes the weight for the jth branch at level i.IWij is a fractional value between 0 and 1 where 0 represents notcrkca l at all and 1 indicates extremely critical.W i j s re specified with the following constraints:

2i IC W , . = 1 i=1 , 2 a n d C W3, =1j =1 j=12i IC W , . = 1 i=1 , 2 a n d C W3, =1j =1 j=1

Thus, a credibility score for the quality assurance branch iscalculated as

7j =1

s2 4 =cw3js3jSimilarly, S ll is computed and an overall score, S ( = Wl,S l l +

W12S12), s obtained as a value on a scale from 0 to 100.The higher the overall score the more confidence we gain forthe acceptability of simulat ion results. However, even a perfectscore would not guarantee that the results will be accepted and usedby the decision makers; because, acceptability is an attr ibute of thedecision maker not an atmbute of the simulation study. Perfectresults may be rejected due to the lack of credibility of the institu-tion performing the study or due to a political reason. Nevertheless,the objective of the simulation project management should be toincrease the confidence as much as possible. A higher overall scoremay not guarantee the acceptance of results but a lower overallscore can easily result in their rejection or an error of type II.

4.1 Formulated Problem VerificationSubstantiation that the formulated problem contains the actualproblem in its entirety and is sufficiently well structured to permitthe derivation of a suff iciently credible solution is calledformulated problem verification [Balci and Nance 19851. For thissubstantiation, a Questionnaire is developed in [Balci and Nance19851 with 38 indicators. People who are intimately knowledgeableof the problem(s) based on experience and training verify theformulated problem by measuring the indicators. The reader isreferred to [Balci and Nance 198.51 for the details of the ver ifica-tion.

4.2 Feasibility Assessment of SimulationAre the benefits and cost of simulation solution estimatedcorrectly? Do the potential benefits of simulation solution justifythe estimated cost of obtaining it? Is it possible to solve the problemusing simulation within the time limit specified? Can all of theresources required by the simulation project be secured? Can all ofthe specific requirements (e.g., access to pertinent classified infor-

mation) of the simulation project be satisfied? These questions arethe indicators of the feasibility of simulation.

Guidelines for Successful Simulation Studies

29

4.3 System and Objectives Definition VerificationWe should justify that the system characteristics are identifiedand the study objectives are explicitly defined with sufficientaccuracy. An error made here may not be caught until very late inthe life cycle resulting in a high cost of correction or an error oftype IIor 111Since systems and objectives may change over a period oftime, will we have the same system and objectives definition at theconclusion of the simulation study (which may last from 6 monthsto several years)? Is the systems environment (boundary) identifiedcorrectly? What counterintuitive behavior may be caused within thesystem and its environment? Will the system significantly drift tolow performance requiring a periodic update of the system defini-tion? Are the interdependency and organization of the systemcharacterized accurately?

4.4 Model QualificationA model should be conceptualized under the guidance of astructured approach such as the Conical Methodology [Nance 1987,19811. One key idea behind the use of a structured approach is tocontrol the model complexity so hat we can successfully verify andvalidate the model. The use of a structured approach is an importantfactor determining the success of a simulation project, especially forlarge-scale and complex models.During the conceptualization of the model, one makes manyassumptions in abstracting the reality. Each assumption should beexplicitly specified. Model Qualification deals with the justificationthat all assumptions made are appropriate and the conceptual modelprovides an adequate representation of the system with respect tothe study objectives.

4.5 Communicative Model Verification and ValidationIn this stage, we confirm the adequacy of the communicativemodel to provide an acceptable level of agreement for the domainof intended application. Domain of Inreended Application[Schlesinger et al. 19791 is the prescribed conditions for which themodel is intended to match the system under study. Level of Agree-ment [Schlesinger et al. 19791 is the required correspondencebetween the model and the system, consistent with the domain ofintended application and the study objectives.Communicative Model Verification and Validation can beconducted by using informal and static analysis techniques shownin Figure 3[Whitner and Balci 19891.

4.6 Programmed Model Verification and ValidationWhitner and Balci [19891 describe software testing techniquesapplicable for Programmed Model Verification and Validation.They provide a taxonomy of testing techniques in six categories asshown in Figure 3. The reader is referred to [Whitner and Balci19891 for the details of the testing techniques.

4.7 Experiment Design VerificationThe design of experiments can be verified by measuring thefollowing indicators: (1) Are the algorithms used for random variategeneration theoretically accurate?; (2) Are the random variategeneration algorithms translated into executable code accurately?(Error may be induced by computer arithmetic or by truncation dueto machine accuracy, especially with order statistics (e.g., X = -lo&(l-U)) [Schmeiser 19811.); (3) How well is the random number

generator tested? (Using a generator which is not rigorously shownto produce uniformly distributed independent numbers withsufficiently large period may invalidate the whole experimentdesign.); (4) Are appropriate statistical techniques implemented todesign and analyze the simulation experiments? How well are theunderlying assumptions satisfied? (See [Law 19831 for severalreasons why output data analyses have not been conducted in anappropriate manner.); ( 5 ) Is the problem of the initial transient (orthe start-up problem) [Wilson and Pritsker 19781 appropriatelyaddressed?; and (6) For comparison studies, are identical


6/8

0.BalciSIMULATION MODEL TESTING TECHNIQUES

Informal Static Dynamic Symbolic Constraint Formal

Desk Checking Syntax Analysis Topd own Testing SymbolicExecution Asxztion Checking Proof of correctnes sWalkthrough Semantic Analy sis Bottom-upTesting Path Analysis InductiveAssertion LamdaCakUlUSCodeInspection SeUclural Analysis Black-box Testing Cause effect Graphing BoundaryAnalysis RedicateCakulUSReview DataFlow Analysis White-boxTesting Partition Analys isAudit Consistency Checking Stress Testing InferenceDebugging Logical DeductionExecut ion Tracing InductionExecution MonitoringExecution RofilingSymbolic DebuggingRegression Testing

PredicateTransformation

Figure3. A Taxonomy for Simulation Model Testing Techniques

experimental conditions replicated correctly for each of thealternative operating policies compared?4.8 Data Validation

In this stage, we confirm that the data used throughout themodel development phases are accurate, complete, unbiased, andappropriate in their original and transformed forms. The data usedcan be classified as (1) model input data and (2) model parametersdata.Data validation deals with the substantiation that each inputdata model used possesses satisfactory accuracy consistent with thestudy objectives, and that the parameter values are accuratelyidentified and used. Here are some indicators to measure datavalidity: (1) Does each input data model possess a sufficientlyaccurate representation?; (2) Are the parameter values identified,measured, or estimated with sufficient accuracy?; (3) How reliableare the instruments used for data collection and measurement?; (4)Are all data transformations done accurately? (e.g., are all datatransformed correctly into the same time unit of the model?); ( 5 ) Isthe dependence between the input variables, if any, represented bythe input data model(s) with sufficient accuracy? (Blindly modelingbivariate relationships using only correlation to measuredependence is cited as a common error by Schmeiser [1981].); and(6) Are all data up-to-date?4.9 Model Validation

Substantiating that the simulation model, within its domain ofapplicability, behaves with satisfactory accuracy consistent with thestudy objectives is called Simulation Model Validation. TheDomain of Applicability is the set of prescribed conditions forwhich the experimental model has been tested, compared againstthe system to the extent possible, and judged suitable for use[Schlesinger et al. 19791.Model validation is performed by comparing model behaviorwith system behavior when both model and system are driven underidentical input conditions. Only under those input conditions we canclaim model validity, because a model which is sufficiently validunder one set of input conditions can be completely absurd underanother. If a model is used in a what if environment or if it is aforecasting model, the possible input conditions may form a verylarge domain over which model validation may become infeasible.The existing literature on simulation model validation [Balciand Sargent 19841 generally falls into two broad areas: subjectivevalidation techniques and statistical techniques proposed for

validation. Tables 1 and 2 list these techniques.The applicability of the techniques in Tables 1and 2 dependsupon the following cases where the system ki n g modeled is: (1)completely observable - all data required for validation can becollected from system, (2)partially observable- ome required datacan be collected, and (3) nonexistent or completely unobservable.The techniques in Table 1 are described below. The statistical tech-niques in Table 2areapplicable only for case 1.Event Validation employs identifiable events or event pattemsas criteria against which to compare model and system behaviors.Events should be identified at a level of generality appropriate withthe study objectives. Observing the same events in both the modeland system outputs does not justify event validity if their temporalsequence differs. It may be necessary to weight events since aparticular event may be more important for the model to replicatethan another.Face Validation is useful as a preliminary approach tovalidation. The project team members, potential users of the model,people knowledgeable about the system under study, based upontheir estimates and intuition, subjectively compare model andsystem behaviors to judge whether the model and its results arereasonable.Field Tests place the model in an operational situation for thepurpose of collecting as much information as possible for modelvalidation. These tests are especially useful for validating models ofmilitary combat systems. Although it i s usually difficult, expensive,and sometimes impossible to devise meaningful field tests forcomplex systems, their use wherever possible helps both the projectteam and decision makers to develop confidence in the model .Graphical Comparisons is a subjective, inelegant, and heuris-tic, yet quite practical approach especially useful as a preliminaryapproach. The graphs of values of model variables over time arecompared with the graphs of values of system variables to investi -gate similarities in periodicities, skewness, numbex and location ofinflection points, logarithmic rise and linearity, phase shift, trendlines, exponential growth constants, etc.Historical Methods are quite philosophical in nature andexplorethreemajor methodological positions conceming the valida-tion of especially economic models: rationalism, empiricism, andpositive econonucs.Hypothesis Validation deals with comparing hypothesized rela-tionships among system variables with simulated ones. Ifx, s iden-tified to bear a given relationship to Y, in the system, then X,should bear a corresponding relationship toYm n a valid model .Internal Validation is another preliminary approach to valida-


7/8

Guidelines for Successful Simulation Studies

Table 1. Subjective Validation TechniquesEvent ValidationFace ValidationField TestsGraphical ComparisonsHistorical MethodsHypothesis ValidationInternal ValidationMultistage ValidationPredictive ValidationSchellenberger's CriteriaSensitivity AnalysisSubmodel TestingTuring Test

Table 2. Statistical Techniques Proposed for ValidationAnalysis of VarianceConfidence Intervals/RegionsFactor AnalysisHotelling's T2 TestsMultivariate Analysis of Variance

Standard MANOVAPermutation MethodsNonparametric Ranking MethodsNonparametric Goodness-of-fit TestsKolmogorov-Smimov TestCramer-Von Mises TestChi-square TestNonparametric Tests of MeansMann-Whitney-Wilcoxon TestAnalysis of Paired ObservationsRegression AnalysisTheil's Inequality CoefficientTime Series AnalysisSpectral AnalysisCorrelation AnalysisError Analysist-Test

tion. Holding all exogenous inputs constant, several replications ofa stochastic model are made to determine the amount of stochasticvariability in the model. The unexplained variance between thesereplications would provide a measure of intemal validity.Multistage Vulidation combines the three historical methodsinto a three-stage approach. In stage 1, a set of hypotheses or postu-lates are formulated using all available information such as observa-tions, general knowledge, relevant theory, and intuition. In stage 2,it is attempted to justify model's assumptions where possible byempirically testing them. The third stage consists of testing themodel's ability to predict system behavior.Predictive Validation requires past data. The model is drivenby past system input data and its forecasts are compared with thecorresponding past system output data to test the predictive abilityof the model.Schellenberger's Criteria include (1) technical validationwhich involves the identification of all divergences between modelassumptions and perceived reality as well as validity of the dataused, (2) operational validation which addresses the question ofhow important the divergences are, and (3) dynamic validationwhich insures that the model will continue to be valid during itslifetime.Sensitivity Analysis is performed by systematically changingthe values of model input variables and parameters over some rangeof interest and observing the effect upon model behavior. Unexpect-

ed effects may reveal invalidity. The input values can also bechanged to induce errors to determine the sensitivity of modelbehavior to such errors. Sensitivity analysis can identify those inputvariables and parameters to the values of which model behavior isvery sensitive. Then, model validity can be enhanced by assuringthat those values are specified with sufficient accuracy.Submodel Testing refers to both submodel verification and sub-model validation, and requires a top-down model decomposition interms of submodels. The experimental model is instrumented tocollect data on all input and output variables of a submodel. Thesystem is similarly instrumented (if possible) to collect similar data.Then, each submodel behavior is compared with correspondingsubsystem behavior. If a subsystem can be modeled analytically(e.g., as an M/M/1 model), its exact solution can be comparedagainst the simulation solution to assess validity quantitatively.Turing Test is based upon the expert knowledge of peopleabout the system under study. These people are presented with twosets of output data obtained, one from the model and one from thesystem, under the same input conditions. Without identifying whichone is which, the people are asked to differentiate between the two.If they succeed, they are asked how they were able to do it. Theirresponse provides valuable feedback for correcting model represen-tation. If they cannot differentiate, our confidence in model validityis increased.4.10 Quality Assurance of Experimental Model

There are other indicators in addition to the ones presented inSections 4.4- 4.9 or assuring the quality of experimental model.These indicators correspond to the 7th branch at level 3 in Figure 2and are derived from software quality characteristics. Executionefficiency, maintainability, portability, reusability are someimportant simulation model quality characteristics.The dependence among the seven branches at level 3 in Figure2 and the dependence among the corresponding indicators shouldbe observed for assuring the quality of experimental model. Anindicator with a low score may result in an unacceptable score foran indicator later in the life cycle. For example, failure to justify thereasonableness of an assumption in Model Qualification may resultin poor model validity. Due to this dependence, it is important todetect errors as early as possible in the model development phases.4.11 Credibility Assessm ent of Simulation Results

Concluding upon sufficient quality of the experimental modelis a necessary but not a sufficient requirement for the credibility ofsimulation results. The experimental model quality is assured withrespect to the definition of system and study objectives. An errormade in defining the system or a study objective or failing toidentify the real problem may cause unacceptable simulation resultsor an error of type 11.4.12 Presentation Verification

The simulation project management should verify the presenta-tion of simulation results before they are presented to the panel ofexpert peers for acceptability assessment. There are four major indi-cators that need to be measured for the verification: (1 ) interpreta-tion of simulation results, (2) documentation of simulation study,(3) communication of simulation results, and (4 ) presentationtechnique.4.12.1 Interpretation of Simulation Results

Since all simulation models are descriptive, simulation resultsmust be interpreted. In the simulation of an interactive computersystem, for example, the model may produce a value of 20 secondsfor the average response time; but, it does not indicate whether thevalue 20 is a "good" result or a "bad" result. Such a judgment ismade by the simulation analyst depending upon the study objec-tives. Under one set of study objectives the value 20 may be toohigh; under another, it may be reasonable.Naturally, if simulation results are not sufficiently credible theinterpretation will be in error. For example, when comparingdifferent operating policies, if experiments are not conducted under

31


8/8

0.Balciidentical conditions,a policy cannot be claimed as the best. If theanalyst fails to sat is fy this requirement, the simulation results willbe erroneous resulting in inaccurate interpretation. Therefore,interpretation accuracy is directly dependent on the credibility ofsimulation results.The project team should review the way the results areinterpreted in every detail to evaluate interpretation accuracy.Errors may be induced due to the complexity of simulation results,especially for large scale and complex models.4.12.2 Documentationofa S imulation Study

Gass 119831 points out that we do not know of any modelassessment or modeling project review that indicated satisfactionwith the available documentation. This problem should beattributed to the lack of automated support for documentationgeneration integrated with model development continuouslythroughout the en& life cycle [Balci 1986; Balci and Nance 19871.A simulation study should be documented with respect to thephases, processes, and CASs of the life cycle.4 . 1 2 3 Communication of Simulation Results

The simulation project team must devote sufficient effort incommunicating technical simulation results to decision makers n alanguage they will understand. They must pay more attention totranslating from the specialized jargon of the discipline into a formthat is meaningful to the nonsimulationist and nonmodeler.4.12.4 Presentation TechniqueSimulation results may be presented to the decision makers asintegrated within a Decision Support System (DSS). With the helpof a DSS. a decision maker can understand and utilize the resultsmuch better. The integration accuracy of simulation results withinthe DSS must be verified.If results are directly presented to the decision makers, thepresentation technique (e.g.. overheads, slides, films, etc.) must beensured to be e f f d v e enough. The project management must makesure that the team members are trained and possess sufficientpresentation skills.4.13 Acceptability of Simulation Results

Acceptability of simulation study results is an attribute of thedecision maker@)or the sponsor. The management of a simulationproject cannot control this attribute; however, they can significantlyinfluence it by following the guidelines provided herein.It is assumed that the acceptance of the study results impliestheir implementation. If the results are not implemented, they areconsidered to be rejected even if they are accepted by the sponsor.5. CONCLUDING REMARKS

Although the life cycle of a simulation study is characterizedwith 13 CASs. a literature review [Balci and Sargent 19841 revealsthat most work has concentrated onmodel validation and very littlehas been published on the other CASs. Model validity is anecessary but not a sufficient requirement for the credibility ofsimulation results. SuEcient attention must be devoted to everyCA S n order for a simulation study to be successful.A simulation study is multifaceted and multidisciplinary asillustrated by the life cycle presented herein. Sufficient effort mustbe devoted to every process of the life cycle. Inadequate coverageof a process. due to insufficient knowledge or time, may result inunacceptable results or an errorof type II.The list of indicators for the CASs is not intended to beexhaustive. Additional indicators that are specific to the area ofapplication should be employed whenever possible. There is alsothe issue of assessing the validity and reliability of these indicatorswhich is extremely difficult if not impossible for the h a d scopeadopted herek however, for a specific area of application it shouldbe achievable.

REFERENCESA M ~ o , .S. and E.C. Russell (1979), The Ten Most FrequentCauses of Simulation Analysis Failure - And How to AvoidThem!, Simulation 32,6,137-140.Balci, 0. (1986). Requirements for Model Development Environ-ments, Computers& Operations Research I3 ,1 ,53-67.Balci, 0 . 1987). Guidelines for Successful Simulation Studies,Technical Remrt TR-85-2. D e e n t of ComDuter Science,VPIBtsU, Blhksburg. VA.. ABalci, 0. (1988), The Implementation of Four Conceptual Frame-works for Simulation Modeling in High-level Lanrmages, InProceedings of the I988 Win& Simul-don Confe&n&, k4.A.Abrams. P.L. Haigh, and J.C. Comfort,Eds. IEEE, iscataway,Balci,0. and R.E. Nance (1985). Formulated Problem VerifEationas an Explicit Requirement of Model Credibility, Simulation45,2,76-86.Balci,0 .and RE. Nance (1987), Simulation Model DevelopmentEnvironments: A Research Prototype, Journal of OperationalResearch Society 38, 8,753-763.Balci, 0. and R.G. Sargent (1981). A Methodology for Cost-RiskAnalysis in the Statistical Validation of Simulation Models,Communications of the ACM 24,4,190-197.Balci, 0 .and R.G. Sargent (1984), A Bibliography on the credi-bility Assessment and Validation of Simulation and Mathemat-ical Models, Simulener I S , 3.15-27.Banks, J. and J.S. Carson (1984). Discrete-Event System Simula-tion, Prentice-Hall. Englewood Cliffs, NJ.Elmaghraby, S.E. (1968). The Role of Modeling in IE Design,Industrial Engineering 19,6,292-305.Fishman, G.S. (1978), Principles of Discrete Event Simulation,Wiley-Interscience, New York, NY.Gass,S.I. (1983), Decision-Aiding Models: Validation, Assess-ment, and Related Issues for Policy Analysis, OperationsResearch 31.4.603-631.Law, A.M. (1983), Statistical Analysis of Simulation OutputData, Operations Research 31,6,983-1029.Law, A.M. and W.D. Kelton (1982). Simulation Modeling andAnalysis, McGraw-Hill. New York,NY.Martin, J. and C. McClure (1985), Diagramming Techniques forAnalysts and Programmers, Prentice-Hall, Englewood Cliffs,NJ.Nance, R.E. (1981). Model Representation in Discrete EventSimulation: The Conical Methodology, Technical ReportCS81003-R. Deuartment of Conmuter Science. VPI&SU,

NJ, 287-295.

Blacksburg,VA.Nance, R.E. (1987), The Conical Methodology: A Framework forSimulation Model DeveloDment. In Merhodolonv and Valida-,-tion,0 .Balci,Ed. SCS, SihDiego, CA, 38-43.Overstreet. C.M. and R.E. Nance (1985), A SpecificationLanguage to Assist in Analysis of Discrete Event SimulationModels, Communications of the ACM 28,2, 190-201.Schlesinger, S., et al. (1979). Terminology for Model Credibility,Simulation 32 ,3 ,103 -104 .Schmeiser, B. (1981), Random Variate Generation, In Proceed-ings of the I981 Winter Simulation Conference, T.I. &en,C.M. Delfosse, and C.M. Shub, Eds. IEEE, Piscataway, NJ,Shannon, R.E. (1975), Systems Simulation: The Art and Science,Rentice-Hall, Englewood Cliffs, NJ.Shannon, R.E. (1986), Intelligent Simulation Environments, InIntelligent Simulation Environments, P.A. Luker and H.H.Adelsberger,Eds. SCS, San Diego, CA, 150-156.Whimer, R.B. and 0. Balci (1989). Guidelines for Selecting andUsing Simulation Model Verification Techniques, In Proceed-ings of the 1989 Winter Simulation Conference,E.A. MacNair,K.J. Musselman, and P. Heidelberger, Eds. IEEE, iscataway,Wilson, J.R. and A.A.B. Ritske r (1978), A Sw e y of Research onthe Simulation Startup Problem, Simulation 31 ,2 ,55 -58 .Woolley, R.N. and M. Pidd (1981), Problem Structuring -A Liter-ature Review, Journal of Operational Research Society 3 2 , 3 ,

227-242.

NJ. 559-568.

197-206.

32

Guidelines for Successful Simulation Studies

Documents

Transcript of Guidelines for Successful Simulation Studies