08F-SIW-007

Providing Effective and Efficient Training:A Model for Comparing Simulator Improvements

Justin H. ProstLumir Research Institute

1304 E. Redmon Dr., Tempe, AZ 85283480-773-5241

Justin.Prost@lumirresearch.com

Brian T. SchreiberLumir Research Institute

195 Bluff Avenue, Grayslake, IL 60030847-946-2171

Brian.Schreiber@lumirresearch.com

Winston Bennett, Jr.Air Force Research Laboratory

6030 South Kent Street, Mesa, AZ 85212480-988-9773

Winston.Bennett@mesa.afmc.afmil

Kenneth KleinleinAir Force Research Laboratory

6030 South Kent Street, Mesa, AZ 85212480-988-9773

Kenneth.Kleinlein@mesa.afmc.afmil

Keywords:Training, Fidelity, System Improvements, Decision-Making

ABSTRACT: Training programs for warjighters have become increasingly dependent on the use of simulationsystems, made necessary by the growing costs of live flight training. The transition to a greater dependence onsimulators has required an increase in concerns about the ability ofsimulators to provide effective training. Researchin the area ofsimulatorfuJelity has focused primarily on training capabilities, evaluating current systems for strengthsand weaknesses. The evaluation of current systems is an essential component in providing the best possible trainingenvironment to pilots; however, the impact ofimprovements is ofequal importance. An informed decision about whichchanges will have the greatest benefit for warjighter readiness would provide a means ofmaximizing the return on theinvestment in the development of the system and creating the most efficient and effective training environment possible.The current work presents a process by which system deficiencies and potential solutions are identified and thenintegrated into quantitative models based on multiple factors. The factors included in the model are potential forimprovement, training gaps, extent to which solutions address deficiencies, material costs ofsolutions, and time costs ofsolutions. A project manager and engineers providedfeedback in three areas: the extent to which a solution addressesa deficiency, the cost ofthe technology, and the engineering hours involved in the change. Three models were computedand compared across solutions, revealing the utility of the empirical process. The models included in the comparisonvaried the weights within and across factors, establishing the importance ofdetermining appropriate weightings. Theprocess and algorithm presented in the current work have far reaching practical applications as a tool for assisting inthe decision-making process during development and improvement of training technologies. Ultimately, a cost-benefitdecision-making process will provide an effective tool for maximiZing training benefits during the improvement ofsimulation systems.

1. IntroductionThe use of simulators to supplement live-flight traininghas been established as an effective and cost efficientmethod for increasing warfighter readiness [1,2,3]. Aging

11

aircraft and budget constraints have increased thedependence on simulation systems as a training method.The growing demand for simulation systems to providesuitable training across the diversity of skills required forcombat readiness has encouraged technological advances

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.

1. REPORT DATE SEP 2008

2. REPORT TYPE Conference Proceedings

3. DATES COVERED 01-01-2007 to 31-08-2008

4. TITLE AND SUBTITLE Providing Effective and Efficient Training: A Model forComparing Simulator Improvements

5a. CONTRACT NUMBER FA8650-05-D-6502

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER 62205F

6. AUTHOR(S) Justin Prost; Brian Schreiber; Winston Bennett Jr; Kenneth Kleinlein

5d. PROJECT NUMBER 1123

5e. TASK NUMBER AS

5f. WORK UNIT NUMBER 09

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Lumir Research Institute,1304 E. Redmon Drive,Tempe,AZ,85283

8. PERFORMING ORGANIZATION REPORT NUMBER ; AFRL-RH-AZ-PR-2008-0007

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) Air Force Research Laboratory/RHA, Warfighter ReadinessResearch Division, 6030 South Kent Street, Mesa, AZ, 85212-6061

10. SPONSOR/MONITOR’S ACRONYM(S) AFRL; AFRL/RHA

11. SPONSOR/MONITOR’S REPORT NUMBER(S) AFRL-RH-AZ-PR-2008-0007

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited

13. SUPPLEMENTARY NOTES In Proceedings of the 2008 Fall Simulation Interoperability Workshop, September 2008, Orlando FL

14. ABSTRACT Training programs for warfighters have become increasingly dependent on the use of simulation systems,made necessary by the growing costs of live flight training. The transition to a greater dependence onsimulators has required an increase in concerns about the ability of simulators to provide effectivetraining. Research in the area of simulator fidelity has focused primarily on training capabilities,evaluating current systems for strengths and weaknesses. The evaluation of current systems is an essentialcomponent in providing the best possible training environment to pilots; however, the impact ofimprovements is of equal importance. An informed decision about which changes will have the greatestbenefit for warfighter readiness would provide a means of maximizing the return on the investment in thedevelopment of the system and creating the most efficient and effective training environment possible. Thecurrent work presents a process by which system deficiencies and potential solutions are identified andthen integrated into quantitative models based on multiple factors. The factors included in the model arepotential for improvement, training gaps, extent to which solutions address deficiencies, material costs ofsolutions, and time costs of solutions. A project manager and engineers provided feedback in three areas:the extent to which a solution addresses a deficiency, the cost of the technology, and the engineering hoursinvolved in the change. Three models were computed and compared across solutions, revealing the utilityof the empirical process. The models included in the comparison varied the weights within and acrossfactors, establishing the importance of determining appropriate weightings. The process and algorithmpresented in the current work have far reaching practical applications as a tool for assisting in thedecision-making process during development and improvement of training technologies. Ultimately, acost-benefit decision-making process will provide an effective tool for maximizing training benefits duringthe improvement of simulation systems.

15. SUBJECT TERMS Training programs; Training; Simulator improvements; Simulation systems; Flight training; Trainingeffectiveness; Simulator fidelity; Warfighter readiness; Decision making

16. SECURITY CLASSIFICATION OF: 17. LIMITATIONOF ABSTRACT

Public Release

18.NUMBEROF PAGES

7

19a. NAME OF RESPONSIBLE PERSON

a. REPORT unclassified

b. ABSTRACT unclassified

c. THIS PAGE unclassified

Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

in simulators. The advances include improvements to thetraining experience within a given simulation system andadvances in networking systems together to increase thescope of the training environment [4]. Rapid changes inavailable technology permit increasingly sophisticatedsystem development and improvement. Althoughsimulation training is designed t<? address very specifictraining demands for pilot readiness and the need foreffective training is great, the method of improvement ofour simulation systems often lacks evidence-baseddecision-making [5].

In previous research, the authors presented a method ofincorporating research into models to inform decision­making [5]. The goal of the method is to provide anempirical process to inform decision-making regardingimprovements to simulation systems. The process can bestructured to maximize training benefits, whilemaintaining an accounting of system and budgetconstraints. The method leveraged work from acompetency-based approach to training, in which theskills, knowledge, and experiences required for pilots tobecome combat mission ready are identified [6J. Theexperiences from the Mission Essential Competencies(MEC) approach were used to evaluate the currenttraining capabilities and deficiencies of a DeployableTactical Trainer (DIT) under development at the AirForce Research Laboratory (AFRL). Study fmdingsrevealed that the incorporation of multiple factors intodecision-making models affected the relative importanceof identified deficiencies.

Prost, Schreiber, and Bennett [5J compared three modelsto investigate the utility of quantifying research data forthe decision-making process. The models investigatedincluded a frequency model, an improvement model, andan improvement model that accounted for training gaps.The frequency model computed the number of times thata deficiency was identified as the primary detriment totraining across all MEC experiences. The model allowedfor a direct comparison of deficiencies by comparing theproportion of endorsements for each deficiency relative tothe total number of responses. The improvement modelweighted the frequencies by the degree to which eachMEC experience could be improved for the simulationsystem. Finally, the improvement model that accountedfor training gaps weighted the frequencies by theimprovement score and an additional weighting based onwhether or not the experience represented a training gap.Results of the study showed that a comparison acrossmodels revealed differential importance of thedeficiencies, depending on whether or not factors wereincluded in the models. The identification of systemdeficiencies and the quantification of the impact of thedeficiencies on the training capabilities of the systemprovided an important step in building a data-drivenprocess to assist decision-making. The fmal step in the

12

08F-SIW-007

development of the process is to extend the work from theidentification and modeling of the deficiencies to addresspotential solutions.

2. Current Work

The current study presents a process for identifying,evaluating, and comparing deficiencies of a simulationsystem, and consequently the solutions for addressingthose deficiencies. The purpose of the process is toprovide project managers, researchers, and engineers witha tool to quantify the factors that influence theimprovement process. Ideally, simulator systemimprovements should maximize increases in trainingcapabilities while considering factors such as currenttraining gaps, the degree to which a solution addresses adeficiency in the system, cost of improvements, andintegration of the solutions into the current system.

The work presented below specifically focuses onintegrating 1) the extent to which a solution addresses adeficiency and 2) the costs involved in the solution intoprevious models that evaluated room for improvementand training gaps. Costs taken into consideration includematerial costs and time costs. To examine the effects ofthe two factors of interest in this study on the empiricallydriven decision-making process, data from previousstudies on room for improvement and training gaps willbe included in the current study for the air-to-air skillarea. The air-to-air skill area will be examined in thecurrent study for presentation purposes only, although theprocess would extend to the entire set of skills in apractical application.

The evidence-based decision-making process is examinedby comparing multiple models to determine the effect ofadding factors for consideration. The models can bedivided into two classes: Deficiency Models and SolutionModels. Deficiency Models come from previous workcomparing factors important in determining the impact ofidentified deficiencies [5]. Solution Models weredeveloped for the current study as a means of producingquantifiable scores for each solution. The scores for eachmodel represent the magnitude of training improvementthat would be realized by the integration of the solutionfor one model, and then the improvement relative to thecosts for the other two models. The purpose of thepresentation of the Solution Models in the current study isto illustrate the ability to quantify the effect ofimprovements, relative to costs, in such a way that thesolutions can be directly compared. The capacity of theproposed process to quantify the factors involved in thedecision-making process can remove some of theguesswork from the development and improvement ofsimulation systems.

3. Method

The current study integrates engineer survey informationabout technological changes to a deployable trainer into amodeling framework for evaluating the effect ofdeficiencies on a system. Due to the integration of newfactors into an existing model, some data presented belowis from previously published work [5].

3.1 Evaluators

Ratings from previous research by seven F-16 subjectmatter experts (SMEs) were used in the current study. TheSMEs completed two surveys. One was an evaluation ofthe fidelity of the DTT, and the other was an evaluation ofthe deficiencies of the DIT. In addition to the data fromprevious work, three evaluators participated in the currentstudy by providing responses to a survey designed toassess aspects of integrating new technology to resolvedeficiencies of the DIT. All evaluators were providedwith detailed documentation of system components,capabilities, and limitations, and were provided multipleopportunities to fly various missions in the system beforeproviding system ratings.

The seven evaluators from the previous research were allmale, all recently retired (3 years on average) F-16 pilotsfrom the United States Air Force (USAF), and had anaverage number of operational F-16 flight hours of 2, 119(average total flight hours of 3,495) [5].

The project manager overseeing the development of theDTT completed the survey for the current study. Thesolutions provided were identified and discussed with twoadditional engineers working on the development of theDIT. All participating individuals have been involved inthe development of the DIT since the first workingsystem.

3.2 Surveys

Data were used from a total of three surveys. Two of thesurveys, the Fidelity and Deficiency surveys, wereadministered during previous research. The finalSolutions survey was developed specifically to expand theprevious model. The two surveys administered in thecontext of previous research are briefly described below(and can be seen in more detail in Reference 5). Thesurvey used in the current research is described in greaterdetail below.

Responses to two surveys (previously described asFidelity and Deficiency surveys) were used. The surveyswere constructed by leveraging MEC research for the F­16. The original surveys used 197 items: 70 air-to-ground(AlG) MEC experiences, 55 suppression of enemy airdefense (SEAD) MEC experiences, 45 air-to-air (AlA)MEC experiences, and 27 bold face EPs for the F-16.The current work focuses specifically on the AlA skill

08F-SIW-007

area of the system; therefore, the fidelity survey data fromthe 45 items in that skill area were included in the currentdata set.

The Fidelity Survey consisted of SMEs providing each ofthe items on the survey with a rating from 0 through 5, onthe following scale:

0= N/A (Capability to experience does not exist)1= Capability to experience exists, but is very poor2= Capability to experience exists, but is poor3= Capability to experience exists, but is marginal4= Capability to experience exists, and is good5= Capability to experience exists, and is very good

The Deficiency Survey asked SMEs to identify thedeficiency with the greatest influence on each of the 197survey items. Respondents could choose from a list of 17deficiencies that were previously identified. For thepurposes of this study, because the focus was on the air­to-air skill area, the top five deficiencies for AIA wereincluded. The most influential deficiencies for the AlAskill area were:

I) Limited out-the-window (OTW) capability2) No helmet mounted sight (HMS) capability3) No EP instructor/operator station4) No electronic countermeasure (ECM)

capability5) Software issue, no weapon system capability

The Solutions survey was designed to investigate thepotential solutions for each of the deficiencies. Surveyssolicited identification of the three most promisingsolutions available for each of the top five deficiencies inthe AlA skill area. Respondents then evaluated a series ofqualities for each of the solutions identified. Thefollowing were the questions presented for each solution:

I) To what extent does the proposed solutionremedy the deficiency?

2) What is the projected cost in dollars for thesolution?

3) What is the projected cost in time for thesolution?

The Solutions survey attempted to identify the utility andcost of each particular solution. To understand the cost ofeach solution, it was important to further differentiatecosts along the dimensions of material and time.

3.3 Apparatus: Current DTT system (taken from [5])

The DIT system (see Figure I) that was evaluatedconsisted of the following major hardware/softwarefunctionalities:

Hardware: F-16 cockpit shell with three out-the-window30-inch displays, the actual F-16 stick/throttle, andsimulation of all cockpit displays and switch functions ona high resolution 23" interactive touch screen display.

13

Image generator was an SDS International AAcuity® PC­IG system. Brief/debrief included SmartBoard and two50-inch displays for Head-Up Display, Radar WarningReceiver, and Multi-Function Display.

Figure 1. EIBmple ofDTI simulation system

Software: The system uses classified Block 30 (SCU 5p)actual operational F-16 software. One database iscurrently installed and available. Debrief software has theability to link and time-synchronize video recordingsfrom multiple players. It also has the ability to networkthrough Distributed Interactive Simulation (DIS) or HighLevel Architecture (HLA) standards. It has chaff/flarecapability, but no ECM. Some classified weapons systemsare available, such as Joint Direct Attack Munition(JDAM).

3.4 Procedure

The Solutions Survey was given to the project managerfor the OTT project. The project manager was asked tocollaborate with appropriate engineers to complete thesurvey. Survey responses were entered into a spreadsheetfor analysis. Previous data was combined with theSolutions Survey data to compute multiple models forcomparison. Solution Models were computed using datafrom the Improvement and Training Gap DeficiencyModel and responses from the Solutions Survey.Computation of models was based on including oneadditional factor for each model.

There were a total of six factors included in the modelspresented. The first three factors (Deficiency Factors)were from previous research, but necessary to computethe current models (taken from Reference 5). The factorswere dermed as follows:

Deficiency Factors:

• Deficiency Evaluation was based on thefrequency of response for each deficiency by theevaluators for each skill.

• Fidelity Evaluation was based on the proportionof improvement possible for each skill,

14

08F-SIW-007

calculated as five minus the average fidelityrating for the skill divided by five.

• Training Gap Evaluation was based on MECresearch [6] establishing the training gaps for theskills. The weightings were 1 for no gap, 2 forpotential gap, and 3 for gap.

Solution Factors:

• Extent of Solution was based on a surveyresponse from 0 to 5 (Not at All to Completely).

• Dollar Cost was based on a survey response fromoto 5 (Prohibitive to Minimal).

• Time Cost was based on a survey response fromoto 5 (Prohibitive to Minimal).

Using the data available from previous survey collectionand the data collected in this study, a series of models wascomputed and then compared. A total of six models werecomputed. Of the six models, three were DeficiencyModels and three were Solutions Models.

Model 1: Frequency Model. In Modell, the proportionspresented represent the total frequency of responses for adeficiency across all items, divided by the total number ofresponses (description from [5]).

Model 2: Improvement Model. In Model 2, theproportions presented are the sum of the weighted scoresfor each deficiency divided by the total score across alldeficiencies and items. The weighted scores in Model 2are computed by multiplying the frequency of adeficiency response by the proportion for improvementpotential (description from [5]).

Model 3: Improvement and Training Gap Model. InModel 3, the proportions presented are the sum of theweighted scores for each deficiency divided by the totalscore across all deficiencies and items. These weightedscores were computed by first multiplying the frequencyof a deficiency response by the proportion forimprovement potential, and then multiplying that productby a weight identifying the level of training gap presentfor each skill (description from [5]).

Model 4: Extent of Deficiency Resolution Model. InModel 4, solutions are presented for each of thedeficiencies. A score is given to each solution based onthe Model 3 score representing the impact of thedeficiency on the system Each solution score for Model 4uses the corresponding Model 3 deficiency score,weighted by the extent to which the solution resolves thedeficiency. The extent of the resolution is calculated asthe proportion of the maximum resolution, based on theratings provided (i.e., the rating divided by 5).

Model 5: Material Cost of Solution Model. In Model 5,the score for each solution is calculated as the Model 4score weighted by the dollar cost rating for the solution.

08F-SIW-007

• Propose and make software changes andprogram enhancements to cockpit code tosupport and implement physics based ECM suchas an external ECM pod, decoys, EMI on FCR,etc.;

• Rather than using physics based modeling,implement simulation and emulation.

Missing weapons in the database:• Integrate available code to support desired

weapons;• Develop and write code to support a specific

weapon capability.

The next step in the analyses of the Solutions Models wasto take survey ratings and transform them forincorporation into models. Table 2 presents the ratingsprovided for each solution to Solutions Survey questions.The ratings represent the value reached by consensus forthe project manager and engineers participating in theevaluation.

Table 2. Ratings for Solutions Survey questions acrossproposed solutions.

4. Results

The weighting for dollar cost was calculated as aproportion of the minimal cost, based on the rating fordollar cost (i.e., rating divided by 5). The resulting scorerepresents the amount of improvement relative tomonetary costs.

Model 6: Time Cost of Solution Model. In Model 6,solution scores were calculated using Model 5 scoreweighted by the time cost rating for the solution. Theweighting for time cost was calculated as a proportion ofthe minimal cost, based on the rating for time cost (i.e.,rating divided by 5). The resulting score represents theamount of improvement relative to monetary and timecosts, allowing for comparison across solutions.

Analyses comparing the relative importance of thedeficiencies for each of the Deficiency Models wereconducted previously [5]. Table Ipresents the proportionof the total deficiency score for each model that could beattributed to the specific deficiencies.

Table 1. Proportion of total score attributable to eachdeficiency for air-to-air experiences for each of the threeDeficiency Models Survey Ratings

Extent Dollar TimeModell Model 2 Model 3Limited OTW 0.54 0.47 0.47 Limited

Mini-DART 3 1 4

NoHMS 0.10 0.09 0.09 OTW DART 5 0 3

NoEP 0.09 0.13 0.09 Partial Dome 2 2 1

NoECM 0.06 0.08 0.07 Available with

NoWeaps 0.05 0.08 0.15 mini-DART 3 1 5Available with

The first part of the Solution Survey task on the air-to-airNoHMS

DART 5 1 5skill area of the F-16 simulation system was to identify Partner withpotential solutions for each of the deficiencies. The AFandproject manager and engineers identified up to three commercialsolutions for each deficiency. The following solutions cards 5 3 3were those presented and then evaluated by participants. Use existing

Limited out-the-window field of vision: NoEP IDS 5 5 4

• Procure and integrate a mini-DART; Develop IDS 5 4 1

• Procure and integrate a DART system; Implement

• Partial Dome. physics based

NoECM software 4 4 0No Helmet Mounted Sight Capabilities: Implement

• Procure and implement a currently available simulation andsolution with mini-DART; emulation 2 3 1

• Procure and implement a currently availableNo

Integratesolution with DART;

Weapsavailable code 5 5 4

• Partner with AF developed system utilizing Develop code 5 5 1actual operatIOnal HMCS helmets andcommercial video cards.

No emergency procedure instructor/operator station:• Integrate existing MTT IDS;• DeveloplWrite IDS.

No electronic countermeasure capabilities:

The Solutions Survey scores were transformed andincorporated into the corresponding models. Threemodels were computed using the Deficiency Model 3scores and the weights from the Solutions Survey ratings.Table 3 presents the model scores for each solution acrossthe three models computed. The model scores representthe potential improvement to the system for a solution.

15

The large change in model scores, from model 4 tomodels 5 and 6, is the introduction of costs. Thedeficiency of limited out-the-window visuals, althoughhaving the greatest impact on the system, is also the mostcost intensive deficiency to address.

Table 3. Model scores for each solution across the threemodels evaluated.

Model ScoresModel Model Model

4 5 6Mini-DART 0.28 0.06 0.04

LimitedDART 0.47 0.00 0.00

OTWPartial Dome 0.19 0.07 0.01Available withmini-DART 0.05 0.01 0.01Available with

NoHMSDART 0.09 0.02 0.02Partner withAFandcommercialcards 0.09 0.05 0.03Use existing

NoEP lOS 0.09 0.09 0.07Develop lOS 0.09 0.07 0.01Implementphysics based

NoECMsoftware 0.05 0.04 0.00Implementsimulation andemulation 0.03 0.02 0.00

NoIntegrateavailable code 0.15 0.15 0.12

WeapsDevelop code 0.15 0.15 0.03

5. Summary

The results of the current study build on previous researchto provide a more comprehensive model for evaluatingthe influence of simulation improvements. Previousfindings showed the importance of considering how muchimprovement could be gained for target trainingexperiences and whether or not training experiences werein areas with current training gaps. The results from theprevious study found that the relative importance of thedeficiencies of a system varied when considering thosetwo factors. The current study incorporates additionalimportant factors to the previous model: the extent towhich a solution addresses a deficiency and the cost of thesolution [5].

A comparison of the models revealed that a data drivendecision-making process is an effective means ofdifferentiating among solutions. Examining changingscores for solutions across models, there is a dramaticshift in the training benefits to the system for solutions.For example, if materials and time costs were not an

08F-SIW-007

issue, any solution for the limited OTW field of visiondeficiency would address more of the overall trainingdeficiencies of the system for air-to-air experiences thanwould solutions for other deficiencies. Once materials andtime costs are taken into consideration, solutions to otherdeficiencies are revealed as having a greater effect on thetraining environment, suggesting that those solutions aremore economical. The example presented in this studyillustrates the capacity of this process to provide datadriven models for comparison; however, an importantpoint of this example is to remember that the weightingsused in the current study between and within factors werearbitrarily set to compare models. For example, theweighting given to materials costs was the same as theweighting

To increase the accuracy of the models produced throughthis process, two changes would be implemented whenthe process is used in a real world environment. First,rather than using ratings to address questions of cost, truevalue estimates would be used. The dollar and hour valueswould transform those factors into continuous variables,rather than discrete ratings. Continuous values for thosefactors would provide for dramatically greater accuracy incomparison across solutions. The second change would beto transform cost estimates relative to the overall budget.Because budget constraints represent the framework fordevelopment, the proportion of the total budget that asolution requires provides a more useful scale forcomparison of the effect of solutions. These changeswould increase the internal accuracy of the model;however, equally important is the accuracy of the modelin regard to including all relevant factors.

The evidence-based decision-making process presented inthe current manuscript incorporates multiple sources ofdata into a single data driven model. The fmal modelsexamined included information about the current fidelityof a simulation system, the existing training gaps of asystem, the extent to which a solution resolvesdeficiencies, and costs of a solution. For presentationpurposes, two costs were presented: material and timecosts. A comprehensive model would also includeadditional solution costs, such as maintenance costs,engineering risk costs, compatibility with currentsimulator technologies, and pilot training costs. Theseadditional costs, which may also influence theeffectiveness and efficiency of training, can beincorporated into models just like the two types of costsincluded in the current study. Ultimately, multiple datadriven models could be compared to determine the bestsimulator improvements, depending on a wide range ofcriteria.

The work presented in the current manuscript exploresone step in a process for evaluating and improvingsimulation systems. The growing dependence on

16

simulation systems has introduced many questions aboutthe capabilities of simulation training versus live flighttraining, as well as situations in which simulation trainingis effective [8, 9, 10]. The ability to evaluate and improvesimulation systems through a precise empirical processwill begin to answer questions about how to appropriatelysupplement training via simulators. Additionally,simulation system improvements can be undertaken moreprecisely in order to meet the rapidly changing demandson the systems.

References

[I] Bell, H. H. & Waag, W. L. Evaluating theeffectiveness of flight simulators for training combatskills: A review. The International Journal ofAviation Psychology. 8 (3), pp. 223-242, 1998.

[2] Gehr, S. E., Schreiber, B. T., & Bennett, W. Jr.Within-Simulator Training Effectiveness Evaluation.InterservicelIndustry Training, Simulation andEducation Conference (I/ITSEC) Proceedings.Orlando, FL: National Security IndustrialAssociation, 2004.

[3] Schreiber, B. T. & Bennett, W. Jr. DistributedMission Operations within simulator trainingeffectiveness baseline study: Summary report.(AFRL-HE-AZ-TR2006-0015-Vol I). Air ForceResearch Laboratory, Warfighter Readiness ResearchDivision, Mesa AZ, 2006.

[4] Schreiber, B. T., Rowe, L. J., & Bennett, W. Jr.Distributed Mission Operations within-simulatortraining effectiveness baseline study: Participantutility and effectiveness opinions and ratings.(AFRL-HE-AZ-TR-2oo6-Q015-Vol IV). Mesa AZ:Air Force Research Laboratory, Human EffectivenessDirectorate, Warfighter Readiness Research Division,Mesa AZ, 2006.

[5] Prost, J.H., Schreiber, B. T., & Bennett, W. Jr.Identification and Evaluation of Simulator SystemDeficiencies. InterservicelIndustry Training,Simulation, and Education Conference (IlITSEC)Proceedings. Orlando, FL: National SecurityIndustrial Association, 2007.

[6] Colegrove, C. M. & A11iger, G. M. Mission EssentialCompetencies: Defining Combat Mission Readinessin a Novel Way. Paper presented at the NATO RTOStudies, Analysis and Simulation (SAS) PanelSymposium. Brussels, Belgium, April 2002.

[7] Allen, J., Buffardi, L., & Hays, R. The Relationshipof Simulator Fidelity to Task and PerformanceVariables (ADA23894 I). Fairfax, VA: GeorgeMason University. 1991.

08F-SIW-007

[8] Dennis, K. A. & Harris, D. Computer-basedsimulation as an adjunct to ab initio flight training.International Journal of Aviation Psychology, 8,261­276, 1998.

[9] Schank, J. F., Thie, H. J, Graf, C. M., Beel, J., &Sollinger, 1. Finding the right balance: Simulator andLive Training for Navy Units. Santa Monica: RANDCorporation. 2002.

[10] Schreiber, B. T., Gehr, S. E., & Bennett, W. Jr.Fidelity Trade-Offs for Deployable Training andRehearsal. InterservicelIndustry Training,Simulation, and Education Conference (I/ITSEC)Proceedings. Orlando, FL: National SecurityIndustrial Association. 2006.

Author Biographies

JUSTIN H. PROST is a Senior Research Scientist withLumir Research Institute. He completed his Ph.D. inDevelopmental Psychology at Arizona State University in200 I. Over the last few years he has been involved withsimulation training research at the Air Force ResearchLaboratory in Mesa, AZ.

BRIAN T. SCHREmER is a Senior Research Scientistwith Lumir Research Institute at the Air Force ResearchLaboratory, Human Effectiveness Directorate WarfighterReadiness Research Division in Mesa, AZ and has anM.S. from the University of Illinois in Champaign­Urbana. For over the past decade, his research hasfocused on Air Force simulation training techniques andmethodologies.

WINSTON BENNETT JR. PhD is a Senior ResearchPsychologist and team leader for the training systemstechnology and performance assessment at the Air ForceResearch Laboratory, Human Effectiveness DirectorateWarfighter Readiness Research Division in Mesa, AZ.He received his PhD in Industrial/OrganizationalPsychology from Texas A&M University in 1995.

KENNETH B KLEINLEIN is a Project Engineer withLockheed Martin and serves as the DMO TestbedDirector and an F-16 Subject Matter Expert at the AirForce Research Laboratory, Warfighter ReadinessResearch Division in Mesa, AZ and has an MAS fromEmbry-Riddle Aeronautical University and a BS from theU.S. Air Force Academy. He is an experienced USAFand commercial pilot with more than 5000 hours of flighttime including 2500+ hours of fighter experience.

17