1

AFOTEC Case Study: F-22 Air Combat Simulation (ACS) Verification and Validation (V&V)

FAA V&V Conference

20 Oct 2011

Jeff Olinger

Technical Director

2

Overview

• AFOTEC Background

• VV&A Process

• ACS Description/capabilities

• V&V Requirements

• V&V Plan/execution

• VV&A Results

• Lessons Learned

3

AFOTEC Mission

AFOTEC tests and evaluates new war-fighting capabilities in operationally realistic

environments, influencing, and informing national resource decisions.

4

Scope & Responsibilities

• Independent operational test agency for AF− Reports directly to CSAF

• Operationally tests all Acquisition Category I, II, & oversight programs− Required by USC Title 10

− Guidance & direction from DoD 5000-series

− ACAT I > $365M RDT&E > $2.19B Procurement

− ACAT II > $140M RDT&E >$660M Procurement

UPDATE

5

Roles & Missions

• Operational testers test the “as is” system

• Developmental testers test the “to be” system

Government Developmental Test exists to insert user inputs into the design and development cycle phase early on. Contractual agreements require trained government DT professionals as partners with industry, otherwise unconstrained cost growth and delays in fielding can occur.

Transparent Planning and Operations

6

Roles and Responsibilities

• DOT&E − Issuing DoD OT&E policy & procedures

− Approves operational test plans for test adequacy

− Reviewing & analyzing results of OT&E for MDAPs

− Providing independent assessments to SecDef, USD(AT&L), & Congress

• AFOTEC− Initial Operational Test & Evaluation, Qualification

Operational Test & Evaluation, Follow-on Test & Evaluation, Operational Utility Evaluation (OUE), Operational Assessment (OA)

• MAJCOMs− Force Development Evaluation, Tactics Development &

Evaluation, Weapons System Evaluation Program, OUE, OA

7

V&V Foundation

• Different V&V requirements for different applications− Technical documentation

− Data sources

− Software

− Test Capabilities

• Common elements− Purpose and process description

− Documentation standards

− Employ agent-based activities to support accreditation

− Criteria to assess risk of use

8

Air Force VV&A Process

Requirement Review Implementation Accreditation

Additional V&V Requirement

V&V Agent(s) V&V Plan

V&V Agent(s) Implement Plan

TRWG Review V&V Final Report

Accreditation Report

Model Development- Upgrades- Backlog (V&V)

Technical Review Working Group

- Accreditation Agent- V&V Agent(s)- Model Manager- Domain/Subj Matter

Experts- Other Experts (As

Req’d)

Accreditor Accredits M&S

Extracted from AFI 16-1001 Fig 1

9

ACS Fundamental Requirements

• Installed system performance− Production representative system

• Software-in-the-loop− Actual software complexity & limitations

− Realistic loading on sensor allocation / timelines

• Man-in-the-loop− Man-machine interface critical to overall evaluation

− Thinking / reactive adversary for realistic operations

• Seamless mission flow & realistic relationship between operational tasks

10

ACS Use in F-22 IOT&E

• Specifically created to support operational effectiveness evaluation during F-22 Engineering & Manufacturing Development (EMD)− Simulation requirements for IOT&E developed early in the

EMD program

• Extensive usage during IOT&E preparation & execution− Pilot training

− Test development exercises Culminated in “Test-the-Test” missions

− Formal verification & validation (V&V)

− Accreditation by AFOTEC

− 5 weeks / 152 trials during formal evaluation

11

ACS Description

12

ACS Capabilities

13

V&V Requirements

• Cost: Contractor proprietary

• Schedule: 3 years− Focus on reuse, re-hosting of existing software/models

• Performance: 10% rule− Functional

− Operating

− Fidelity

14

ACS Models

• Requirements identification key to VV&A success− Includes models, modifications, and interfaces

Threat Model Analysis Program (TMAP)

− Fidelity requirements focused on System Under Test (SUT) capabilities

• Create models/modify ACS− Verify stand-alone models and modifications

• New/modified models integrated into ACS− Verify integrated performance

− As model suppliers, IPCs* must work closely with ACS Support fidelity of interaction with other ACS models Real-time requirements

* IPC: Intelligence Production Center (NASIC, MSIC, ONI)

15

V&V Plan

• Accredit ACS for a specific purpose− Evaluate aspects of F-22 performance not obtainable in

OAR

− ACS provides “test for score” environment

• V&V each ACS element and ACS as a whole− Aircraft (F-22)

− Digital Integrated Air Defense System (DIADS)

− Blue and Red airborne players/threats

− Real time surface to air missile (RTSAM)

− Background (player density)

− Environment (ECM, clutter, weather)

− Endgame

• Separate plans…single report approved by Commander

16

V&V Plan

• Used Simulation, Test and Evaluation Process (STEP)− Goal: Give Commander sufficient information to make

sound decision about using ACS for F-22 OT&E

Plan VV&A &

Sys Effect Tests

Predict OAR Results

Conduct

OAR Evaluation

Analyze Data/

Compare Results

Post-Predict

OAR Results

Validate&

Accredit ACS

OT Scenario Evaluation

ACS ACS

Modify Models & Database

Re-Validated ACS Models &

Database

ACS

Verification/Validation Requirements

F-22 Bkgrnd Players

Environment

Endgame

RTSAM

DIADS

Red Airborne

Blue Airborne

17

V&V Execution

• Verification− Prime contractor and major subcontractor efforts

− Government Labs

− Intelligence Centers

• Validation − Conceptual model validation

− Results validation

− Correlation to testing validation

− Face validation

• Validation Working Groups (VWGs)

18

Validation & Accreditation

• Validate performance− Validation Working Groups (VWG)

Stakeholders supported by appropriate subject matter experts

− Correlation to Open Air Range (OAR) is capstone event Test-the-Test (TTT)

− Correct defects – re-verify/re-validate as required

• Accreditation by AFOTEC− V&V process supports accreditation with continuous

documentation

19

Test-the-Test (TTT) Process

• TTT is the final step in validation

• OAR modeled with anticipated test conditions− TMAP followed for exploited or surrogate threats in OAR

− Executed with same procedures, safety limits, etc. as the OAR

− Participants same as OAR – OT&E & Aggressor pilots, etc.

• Results compared following OAR testing− Quantitative metrics: mission-level and supporting MOEs

− Face validation based on participant comments

• Deltas are investigated and explained− Differences in test conditions or procedures identified

− Changes to models assessed & implemented as required

• TTT re-fly− Conduct if necessary

− Correlation of virtual & OAR results must satisfy accreditation authority

20

V&V Results

• Functional verification by contractor teams

• ACS validated by AFOTEC− 98 component experiments

− Flew 2500+ simulated sorties over 21 weeks

• Limitations (all minor)− Four for F-22 system models

− Four for blue airborne models

− Two Digital Integrated Air Defense Systems

− One each red airborne model, background player, endgame, environment

• ACS accredited for use in F-22 OT&E

21

Lessons Learned

• Attributes for VV&A success− Models created with fidelity focused towards replicating

installed system performance required for mission level evaluation Know conditions/variables that drive sensor performance

− Identify data sources that support validation DT scope & methodology are backbone to validation

− Know SUT capabilities & user CONEMP

− VV&A must occur at the subsystem, system, and system of systems levels

22

Summary

• Complex V&V effort− Player interactions

− Man-machine interfaces

• Required to accomplish F-22 evaluation− Represent realistic operational environment

− Mitigate test range limitations

• Keys to success− Early influence

− Leverage existing guidance and practices

− Teamwork

− Document work

23

24

Questions?