Military GPS User Equipment (MGUE) Integrated System Test (IST) 3-3;

A Statistical Engineering Case StudyLuis A. Cortes, Ph.D., P.E.

The MITRE Corporation

David Hazelton, Ph.D. Roberta Calderwood, Ph.D.SMC/GPEV American Systems Corporation

Rachel Simone, Capt (USAF)SMC/GPEV

5 October 2016

Approved for Public Release; Distribution Unlimited. Case Number 16-3635©2016 The MITRE Corporation. ALL RIGHTS RESERVED.

| 2 || 2 |

Discussion Topics

Introduction

– Policy

– Tradeoff Space in Design and Analysis of Experiments

Military GPS User Equipment (MGUE)

– Program Background

– Developmental Evaluation Framework

– Design of Experiments for DT&E

Design of Experiments for MGUE IST 3-3 Phase 2

– Context

– Application

– Examples (GRAM-S/M)

Lessons learned

| 3 || 3 |

IntroductionDoD Instruction 5000.02

Developmental Evaluation Framework

“Describe an evaluation methodology in the TEMP… to identify key data that will contribute to assessing progress toward achieving: key performance parameters, critical technical parameters, key system attributes, …… and others as needed.”

Design of Experiments

“The Program Manager will…use scientific test and analysis techniques to design an effective and efficient test program that will produce the required data to characterize system behavior across an appropriately selected set of factors and conditions.” ……

“Resource estimates … will be derived from defensible statistical measures of merit (power and confidence) associated with quantification of the differences among the factors affecting operational performance as well as the risk to the government of accepting a poorly performing system or incorrectly rejecting a system with acceptable performance. ”

Enable the delivery of decision-quality information to decision makers

DoD Instruction 5000.02 – Operation of the Defense Acquisition System; Under Secretary of Defense (Acquisition, Technology, & Logistics); 7 January 2015

| 4 || 4 |

Risk of concluding a factor affects the

response when it doesn’t

Number of test runs

Risk of concluding a factor doesn’t affect the

response when it does

IntroductionTradeoff Space in Design and Analysis of Experiments

The need to understand the true nature of the system is unlimited, but the resources to satisfy the need are often scarce

Practical difference to be detected in the

response

| 5 |

Military GPS User Equipment (MGUE)Program Background

The MGUE Program is the User Equipment System of the GPS Enterprise

The program develops and test GPS cards capable of using Military-Code (M-Code) and legacy signals to provide

– improved security

– greater resistance to jamming and interference

– faster crypto key distribution

– increased accuracy

– more robust operation in urban, mountainous, and under foliage environments

MGUE Increment 1 builds upon the GPS Modernized User Equipment (MUE) and MGUE Technology Development efforts to deliver GPS M-Code receivers to lead platforms within each Military Service to prepare the services for M-Code receiver procurements

Milestone A was approved 9 April 2012 with sole source contracts released to Raytheon, Rockwell Collins, and L-3 Interstate Electronics Corporation (L-3/IEC) for Increment 1

Acquisition Decision Memorandum (ADM) of 20 Feb 2014

– directs the program to develop an accelerated acquisition strategy

– supports a Congressional Mandate to provide GPS M-Code-capable equipment for use after FY17

Increment 1 GPS receiver form factors

– Ground Based-GPS Receiver Application Module Modernized (GB-GRAM-M) - ground domain interface

– GRAM Standard Electronics Module Type E M-Code/Modernized (GRAM S/M) - aviation and maritime domain interface

Ref: Enterprise Test & Evaluation Master Plan (E-Temp) for Global Positioning System (GPS) Acquisition Category (ACAT) 1D, Rev. B - 9 Oct 2014. Space and Missile Systems Center (SMC), Global Positioning Systems Directorate (SMC/GP), LA AFB, Los Angeles, CA.

| 6 || 6 |

Military GPS User Equipment (MGUE)Developmental Evaluation Framework

Developmental Test Objectives

Perform TRD requirements verification Provide progress reports via CTPs and TPMs Support CDD platform integration activities Support PEO certification and readiness for OT

through the 63-119 process Inform milestone, production, and fielding decisions

Table 3-15. MGUE Decision Support Questions

DSQ #1 Can GB-GRAM-M support legacy and modernized functions?

DSQ #2 Can GRAM-S/M support legacy and modernized functions?

DSQ #3 Can GB-GRAM-M be integrated into lead platforms?

DSQ #4 Can GRAM-S/M be integrated into lead platforms?

DSQ #5 Is MGUE secure?

Response variables for DOE-based test designs

| 7 || 7 |

Military GPS User Equipment (MGUE)Design of Experiments for DT&E

The DT&E DOE section of the TEMP identifies where to find the DOE-based test design and related information

| 8 || 8 |

Military GPS User Equipment (MGUE)Test Strategy

Incorporating Design of Experiments into the Test Strategy

DT&E follows a progression from laboratory, chamber, and open-air card-level testing to end-to-end integration into lead platforms testing

Service-nominated lead platforms- Army; RQ-11 Raven Small Unmanned Aerial Vehicle- Marine Corps; Joint Light Tactical Vehicle (JLTV)- Air Force; B-2 Bomber- Navy; USS Arleigh Burke Class Destroyer

| 9 |

Incorporated DOE-based test designs into the MGUE test strategy – crafted a sequential test strategy

Decomposed 120 TPMs originally considered for the test – not enough decision-quality information to answer the DSQs

Traced and categorized ~ 9,000 technical requirements – more than enough decision-quality information to answers the DSQs

Identified requirements for test – long test cycle time

Evaluated various courses of action to reduce test cycle time – gaps in requirements testing

Produced doe-based test designs and aided their evaluation with laboratory tests of legacy cards –expect decision-quality information with a reduction in test cycle time

Developed Detailed Test Procedures based on the doe-based test designs – IST 3-3 Phase 2 testing is in progress

Design of Experiments for IST 3-3 Phase 2The Story Board

Model Analysis, Reduction, and Selection

Initial Technical Performance Measures

TPM Decomposition

MGUE IST 3-3 Sequential Test Strategy

“DOE Trees”

| 10 || 10 |

Test Cases GRAM-S/M GB-GRAM

Position Accuracy 48 25

Velocity Accuracy 28 25

UTC Time Transfer Accuracy 11 18

Tracking Accuracy 131 n/a

Acquisition (TTFF) 166 147

Acquisition (TTFF-Multipath) n/a 144

Reacquisition (TTSF & TTRS) 74 130

Reacquisition (TTSF-Multipath) 12 36

Total 470 525

Requirements classification– Receiver Tracking States

– PVT Solution

– Acquisition and Track Sequence

– PVT Solution Aids

– Time to First Fix (TTFF)

– Time to Subsequent Fix (TTSF)

– Anti-spoofing

– Power

– Interface

Verification categorization*– Inspection

– Analysis

– Demonstration

– Test

Test + Demonstration– 937 for GRAM-S/M

– 701 for GB-GRAM

Design of Experiments for IST 3-3 Phase 2Technical Requirements Classification Summary

MGUE IST 3-3 Phase 2 DOE-Based Test Cases

* DOE-Based Test Cases are included in the Integrated System Test (IST) 3-3 Phase 2 Test Plan, GPEV-TP-15-10, 4 March 2016.

| 11 || 11 |

Design of Experiments for IST 3-3 Phase 2Notional Input-Process-Output

FactorsTechnical Performance

Measures

33,600 Factor Combinations(GRAM-S/M)

470 TPMs(GRAM-S/M)

| 12 || 12 |

Design of Experiments for IST 3-3 Phase 2Factor-Response Relationships

FactorsTechnical Performance

Measures

| 13 || 13 |

Design of Experiments for IST 3-3 Phase 2TF 524D Velocity Accuracy Laboratory Test and Analysis

Velocity Accuracy(North, East, Up)

Fixed FactorsBFEA Mitigation; State; Code; Data Rate;

GPS Challenge Environment

Aiding (2)

Design Factors (Levels) Response Variables

Dynamics (2)

Frequency Channel (3)

Velocity Accuracy is the difference between the GPS receiver bi-directional data port (BDDP) velocity and the reference or “ground truth” velocity

Objectives

1. Use the data analysis to get insight into the GRAM-S/M factor space 2. Assess several courses of actions aimed at streamlining the test cycle time

| 14 || 14 |

Design of Experiments for IST 3-3 Phase 2TF 524D Velocity Accuracy Laboratory Test and Analysis

Notional Design Factors and Levels Velocity Errors

Velocity Errors

| 15 || 15 |

Design of Experiments for IST 3-3 Phase 2Test Design Approach

| 16 || 16 |

Design of Experiments for IST 3-3 Phase 2Example 1 -- GRAM-S/M Velocity Accuracy

Responses

– Velocity Accuracy

Fixed factors

– State

– BFEA Mitigation

– GPS Challenge Environment

– Data Rate

Design Factors and Levels

– Aiding

– Dynamics

– Code

– Frequency Channels

| 17 || 17 |

Full FactorialHalf-fraction ModifiedHalf-fraction Modified

Test Points – 12

0.54 < Power < 0.82

D-Efficiency – 1.00

G-Efficiency – 1.00

A-Efficiency – 1.00

Test Points – 24

0.87 < Power < 0.91

D-Efficiency – 1.00

G-Efficiency – 1.00

A-Efficiency – 1.00

Test Points – 12

0.25 < Power < 0.83

D-Efficiency – 0.83

G-Efficiency – 0.65

A-Efficiency – 0.67

Full model terms

– Main effects – Aiding, Dynamics, Code, Frequency Channel

– Two-factor interactions –Aiding*Dynamics, Aiding*Code, Aiding*Frequency Channel, Dynamics*Code, Dynamics*Frequency Channel, Code*Frequency Channel

Laboratory test results of legacy cards and requirements analysis pointed to the following reduced model terms:

– Main effects - Aiding, Dynamics, Code

– Two-factor interactions -Aiding*Dynamics, Aiding*Code, Dynamics*Code

Model Reduction based on F524 Test Results

Design of Experiments for IST 3-3 Phase 2Example 1 -- GRAM-S/M Velocity Accuracy

“Pairwise correlation between coefficients” charts from JMP 12, SAS Corporation

| 18 || 18 |

Design of Experiments for IST 3-3 Phase 2Example 2 -- GRAM-S/M Track Accuracy

Responses

– Deltarange Accuracy

– Pseudorange Accuracy

The Track Accuracy tree was “pruned and grafted” into two designs

– One to accommodate the responses for State 3

– One to accommodate the responses for State 5

Responses can be added linearly

Design factors and Levels

– State – 2 levels

– Dynamics – 2 levels

– Aiding – 2 levels

– Code – 3 levels

– GPS Challenge Environment – 12 levels

Two factors were intentionally confounded into a single factor due to requirements and test-set-up limitations

One factors was intentionally split into two factors

Fixed factors

– Frequency Channel

– Data Rate (M&S consideration)

Full model terms

– Main effects – 4 effects

– Two-factor interaction effects – 6 sets of categorical interactions (A*B, A*C, etc.)

The full model could be reduced to:

– Main effects – 3 effects

– Two-factor interactions – 3 sets of categorical interactions

| 19 || 19 |

Design of Experiments for IST 3-3 Phase 2Example 2 -- GRAM-S/M Track Accuracy (Design 1)

Model Reduction

Test Points – 24

0.27 < Power < 0.97

D-Efficiency – 0.74

G-Efficiency – 0.52

A-Efficiency – 0.44

Test Points – 13

0.07 < Power < 0.16

D-Efficiency – 0.61

G-Efficiency – 0.45

A-Efficiency – 0.33

Test Points – 39

0.42 < Power < 0.99

D-Efficiency – 0.75

G-Efficiency – 0.54

A-Efficiency – 0.45

Test Points – 51

0.60 < Power < 0.99

D-Efficiency – 0.76

G-Efficiency – 0.55

A-Efficiency – 0.46

| 20 || 20 |

Design of Experiments for IST 3-3 Phase 2Example 2 -- GRAM-S/M Track Accuracy (Design 2)

Model Reduction

Test Points 17

0.12 < Power < 0.66

Efficiency – 0.91

G-Efficiency – 0.74

A-Efficiency – 0.80

Test Points 24

0.16 < Power < 0.98

D-Efficiency – 0.92

G-Efficiency – 0.67

A-Efficiency – 0.84

Test Points 64

0.54 < Power < 0.99

D-Efficiency – 0.98

G-Efficiency – 0.87

A-Efficiency – 0.97

Notes: #No other terms can be estimated

| 21 || 21 |

Summary

The application of Design and Analysis of Experiments (DOE) extends beyond helping managing the T&E administrative process and producing defensible test plans

DOE provides essential means for transforming data into knowledge and knowledge into decision-quality information

DOE leveraged the DEF and provides a seamless transition to test design, data analysis, evaluation, assessment, and reporting

DOE is expected to result in significant test efficiencies andto provide better information to inform acquisition decisions

| 22 || 22 |

Key Take-Away

Leverage the Developmental Evaluation Framework and use the results from DOE-based tests to inform the key acquisition decisions

Do not limit the process to loading factors into a software package

Map the factor space and the system requirements

Use historical or test data to inform the test design process

Provide basic DOE training to leadership and critical test team personnel

Assign dedicated personnel versed in DOE and integrate them into the test team

| 23 || 23 |

Acknowledgement

The MGUE DOE Working Group

SMC/GPEV

The MITRE Corporation

American Systems

SPAWARSYSCEN - Pacific

STAT COE

The Aerospace Corporation

THANK YOU!