1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project...

11

Validation & Measurement Methods for the PHARE Demonstrations Validation & Measurement Methods for the PHARE Demonstrations

R A WhitakerR A Whitaker

Validation Project LeaderValidation Project Leader

22

Analysis Is Not Easy!Analysis Is Not Easy!

33

Unreasonable DemandsUnreasonable Demands

44

Not UnderstoodNot Understood

55

Does Anyone Listen?Does Anyone Listen?

66

European Wide ValidationEuropean Wide Validation

NATSNATS

DLRDLR

CENACENA

NLRNLR

EECEEC

77

European Wide Problems!European Wide Problems!

88

European Wide SolutionEuropean Wide Solution

99

What is Validation?What is Validation?

““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be

demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”

Validation builds confidence that Validation builds confidence that the system is fit for purposethe system is fit for purpose

1010

Methodology Matureswith TimeMethodology Matureswith Time

PD1

PC/TC

PD1+

PD2

PD2+

PD3

PD1++

1111

Experimental DesignExperimental Design

PD1 Validation PlanningPD1 Validation Planning

1212

PHARE Experimental DesignPHARE Experimental Design

B A S E L IN E

O R G 1Too ls O n ly

O R G 2Too ls + 3 0 % D /L

O R G 3Too ls + 7 0 % D /L

Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be For Users to Understand Results - Baseline must be

Close to Current SystemClose to Current System

1313


B A S E L IN E


O R G 2Too ls + 3 0 % D /L

O R G 3Too ls + 7 0 % D /L

Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be Close to For Users to Understand Results - Baseline must be Close to

Current SystemCurrent System Progressive Introduction of Experimental Variables in New Progressive Introduction of Experimental Variables in New

OrganisationsOrganisations

1414


B A S E L IN E


O R G 2Too ls + 3 0 % D /L

O R G 3Too ls + 7 0 % D /L




1515


B A S E L IN E


O R G 2Too ls + 3 0 % D /L

O R G 3Too ls + 7 0 % D /L




1616

Experimental DesignExperimental Design Identifying Data CollectionIdentifying Data Collection

Objective DataObjective Data What; How; FormatWhat; How; Format

Subjective DataSubjective Data QuestionnairesQuestionnaires DebriefsDebriefs

Concentration on Controller Concentration on Controller WorkloadWorkload Instantaneous Self Assessment (ISA)Instantaneous Self Assessment (ISA) NASA TLX (paper)NASA TLX (paper)


1717

Data Collected In One FileData Collected In One File Tedious data analysisTedious data analysis

No Recorded ObservationsNo Recorded Observations ““so-and-so said such-and-so-and-so said such-and-

such……I think”such……I think”

Limited Material For Debrief Limited Material For Debrief DiscussionsDiscussions

Video Data - but limited Video Data - but limited Controller Cognitive De-BriefController Cognitive De-Brief

PD1 Validation ExecutionPD1 Validation Execution

1818

No Real Thoughts on Analysis No Real Thoughts on Analysis TechniquesTechniques Descriptive StatisticsDescriptive Statistics

Appropriate for Nominal Data?Appropriate for Nominal Data?

Inferential Statistics Employed as Inferential Statistics Employed as Analysis ProgressedAnalysis Progressed

Allowed Confidence IntervalsAllowed Confidence Intervals

Limited Considerations to Limited Considerations to Measures of MeritMeasures of Merit Capacity = fn(workload)?Capacity = fn(workload)? Quality of ServiceQuality of Service


1919

PC/TC, PD1+, PD2, PD2+ Validation PreparationPC/TC, PD1+, PD2, PD2+ Validation Preparation

Clearly Defined AimsClearly Defined Aims ObjectivesObjectives

HypothesesHypotheses

Measures of MeritMeasures of Merit Controller WorkloadController Workload Quality of ServiceQuality of Service CapacityCapacity UsabilityUsability

Data RequiredData Required Accessible StorageAccessible Storage

Formal ObservationFormal Observation PsychologistsPsychologists

2020

Formal DocumentationFormal Documentation

Engineering PlanEngineering Plan Layout, tools, equipmentLayout, tools, equipment

Analysis PlanAnalysis Plan Aims, objectives, H0, H1Aims, objectives, H0, H1 DesignDesign

Scenario,Traffic samplesScenario,Traffic samples No. exercises, DurationNo. exercises, Duration Data recordingData recording OrganisationsOrganisations Ensures matched pairsEnsures matched pairs

MeasurementsMeasurements Analysis technique, Stat testsAnalysis technique, Stat tests

2121

PC/TC, PD1+, PD2, PD2+ Validation ExecutionPC/TC, PD1+, PD2, PD2+ Validation Execution

Use of both Descriptive and Use of both Descriptive and Inferential StatisticsInferential Statistics Defined by Analysis PlanDefined by Analysis Plan

Supported with Subjective And Supported with Subjective And Formally Observed DataFormally Observed Data

Immediate Input to DebriefImmediate Input to Debrief Electronic TLXElectronic TLX Trajectory plotsTrajectory plots

Relatively Small and Well Relatively Small and Well Informed Core Group of Informed Core Group of “validators”“validators”

2222


Validation Group expanded to Validation Group expanded to “non-analysts”“non-analysts”

Needed to Disseminate Needed to Disseminate Knowledge - DocumentedKnowledge - Documented Experimental Plan Experimental Plan

RequirementsRequirements Analysis Plan RequirementsAnalysis Plan Requirements

Aims, Objectives, H0, H1Aims, Objectives, H0, H1 …………..

Analytical MethodsAnalytical Methods

Acted as ConsultancyActed as Consultancy

2323


Planning Worked Well - Planning Worked Well - Execution of Analysis Execution of Analysis Limited to CENALimited to CENA

Documentation can Only be Documentation can Only be a Guide to Practitioners of a Guide to Practitioners of Best PracticeBest Practice Can’t Learn Analysis / Can’t Learn Analysis /

Validation from a BookValidation from a Book

2424

PD1++ Validation Planning PD1++ Validation Planning

Planning - “routine”Planning - “routine” Fast-Time Simulation (TAAM) Fast-Time Simulation (TAAM)

IntroducedIntroduced Development of airspace, traffic Development of airspace, traffic

samplessamples Early examination of potential Early examination of potential

problem areasproblem areas

TrainingTraining Introduced Temporary Operating Introduced Temporary Operating

InstructionsInstructions

2525

PD1++ Validation Execution PD1++ Validation Execution

Caught out by ComplacencyCaught out by Complacency Capacity measures used to date Capacity measures used to date

only applicable to non-changing only applicable to non-changing airspaceairspace

Cannot compare capacity between Cannot compare capacity between different airspace designs as used different airspace designs as used in PD1++in PD1++

Initiated research studyInitiated research study

2626

Effectiveness of PHARE ValidationEffectiveness of PHARE Validation

Core Validation Group with Representation from Core Validation Group with Representation from all Participating Sitesall Participating Sites ...those involved in the validation should have a thorough ...those involved in the validation should have a thorough

understanding of the operational concept under investigation understanding of the operational concept under investigation and of any restrictions/specifics of the simulation environment...and of any restrictions/specifics of the simulation environment...

Common Methodology CrucialCommon Methodology Crucial ……traceability of all input and output data is important for traceability of all input and output data is important for

correct analysis and can not be over-emphasised...correct analysis and can not be over-emphasised... ...the metrics applied in a research programme like PHARE ...the metrics applied in a research programme like PHARE

must be relevant to the performance of real life ATM...must be relevant to the performance of real life ATM...

2727


Methodology Improved with ExperienceMethodology Improved with Experience ……the Validation Methodology avoided disruptive data the Validation Methodology avoided disruptive data

collection. The use of intrusive measurements like eye collection. The use of intrusive measurements like eye tracking and heart-rate variability measurements could be tracking and heart-rate variability measurements could be valuable additions...valuable additions...

……the application of the appropriate technology to perform the application of the appropriate technology to perform the investigations is very important. The early choice of the investigations is very important. The early choice of PHARE to perform real time simulations has put a too strong PHARE to perform real time simulations has put a too strong focus on this type...focus on this type...

2828


Validation Should be Applied Throughout Validation Should be Applied Throughout Project - Not Just at the EndProject - Not Just at the End ...there has been too little input from the Validation project in ...there has been too little input from the Validation project in

the beginning of the PD/3 project. In hindsight it can be said the beginning of the PD/3 project. In hindsight it can be said that a stronger focus on the validation objectives of the that a stronger focus on the validation objectives of the project should have led to a better project definition...project should have led to a better project definition...

2929

Validation is a Living SubjectValidation is a Living Subject

PHARE Validation Methodology developed PHARE Validation Methodology developed over time and through experienceover time and through experience

Methods are documentedMethods are documented

3030

Validation is a Living SubjectValidation is a Living Subject

3131

Validation & Measurement Methods for the PHARE Demonstrations Validation & Measurement Methods for the PHARE Demonstrations

R A WhitakerR A Whitaker

Validation Project LeaderValidation Project Leadernextnext

1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project...

Documents

Transcript of 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project...