11
Validation & Measurement Methods for the PHARE Demonstrations Validation & Measurement Methods for the PHARE Demonstrations
R A WhitakerR A Whitaker
Validation Project LeaderValidation Project Leader
22
Analysis Is Not Easy!Analysis Is Not Easy!
33
Unreasonable DemandsUnreasonable Demands
44
Not UnderstoodNot Understood
55
Does Anyone Listen?Does Anyone Listen?
66
European Wide ValidationEuropean Wide Validation
NATSNATS
DLRDLR
CENACENA
NLRNLR
EECEEC
77
European Wide Problems!European Wide Problems!
88
European Wide SolutionEuropean Wide Solution
99
What is Validation?What is Validation?
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
Validation builds confidence that Validation builds confidence that the system is fit for purposethe system is fit for purpose
1010
Methodology Matureswith TimeMethodology Matureswith Time
PD1
PC/TC
PD1+
PD2
PD2+
PD3
PD1++
1111
Experimental DesignExperimental Design
PD1 Validation PlanningPD1 Validation Planning
1212
PHARE Experimental DesignPHARE Experimental Design
B A S E L IN E
O R G 1Too ls O n ly
O R G 2Too ls + 3 0 % D /L
O R G 3Too ls + 7 0 % D /L
Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be For Users to Understand Results - Baseline must be
Close to Current SystemClose to Current System
1313
PHARE Experimental DesignPHARE Experimental Design
B A S E L IN E
O R G 1Too ls O n ly
O R G 2Too ls + 3 0 % D /L
O R G 3Too ls + 7 0 % D /L
Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be Close to For Users to Understand Results - Baseline must be Close to
Current SystemCurrent System Progressive Introduction of Experimental Variables in New Progressive Introduction of Experimental Variables in New
OrganisationsOrganisations
1414
PHARE Experimental DesignPHARE Experimental Design
B A S E L IN E
O R G 1Too ls O n ly
O R G 2Too ls + 3 0 % D /L
O R G 3Too ls + 7 0 % D /L
Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be Close to For Users to Understand Results - Baseline must be Close to
Current SystemCurrent System Progressive Introduction of Experimental Variables in New Progressive Introduction of Experimental Variables in New
OrganisationsOrganisations
1515
PHARE Experimental DesignPHARE Experimental Design
B A S E L IN E
O R G 1Too ls O n ly
O R G 2Too ls + 3 0 % D /L
O R G 3Too ls + 7 0 % D /L
Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be Close to For Users to Understand Results - Baseline must be Close to
Current SystemCurrent System Progressive Introduction of Experimental Variables in New Progressive Introduction of Experimental Variables in New
OrganisationsOrganisations
1616
Experimental DesignExperimental Design Identifying Data CollectionIdentifying Data Collection
Objective DataObjective Data What; How; FormatWhat; How; Format
Subjective DataSubjective Data QuestionnairesQuestionnaires DebriefsDebriefs
Concentration on Controller Concentration on Controller WorkloadWorkload Instantaneous Self Assessment (ISA)Instantaneous Self Assessment (ISA) NASA TLX (paper)NASA TLX (paper)
PD1 Validation PlanningPD1 Validation Planning
1717
Data Collected In One FileData Collected In One File Tedious data analysisTedious data analysis
No Recorded ObservationsNo Recorded Observations ““so-and-so said such-and-so-and-so said such-and-
such……I think”such……I think”
Limited Material For Debrief Limited Material For Debrief DiscussionsDiscussions
Video Data - but limited Video Data - but limited Controller Cognitive De-BriefController Cognitive De-Brief
PD1 Validation ExecutionPD1 Validation Execution
1818
No Real Thoughts on Analysis No Real Thoughts on Analysis TechniquesTechniques Descriptive StatisticsDescriptive Statistics
Appropriate for Nominal Data?Appropriate for Nominal Data?
Inferential Statistics Employed as Inferential Statistics Employed as Analysis ProgressedAnalysis Progressed
Allowed Confidence IntervalsAllowed Confidence Intervals
Limited Considerations to Limited Considerations to Measures of MeritMeasures of Merit Capacity = fn(workload)?Capacity = fn(workload)? Quality of ServiceQuality of Service
PD1 Validation ExecutionPD1 Validation Execution
1919
PC/TC, PD1+, PD2, PD2+ Validation PreparationPC/TC, PD1+, PD2, PD2+ Validation Preparation
Clearly Defined AimsClearly Defined Aims ObjectivesObjectives
HypothesesHypotheses
Measures of MeritMeasures of Merit Controller WorkloadController Workload Quality of ServiceQuality of Service CapacityCapacity UsabilityUsability
Data RequiredData Required Accessible StorageAccessible Storage
Formal ObservationFormal Observation PsychologistsPsychologists
2020
Formal DocumentationFormal Documentation
Engineering PlanEngineering Plan Layout, tools, equipmentLayout, tools, equipment
Analysis PlanAnalysis Plan Aims, objectives, H0, H1Aims, objectives, H0, H1 DesignDesign
Scenario,Traffic samplesScenario,Traffic samples No. exercises, DurationNo. exercises, Duration Data recordingData recording OrganisationsOrganisations Ensures matched pairsEnsures matched pairs
MeasurementsMeasurements Analysis technique, Stat testsAnalysis technique, Stat tests
2121
PC/TC, PD1+, PD2, PD2+ Validation ExecutionPC/TC, PD1+, PD2, PD2+ Validation Execution
Use of both Descriptive and Use of both Descriptive and Inferential StatisticsInferential Statistics Defined by Analysis PlanDefined by Analysis Plan
Supported with Subjective And Supported with Subjective And Formally Observed DataFormally Observed Data
Immediate Input to DebriefImmediate Input to Debrief Electronic TLXElectronic TLX Trajectory plotsTrajectory plots
Relatively Small and Well Relatively Small and Well Informed Core Group of Informed Core Group of “validators”“validators”
2222
PD3 Validation PlanningPD3 Validation Planning
Validation Group expanded to Validation Group expanded to “non-analysts”“non-analysts”
Needed to Disseminate Needed to Disseminate Knowledge - DocumentedKnowledge - Documented Experimental Plan Experimental Plan
RequirementsRequirements Analysis Plan RequirementsAnalysis Plan Requirements
Aims, Objectives, H0, H1Aims, Objectives, H0, H1 …………..
Analytical MethodsAnalytical Methods
Acted as ConsultancyActed as Consultancy
2323
PD3 Validation ExecutionPD3 Validation Execution
Planning Worked Well - Planning Worked Well - Execution of Analysis Execution of Analysis Limited to CENALimited to CENA
Documentation can Only be Documentation can Only be a Guide to Practitioners of a Guide to Practitioners of Best PracticeBest Practice Can’t Learn Analysis / Can’t Learn Analysis /
Validation from a BookValidation from a Book
2424
PD1++ Validation Planning PD1++ Validation Planning
Planning - “routine”Planning - “routine” Fast-Time Simulation (TAAM) Fast-Time Simulation (TAAM)
IntroducedIntroduced Development of airspace, traffic Development of airspace, traffic
samplessamples Early examination of potential Early examination of potential
problem areasproblem areas
TrainingTraining Introduced Temporary Operating Introduced Temporary Operating
InstructionsInstructions
2525
PD1++ Validation Execution PD1++ Validation Execution
Caught out by ComplacencyCaught out by Complacency Capacity measures used to date Capacity measures used to date
only applicable to non-changing only applicable to non-changing airspaceairspace
Cannot compare capacity between Cannot compare capacity between different airspace designs as used different airspace designs as used in PD1++in PD1++
Initiated research studyInitiated research study
2626
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Core Validation Group with Representation from Core Validation Group with Representation from all Participating Sitesall Participating Sites ...those involved in the validation should have a thorough ...those involved in the validation should have a thorough
understanding of the operational concept under investigation understanding of the operational concept under investigation and of any restrictions/specifics of the simulation environment...and of any restrictions/specifics of the simulation environment...
Common Methodology CrucialCommon Methodology Crucial ……traceability of all input and output data is important for traceability of all input and output data is important for
correct analysis and can not be over-emphasised...correct analysis and can not be over-emphasised... ...the metrics applied in a research programme like PHARE ...the metrics applied in a research programme like PHARE
must be relevant to the performance of real life ATM...must be relevant to the performance of real life ATM...
2727
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Methodology Improved with ExperienceMethodology Improved with Experience ……the Validation Methodology avoided disruptive data the Validation Methodology avoided disruptive data
collection. The use of intrusive measurements like eye collection. The use of intrusive measurements like eye tracking and heart-rate variability measurements could be tracking and heart-rate variability measurements could be valuable additions...valuable additions...
……the application of the appropriate technology to perform the application of the appropriate technology to perform the investigations is very important. The early choice of the investigations is very important. The early choice of PHARE to perform real time simulations has put a too strong PHARE to perform real time simulations has put a too strong focus on this type...focus on this type...
2828
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Validation Should be Applied Throughout Validation Should be Applied Throughout Project - Not Just at the EndProject - Not Just at the End ...there has been too little input from the Validation project in ...there has been too little input from the Validation project in
the beginning of the PD/3 project. In hindsight it can be said the beginning of the PD/3 project. In hindsight it can be said that a stronger focus on the validation objectives of the that a stronger focus on the validation objectives of the project should have led to a better project definition...project should have led to a better project definition...
2929
Validation is a Living SubjectValidation is a Living Subject
PHARE Validation Methodology developed PHARE Validation Methodology developed over time and through experienceover time and through experience
Methods are documentedMethods are documented
3030
Validation is a Living SubjectValidation is a Living Subject
3131
Validation & Measurement Methods for the PHARE Demonstrations Validation & Measurement Methods for the PHARE Demonstrations
R A WhitakerR A Whitaker
Validation Project LeaderValidation Project Leadernextnext