1

EUROCONTROL CARE - ATM Innovative Concepts

Public Dissemination Forum

EUROCONTROL HQ, Brussels, March 2004

Presenting Uncertainty to Controllers and Pilots

David Nicholls, RM Consultants Ltd, UK

Patrizia Marti and Valentina Barsotti, Deep Blue s.r.l., Italy

risk management consultantsEUROCONTROL

HF Lab EUROCONTROL Human

Factors Laboratory

NEW LOGO

2

Uncertainty in ATM

If there were no uncertainty, would we need controllers and pilots …?

3

OUTLINE

1. Background

2. Objectives, Scope and Methods

3. Uncertainty in Conflict Detection

4 Uncertainty in Multi Sector Planning

5. Transfer to Other Contexts

6. Lessons Learned

4

1. BACKGROUND

5

Preliminary study (2001)

• RMC, University of Glasgow, pilot & HF consultants

• Where is there uncertainty in aviation?• How do stakeholders think about and manage

uncertainty? • Where might there be benefit in presentation?• What presentational forms are already used?

Proposals for main CARE study

6

Types of uncertainty

• Variability • Inaccuracy of measurement, prediction,

display • Missing data, lack of knowledge,

incomplete understanding • Information that is ambiguous or hard to

interpret• Subjective uncertainty - degree of belief

or trust

7

????

Some aviation examples

• When will we get start clearance?

• Accuracy of TCAS • Was that a real GPWS?• What’s it doing now?• Is the tailwind really

only 10 kts? • Land or go-around?

• Why are they sending me this way?

• How long will it take to climb?

• Flight Plan consistency • Did he really

understand?• Shift handover in a

hurry• How big is the storm?

8

we don’t have uncertainty in

ATM

we don’t have uncertainty in

ATMwe can only work with

information that is certain

we can only work with

information that is certain

Current approach to uncertainty

A culture of eliminating or minimising uncertainty:

• more accurate and reliable tools• standard operating procedures • controllers/ pilots see their role as to

reduce uncertainty

9

Why do more?

• Pilots and controllers base some decisions on incomplete or uncertain information, especially where prediction is involved

• New tools tend to increase the use of, and reliance on, predicted information

• Even where better accuracy/ reliability compensates for greater reliance, could we do even better by presenting uncertainty?

10

State of the art

applicable guidance for

ATM

general psychology of uncertainty and decision making experimental

studies for specific aviation tasks and

tools

Number of references on presenting uncertainty

Generic Specific

11

Potential advantages• more comprehensive

picture for decision-making

• better indication that system is straying outside normal limits

• improved situation awareness, attention, enjoyment - an antidote to the problems of automation?

Potential disadvantages

• increased display clutter • increased cognitive load • increased overall

uncertainty in system and potential for confusion, because there is more scope for individual judgment

Some advantages/ disadvantages of presenting

uncertainty

12

2. OBJECTIVES, SCOPE AND METHODS

13

Objectives

• Develop, using case studies, principles for deciding what information about uncertainty is useful, how & to whom it should be presented

• Intended benefits:– practical guidance for ATM in general– contribute to the development of HMIs for

case study tools

14

Case studies

• Conflict prediction (TP/ MTCD/ CORA)

TLS = Tactical Load SmootherMSP = Multi-Sector PlannerTP = Trajectory PredictionMTCD = Medium Term Conflict DetectionCORA = Conflict Resolution Assistant

• Multi Sector Planning (TLS)

• The case studies relate to en-route ATC - not necessarily the most critical uncertainties in ATM

• Selected as examples of interest at the time, but lessons learned may be generalised

15

Scope of study

• Develop and evaluate prototypes to a level that allows an informed decision on more formal development (e.g. full simulation)

• Presentation, not reduction at source • Information about uncertainty, not

uncertain information• Only uncertainties that are to some extent

predictable - i.e. bounded and foreseeable • Not concerned with verbal uncertainty

16

Overview of methods

Multi-Sector Planning• new concept - free to

explore radical options • development based on

CREA! method - extensive user involvement

• new roles, tools and ways of working

• evaluation from animated storyboards

Conflict Prediction• existing tools, close to

operational use• ‘traditional’, problem-

solving design approach • add-ons to existing

HMIs, exploring issues around data, algorithms

• evaluation in ERC Human Factors lab

17

Project organisation

EUROCONTROL HQ

RM Consultants Ltd

Deep Blue s.r.l

EUROCONTROL Research Centre

Graffica

Controller input: ENAV, CANAC, NATS

18

3. UNCERTAINTY IN CONFLICT PREDICTION (TP/ MTCD/ CORA)

19

Development/ evaluation method for conflict prediction

Develop algorithms (simple distributions with real or subjective

data

Develop algorithms (simple distributions with real or subjective

data

Identify suitable tools and displays:

• VAW – uncertainty in climb/ descent

• PPD – uncertainty in conflict time and distance

Identify suitable tools and displays:

• VAW – uncertainty in climb/ descent

• PPD – uncertainty in conflict time and distance

Implement in eDEP platformImplement in eDEP platform

Small-scale evaluation in ERC Human Factors lab

Small-scale evaluation in ERC Human Factors lab

• analysis of MTCD/ TP/ CORA documents, simulation reports, discussions with designers and users

• analysis of MTCD/ TP/ CORA documents, simulation reports, discussions with designers and users

• workshop at ERC to brainstorm and short-list tools and sketch displays

• workshop at ERC to brainstorm and short-list tools and sketch displays

20

Uncertainty in climb/ descent

• Time of initiating climb/ descent: triangular distribution

• Rate of climb/ descent: triangular distribution • Variability about nominal climb/ descent:

normal distribution, growing then decaying with time

• Smoothing and truncation

21

Vertical Aid Window (VAW)

Late, shallow descent

Early, steep descent

22

Uncertainty in conflict time and distance

• Along-track error: normal distribution, linear growth with time

• Cross-track error: uniform distribution, constant with time

• From these the MTCD algorithms calculate confidence bounds on when the CPA will occur and what the separation will be

t = 1

t = 10

t = 1

t = 5

23

PPD without uncertainty T

IME

to

CP

A (

min

utes

)

DISTANCE (nm) at CPA

3

1

2

User might decide to resolve conflict 3 before conflict 2

15

10

5

00 105

24

PPD with uncertaintyT

IME

to

CP

A (

min

utes

)

DISTANCE (nm) at CPA

certain and close

3

1

a long way off and uncertain

a long way off but more certain

2

User might decide to resolve conflict 2 before conflict 3. Greater uncertainty = less urgent, in this context.

15

10

5

0

• Ellipses show 95% confidence on time and distance

• Colour saturation increases as bubble shrinks, to counter the impression that ‘big = important’

0 105

25

Experiment at ERC

• Two groups of 4 controllers, two days each • Controllers worked both PC and TC roles• No pilots, no R/T• High and low traffic samples• Uncertainty displays selected ON or OFF in each

run• Subjective measures – questionnaires, debriefings

NASA TLX workload• Initial results available

26

Questionnaire results

• Majority of negative views, except with regard to workload and colours of PPD

• More found PPD useful than VAW

Number of controllers responding

Yes

No

PPD

increases SA?

helpful?

increases workload?

would like one?

matches perception of u/c

VAW

Yes

No

27

TLX workload

TL

X -

Me

an

an

d S

tan

dar

d D

evi

atio

n

70

50

40

30

ON OFF• For high traffic,

TLX* appears lower when uncertainty is displayed

* TLX is a self-assessed, post-run measure (range 0-100) based on scores for mental and temporal demand, effort, performance and frustration.

High traffic

Low traffic

60

Uncertainty Display

20

28

Comments on controller views

• About half of the controllers were unfamiliar with MTCD/ PPD. Often comments (and scores?) related to these tools in general , not to the uncertainty displays specifically

• There were few occasions in which a conflict was seen against the VAW uncertainty area, and hence not many opportunities to use this display

‘too many windows’

29

Controller suggestions

• Better integration of the 3/ 4 dimensions • Better integration with the whole-traffic picture -

separate windows don’t help build the picture and distract from or clutter the main radar view

• ‘Younger controllers might like it’ • Uncertainty displays could be useful (with

training) for:– longer term planning roles e.g. MSP, – ground control and arrival/ departure planning

(especially in a Collaborative Decision Making environment or with DMAN/AMAN)

30

4. UNCERTAINTY IN MULTI-SECTOR PLANNING

31

Deep Blue presentation ….

32

5. TRANSFER TO OTHER CONTEXTS

33

Transferability

• Case studies were for en-route ATC, and solutions must be context-specific

• But uncertainty presentation may be transferable to other contexts - especially where users have long look-ahead times and/ or major uncertainties in the information they use …..

34

Flight deck

• Pilots may benefit where there is: – increased use of predictive tools– full or partial self-separation (‘Free Flight’) - pilots taking

more responsibility for separation may need a shared appreciation of uncertainty with the controller

– …..?

• Flight deck environment differs from control room:– much of pilots’ work is objective-oriented and immediate– main concern is with own aircraft, not with many– physical space for new displays is limited– …..

35

Arrival/ departure management• ground & tower controllers and airlines involved in

arrival/ departure management

Unc

erta

inty

in E

TA

at

KO

KS

I

0

KO

KSI

AirborneInbound Turn-round Taxi

On

stan

d

Pax

/ Bag

s lo

aded

Pus

h ba

ckE

ngin

e st

art

At r

unw

ay h

old

Tak

e of

f

36

6. LESSONS LEARNED

37

What to present

• Culture of eliminating uncertainty is deeply ingrained, but …..

• Results of MSP study, particular, indicate potential benefits of presentation in some contexts

• Deciding, a priori, what specific uncertainty information will be useful is difficult

38

Who needs uncertainty information?

• Users with time to analyse decisions, and/or major uncertainties in information, such as:– long term planners (e.g MSP) – ground & tower controllers/ airlines?– pilots in Free Flight?

• Less useful for those making immediate decisions from short term information

39

Presentation means …

‘Presentation’ is not limited to explicit display. It

can include:• making the uncertainty evident• showing how the situation may develop

- alternative futures• providing ‘what-if’ interactivity• prompting the user to consider the

certainty of their own judgment

40

Design ‘rules’ - 1

• provide ‘what-if’ interaction• give information about causes - whether

within or outside user control • visualise time - distinguish current and

future • indicate where tools / data are unreliable

or incomplete

Solutions must be scenario-specific, but some ‘rules’ may have wider application:

41

Design ‘rules’ - 2

• present uncertainty on demand• integrate with main traffic picture displays• support correct inferences from new data

(Bayesian updating, avoiding cognitive biases of anchoring/ recency)

• ensure a clear understanding of ‘uncertainty’ but don’t be too ‘mathematical’

….

42

Development process

• A CREA!-based process has been developed and demonstrated

• It uses real-life scenarios to introduce the idea of uncertainty and scenario-based, iterative, user-centered development.

• It appears more likely to lead to acceptable solutions than traditional ‘problem-solving’

• The CREA! method has been found to be scaleable, adaptable and effective