Sara Bagassi, University of Bolognasara.bagassi@unibo.it

Wear to Work: SESAR Augmented Reality Technology for Enhancing Air Traffic Operations in Airport Control Towers.RETINA Project

AVARS Conference

6 June 2018, Geneve

RETINA Project 46

RETINA concept

RETINA (Resilient Synthetic Vision for Advanced Control Tower Air Navigation Service Provision) is the

concept of enhancing human sight capabilities and situation awareness in the control tower by means of

synthetic vision.

RETINA Project 47

RETINA concept

In the RETINA concept, controllers are no longer limited by what thehuman eye can physically see out of the tower windows.

As trust in digital data will continue to grow, RETINA’s concept allowsthe controller to have a head-up view of the airport traffic even inlow visibility conditions similar to the synthetic vision currently usedin the cockpit.

RETINA builds upon the technologies developed in SESAR, such asremote tower, safety nets, SWIM, to provide augmented reality toolsfor the tower controller.

RETINA Project 48

RETINA concept: motivation

Image: Liverpool International Airport (nowadays)

Image: Washington International Airport (1962)

RETINA Project 49

*Low Visibility Conditions

*Normal Visibility Conditions

> 12 nm

Low Visibility Procedures

RETINA Project 50

RETINA concept

RETINA Project 51

SENSORS DATASETS 4D MODELAR

INTERFACEAIR TRAFFIC

CONTROLLER

RETINA HMD

SEE-THROUGH HEAD-MOUNTED DISPLAY

DEPARTURES

ID Type/WCAT

ICAO Class CTOT/EOBT

Ready/ Speed

ARRIVALS

ID Type/WCAT ICAO Class

DTD Altitude Speed

Clearance to land

OUTSIDE VIEW

AR OVERLAYCOMBINED

VIEW

RETINA Project 52

RETINA HMD

SEE-THROUGH HEAD-MOUNTED DISPLAY

RETINA Project 53

SENSORS DATASETS 4D MODELAR

INTERFACEAIR TRAFFIC

CONTROLLER

RETINA SD

SEE-THROUGH SPATIAL DISPLAY

DEPARTURES

ID Type/WCAT

ICAO Class CTOT/EOBT

Ready/ Speed

ARRIVALS

ID Type/WCAT ICAO Class

DTD Altitude Speed

Clearance to land

OUTSIDE VIEW

AR OVERLAYCOMBINED

VIEW

RETINA Project 54

RETINA SD

SEE-THROUGH SPATIAL DISPLAY

RETINA Project 55

RETINA Project 56

Validation Platform

RETINA Project 57

Validation Platform

RETINA Project 58

Visibility Conditions

RETINA Project 59

Baseline – Spatial Display –Head Mounted Display

RETINA Project 60

Experimental Plan

Validation Exercise 1-10

SubjectiveMeasurements

Questionnaires: to assess workload, performance andinformation accessibility.

Debriefs: to record the opinions and feelings ofparticipants with respect to the operational concept.

ObjectiveMeasurements

Head down time

Head up time

Number of switches head-down/head-up

Throughput

BatchCONDIVI

SEQUIPMENT

BASELINE HMD SD

Batch1CONDIVIS

1

EXE1

MEDIUM-HIGH TRAFFIC

EXE2

MEDIUM-HIGH TRAFFIC

EXE3

MEDIUM- HIGH TRAFFIC

Batch2CONDIVIS

2

EXE4

MEDIUM TRAFFIC

EXE5

MEDIUM TRAFFIC

EXE7

MEDIUM-HIGH TRAFFIC

EXE6

MEDIUM TRAFFIC

Batch3CONDIVIS

3

EXE8

MEDIUM TRAFFIC

STANDARD RESTRICTIONS

EXE9

MEDIUM TRAFFIC

STANDARD RESTRICTIONS

EXE 10

MEDIUM TRAFFIC LIMITED

RESTRICTIONS

Batch 4EXE11

USABILITY TEST

RETINA Project 61

Results: Head-Down vs Head-Up Time

41%15% 8%

59%85% 92%

0%

20%

40%

60%

80%

100%

Baseline SD HMD

CONDIVIS1

HEAD DOWN TIME HEAD UP TIME

61%

21% 10% 16%

39%

79% 90% 84%

0%

20%

40%

60%

80%

100%

Baseline SD HMD HMD(Med-Hitraffic)

CONDIVIS2

HEAD DOWN TIME HEAD UP TIME

74%

11% 9%

26%

89% 91%

0%

20%

40%

60%

80%

100%

Baseline HMD Standardrestrictions

HMD Limitedrestrictions

CONDIVIS3

HEAD DOWN TIME HEAD UP TIME

• RETINA Solutions Provide a Dramatic Reduction of Head-Down Time

• The effect is maximized by HMD solution

RETINA Project 62

Results: Number of Switches

• RETINA Solutions Provide a Dramatic Reduction of the No. of Switches Head Down-Head Up

• The effect is maximized by HMD solution

193

5578

-

50.0

100.0

150.0

200.0

250.0

Baseline HMD Standardrestrictions

HMD Limitedrestrictions

CONDIVIS3 No. of Switches279

161

75 67

0

50

100

150

200

250

300

Baseline SD HMD HMD(Med-Hitraffic)

CONDIVIS2 No. of Switches

356

213

82

0

50

100

150

200

250

300

350

400

Baseline SD HMD

CONDIVIS1 No. of Switches

RETINA Project 63

Results: Throughput

• RETINA Solutions Provide an overall increase in the throughput

17

20 19

0

5

10

15

20

25

Baseline SD HMD

Mo

vem

ents

/ho

ur

CONDIVIS1 Throughput

1821

19

25

0

5

10

15

20

25

30

Baseline SD HMD HMD (Med-HiTraffic)

Mo

vem

net

s/h

ou

r

CONDIVIS2 Troughput

1714

21

0

5

10

15

20

25

Baseline HMD StandardRestriction

HMD LimitedRestrictions

Mo

vem

ents

/ho

ur

CONDIVIS3 Troughput

RETINA Project 64

Results: Workload and Performance

• RETINA Solution SD provides a significant reduction of all workload components

• RETINA Solution HMD provides an overall increase in performance, but it also increases physical workload

0

5

10

CONDIVIS3 - Perceived Workload (NASA TLX)

BASELINE HMD Standard Restrictions

HMD Limited Restrictions

0

5

10

CONDIVIS2 - Perceived Workload (NASA TLX)

BASELINE SD HMD HMD (MED-HI traffic)

0

5

10

CONDIVIS1 - Perceived Workload (NASA TLX)

BASELINE SD HMD

RETINA Project 65

Results: Information Accessibility

• RETINA Solutions provide an increase in information accessibility

0

5

10

CONDIVIS1 - Information Accessibility

BASELINE SD HMD

02468

10

CONDIVIS2 - Information Accessibility

BASELINE SD HMD HMD (MED-HI traffic)

0

5

10

CONDIVIS3 - Information Accessibility

BASELINE HMD Standard Restrictions

HMD Limited Restrictions

RETINA Project 66

Conclusions

• The RETINA concept has a clear effect in stimulating the ATCO to work in a head-up

position more than in a head-down position.

• The ATCO is provided with a unique conformal representation of all the needed

information that is currently provided by means of several visual inputs.

• When using SD and HMD solution, all workload components are reduced or maintained

except Physical Workload and/or Frustration, slightly increased compared to the one

obtained with baseline equipment.

• When low visibility conditions apply, the use of RETINA tools provides the ATCO with a

head up conformal view of all needed information, leading to the reduction of current

restrictions due to LVP.

This project has received funding from the SESAR Joint Undertaking under the European Union’s

Horizon 2020 research and innovation programme under grant agreement No 699370

The opinions expressed herein reflect the author’s view only.

Under no circumstances shall the SESAR Joint Undertaking be responsible for any use that may be made of the information contained herein.

Thank you very much for your attention!

RETINA Project