EUROPEAN ORGANISATION EUROCONTROL · EEC Task AT66 EATCHIP Task DPS.ET1.ST07 Period...

EUROPEAN ORGANISATION

FOR THE SAFETY OF AIR NAVIGATION

EUROCONTROL

EUROCONTROL EXPERIMENTAL CENTRE

Speech Recognition Experiments with theTRACON/Pro ATC Simulator

EEC Note N° 2/95EEC Task AT66

EATCHIP Task DPS. ET1. ST07

Approved by Head of Division 2Issued: February 1995

The information contained in this document is the property of the EUROCONTROL Agency and no part should be reproduced in any form without the Agency's permission. It does not necessarily represent the official policy of the Agency.

NOTE DOCUMENTATION PAGE

Reference:

EEC Note No. 2/95

Security Classification:

UnclassifiedOriginator Code:

EEC Division B2

Originator (Corporate Author) Name/Location:

EUROCONTROL Experimental CentreB. P. 15F 91222 BRETIGNY SUR ORGE CedexTelephone (+33 1) 69 88 75 00

Sponsor Code:

EATCHIP Development Directorate

Sponsor (Contract Authority) Name/Location:

EUROCONTROL Agency96, rue de la FuséeB 1130 BruxellesTelephone (+32 2) 729 35 11

Title: Speech Recognition Experiments with the TRACON/Pro ATC Simulator

Author:Horst HERING

Date

2/95

Pages

50

Figs

11

Refs Annexes

7Det. Task Specification:

EEC Task AT66 EATCHIP Task DPS.ET1.ST07

Period

Feb.1994-May1994

Task No. Sponsor

HMW1

Task No.Originator

AT66Distribution Statement:(a) Controlled by : Head of Division B2(b) Special limitations : None(c) Sent to NTIS : YesDescriptors (keywords): Speech Recognition, Continuous Speech, Speech Interface to ATC

Simulator, Voice Recognition, TRACON/Pro Simulator, Wesson International, Verbex Voice System, Workload Measurement

Abstract: This note presents findings of speech recognition experiments conducted at the Eurocontrol Experimental Centre using Wesson’ s TRACON/Pro ATC (Air Traffic Control) simulator. The built in Verbex Voice System is a continuous speech, speaker dependent, word based system. The results (e.g. mean overall recognition 81,2%) has to been seen under the aspect of the realistic, heavy workload of the subjects during the experiments.

This document has been colated by mechanical means. Should there be missing pages, please report to:

Publications Office

EUROCONTROL Experimental CentreB. P. 1591222 BRETIGNY SUR ORGE CEDEX (France)

EEC Speech Recognition Experiments with the TRACON/Pro ATC Simulator

1

TABLE OF CONTENTS

1. Abstract 3

2. Introduction 3

3. Experimental Environment 33.1 ATC Simulator 33.2 Programming and use of the Verbex System 53.3 Voice Synthesis of the ATC Simulator 53.4 Methods of Subjective Workload Measurement 6

4. Specific Limitations for the Experiment 6

5. Training Course for the Experiment 75.1 ATC Simulator Training 75.2 Voice Training 7

6. Method of Measurement 8

7. Conduct of the Measured Experiment with Voice Input 9

8. Results 10

9. Analysis of the Influence of the specific Experiment setup 13

10. Discussion 18

11. Conclusions 20

12. Acknowledgments 21

13. Bibliography 21

Appendix A 23The Speech Recognition Interface of the TRACON/Pro ATC Simulator

Appendix B 27Extract of the Grammar File used for this Experiment

Appendix C 33Experiment’s Word List

Appendix D 35Script Listing for “QUICKEURO”

Appendix E 43Sample of the used Tally Table

Appendix F 45Recorded Exercise Data

Appendix G 47Exercise Results per Subject


3

1. Abstract

A TRACON/Pro (Terminal Radar Approach Control/Professional) Radar Training System was installed to provide air traffic control tasks for a group of subjects. Experiments causing heavy workload were set up to investigate the influence of different workload measurement methods on the ATC (Air Traffic Control) task. The TRACON/Pro simulator has a built in voice recog-nition device from Verbex Voice Systems. The speaker dependent ‘Verbex’ is a continuous speech, word based system. The aim was to investigate the performance of such a speech rec-ognizer as a voice interface from the controller to an ATC simulator.

2. Introduction

The Eurocontrol Experimental Centre (EEC) report 184 (Meyenberg and Phillipart 1985) and 204 (Prosser and Schmit 1988) describe the previous studies of speech processing at the EEC. The second report was based on the use of the vocal analyser “LOGOS 1” manufactured “LOGICA” France.

The present-day state of the art hardware is quite different from that used in the 1980’s. Nowa-days, manufacturers announce smaller, quicker and more reliable products for speech recogni-tion. A subsidiary aim of this study was to verify the claims of these advertisements.

In spring 1994 the EEC carried out studies of the “Effects of Self-Assessment on Performing Air Traffic Control” documented in a forthcoming EEC report. The speech recognition data for this note were collected during sessions for workload measurement.

3. Experimental Environment

3.1 ATC Simulator

For this experimentation a simulator from Wesson International Inc. Austin/Texas, USA was used. The software house Wesson has integrated off-the-shelf hardware components such as


4

speech recognition and synthesis in a 486 based PC with 2 monitors. A 20” colour monitor operates as radar screen and a 14” monochrome display is used for the flight strips. Input/Out-put devices are keyboard, rolling ball, controller headset with push-to-talk switch and printer. With an optional LAN (Local Area Network) several working positions may simulate a sector-ized ATC centre.

The TRACON/Pro is a PC (Personal Computer) based radar training system designed for mili-tary and civilian air traffic controllers in the USA. The simulator provides interaction with the controller in a very similar way to a real ATC environment. Simulated aircraft manoeuvre according to pre - stored flight plans with FAA (Federal Aviation Administration) specific rules. The acting controller may influence the aircraft trajectory in the simulation by keyboard, rolling ball and/or voice inputs to the computer.

TRACON/Pro is a user configurable system. Several text files are used for the simulator pro-gramming. A similar syntax, to Microsoft Windows (‘xxx.INI’ initialization files) is employed. TRACON/Pro notes and records trainee errors, including conflict warning, low altitude alert and rule violation.

The main TRACON/Pro simulator features are radarscope simulation, free scenario definition, aircraft specification including performance and callsign, pilot voice responses, flight progress strip simulation, weather simulation and the multiple scenario control facility (keyboard, roll-ing ball and voice).

Speech Input(Microphone)

Speech Output(Speaker)

Radar Screen

Flight Info

Speech

Recognizer

Keyboard

Rolling Ball

Speech

Synthesizer

ATC -Simulator

TRACON/Pro


5

3.2 Programming and use of the Verbex System

A continuous voice input system, like the Verbex system, recognizes multiple words in any order (permitted by the grammar) without requiring pauses. Users can speak words in a variety of sequences without having to train the system with each sequence and allows a virtually unlimited number of statements from a defined set of words and numbers. The rules for these sequences are user defined and fixed in ‘grammar’ files. The maximal duration of a sequence is fixed to 24 seconds. The Verbex system interprets a speech gap of more than 400 millisec-onds as end of the last utterance and the start of a new sequence.

The grammar definition files are written in Verbex standard notation (VSN) similar to ‘Backus Normal Form’ in order to shorten and organize the writing of grammar rules. The VSN gram-mar definition files are compiled with a convert utility. The main parts of the grammar files used with the TRACON/Pro simulator are from Wesson International.

Appendix B shows parts of the grammar definition file for this experiment. The reduced set of commands associated with the ‘Quick’ scenario built in the TRACON/Pro simulator was used. After initial difficulties in modifying this scenario, Wesson Inc. send us another version of ‘Quick’ (as Wesson hasn’t planned to modify this scenario).

At system start the compiled files have to be downloaded to the ‘Speech Commander‘ board. In a first voice training (Verbex utility) a personal voice file is created on the speech board. This voice file should be stored on the harddisk of the host computer for later use. Now the rec-ognizer is ready for voice inputs of the trainee. For another person his own personalized voice file has to be downloaded before the system will work.

The hardware of the speech recognition interface is described in appendix A.

3.3 Voice Synthesis of the ATC Simulator

The TRACON/Pro ATC simulator use recorded voices for the pilots responses to the student controller. For this purpose a ‘Sound Blaster’ board from Creative Labs Inc. Santa Clara, USA is used. Pilot responses are recorded as expressions, isolated words and digits. The pilot responses are composed by the computer from the recorded expressions, words and digits, and are synthesised by the ‘Sound Blaster’ board.A complete set of responses should be recorded from a single person. If several persons record a set, the ATC simulator will use different pilot accents (voice recording) for different aircraft.


6

3.4 Methods of Subjective Workload Measurement

The subjective Workload techniques SWAT (Subjective Workload Assessment Technique), ISA (Instantaneous Self Assessment) and NASA - TLX (NASA - Task Load Index) were used for the main experiment.

The SWAT technique requests the assessment for time pressure, mental effort and stress with three different levels.

The ISA technique uses five different levels for the workload assessment. The recording of the ISA data were made with the ERGO (Eurocontrol Recording and Graphical display On-line) system (EEC development for the simulator environment).

During the measured experiments, both techniques were applied verbally and with specific keypads. The frequency of the interrogation was two minutes.

Data collection for the NASA - TLX assessment was made using a questionnaire after the end of the experiment.

4. Specific Limitations for the Experiment

Under the supervision of the Eurocontrol ergonomics expert Dr. H. David a student (Mr. Simon Pledger, Ergonomics Department of Human Sciences, Loughborough University, GB), designed and programmed the setup of the simulator for this experiment. For the scenarios, en-route IFR (Instrument Flight Rule) traffic, was generated. Each scenario produced all aircraft on screen within the first fifteen minutes. The measured traffic samples had 14, 18 or 22 air-craft. For each set of aircraft (14,18 or 18 aircraft) eleven alternative scenarios were available. In each sample set, the same number of possible conflicts (i.e. for fourteen AC the number of conflicts are five, - about one third of number of AC in the sample) were pre-programmed. The number of conflicts might increase as a result of the controller’s actions. The scenarios were designed to cause heavy controller workload to show possible influence of the workload meas-urements on performing the ATC job.

The heavy workload was expected to cause adverse effects on the speech recognition, and while this was only a secondary aspect of the experiment, the results have to be seen in this light.


7

For time reasons the experiment was set up with a limited number of voice instructions only.

The phraseology employed for voice inputs was ‘American’ style (FAA rules), which is slightly different from the ‘European’ one.

The set of available voice commands for the exercise is shown in appendix C.

5. Training Course for the Experiment

5.1 ATC Simulator Training

For the pre - simulation training of the subjects, traffic sets with 10, 12, 14,16,18 20 and 22 AC were used. They were handled by combined keyboard and mouse inputs or with voice, key-board and mouse inputs. The experimenter decided when the maximal workload of a subject was reached and then started with the measured sessions.

5.2 Voice Training

The Verbex voice engine works with known voices only. The initial voice training of a subject consisted of two parts. The first part trained on single words and expressions. The second con-tained sentences made up from words and expressions trained in the first part.

In the first part the training program prompted several times for the voice input of a word. If two digital voice patterns of the same word are recognized as to be similar the system went to the next word.

Phase two of the voice training prompted for the voice input of sentences using words trained in part one. If the recognition was correct the next phrase was processed otherwise the program prompted the speaker to state if he made a speaking error. In the case of an error, the recognizer prompted for a repetitive input otherwise the pronunciation was learned by the program.

For this experiment a simplified voice recognition version of the TRACON/Pro system was used. For our purpose modified voice training based on the TRACON/Pro ‘Quick Voice’, incorporated the relevant phraseology for the scenarios. Phase one needed about five to ten minutes and trained the 53 ‘words’ shown in appendix C. Phase two used 99 scripts and took


8

approximately half an hour. About 90 scripts were taken, in order, from the 343 phrases of the script listing (appendix D), begun at a random point. The system was also trained with some special keywords like ‘noise check’ and ‘disregard’.

Later repetitive voice training consisted of part two only.

Each controller had his own voice pattern stored in a file (recorded and updated during the training). This file had to be downloaded to the Verbex board before the voice engine could work.

6. Method of Measurement

During the measured workload sessions an observer monitored the experiments. The observer watched the acting controllers speech and compared it with what was understood by the Ver-bex voice system. The observation was recorded in a tally table.

Voice inputs and pilot responses are shown in the lower left corner of the radar screen. Each voice I/O (controller’s speech/ pilot response) occupies a line. For a refresh, the old lines are scrolled up (three lines) as history.

The tally table used is shown in appendix E.

Only correct spoken phrases were taken in account. The right structure of a sentence is:

The digits are optional as not all commands (i.e. ‘say speed’) need them. The use of multiple commands with the optional digits in the same utterance is permitted.

The controller may say for example: “TWA one forty nine” or “TransWorld one forty nine” as a callsign. A human pilot will understand both (and other combinations). But the Verbex speech engine will only accept the trained one (TWA one forty nine). For the measurements the use of both is validated as correct speech input. But the second one causes a recognition failure and is counted as such. These problems may be solved by programming but by the time when we became aware of this fact, the experiments were in progress and we were not able to make the required changes.

Callsign Command Digit


9

7. Conduct of the Measured Experiment with Voice Input

The experiments were conducted with eight different controllers from Eurocontrol (Bretigny and Brussels). The seven native English (British and Irish) speaking and one Dutch control-lers’ experience varied from three to fifteen years. The experimental language was English. After familiarization and training with the TRACON/Pro simulator and the speech recognizer the measured experiments took place.

During a day all subjects made five different experiments with the number of AC of their max-imum workload. Four different methods of subjective workload measurements were employed.

Half of the subjects made a sixth experiment a day later (minimum), without any further train-ing in between.

For the experiment the subject was the acting controller of the perturbed ATC sector. He had to assume the incoming traffic and control it, according to the flightplan, to the sector exit points with ‘handover’ to the next control centre. The pre - programmed conflicts had to be prevented by aircraft manoeuvres. The ‘pilot’ (computer) commands had to be given verbally, but the pri-ority were set to fulfil the control task, so in the event of complete failure of verbal inputs, the controller was able to revert to the keyboard.

A single traffic sample took about half an hour. Pauses in between the five control scenarios were made in coordination with the subject.


10

8. Results

This report is based on voice recognition ‘recordings’ between the 25th February 1994 and the 2nd May 1994. 2662 voice inputs to the TRACON/Pro ATC simulator were evaluated, made by eight different controllers within 44 sessions. Each session took about half an hour.

In average during these 22 hours of speech recognition exercises, a voice input was made every 30 seconds.

The observed response time of the speech and simulator system (from end of voice input to the display of the recognized command on the radar screen of the ATC simulator) was only a frac-tion of a second (estimated at 100...200 milliseconds).

The overall performance of the Verbex speech recognition system is shown in the following charts and tables under different aspects.

The detailed results of the recorded tally tables are given in appendix F.

Under the label ‘Reverted to Keyboard’ all unsuccessful voice inputs are collected. No differ-ence is made if a controller after lack of success (with the same command) reverted to key inputs for the simulator or if he changed his mind and stopped a course of action.

All SubjectsRecognized first Time 2162

second Time 259

third Time 55

> third Time 34

Reverted to Keyboard 152

SUM 2662

AAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAA



AAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAA


AAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAA

AAAAA

AAAAAAAAAAAAAAAAAAAAA




AAAAAAAAAAAA






AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

first Time 81,2%

second Time 9,7%

third Time 2,1%> third Time 1,3%

Revert to Keyboard 5,7%

Voice Input Recognized


11

Per Subject

‘Recognized’ means that the complete utterance was correctly interpreted by the speech recog-nizer. The recognition of parts of the message, e.g. only the callsign, command or digits, was very seldom (18 cases of the 2662 inputs: < 0,7%). For this reason appendix F only presents the results of the tally table in a grouped form. Understanding more than was said was rare. Twenty times (< 0,8%), an extra, unspoken command was recognized by the speech engine, and of these only with commands which did not require following digits (e.g. roger, say speed...). This is a consequence of the structure of the grammar (see appendix B). The spoken and the extra, unspoken command were always identical. These cases were counted as correct.

Nbr

. of

Vo

ice

Inpu

ts

Re

cog

nize

d

first

Tim

e

Re

cog

nize

d

seco

nd T

ime

Re

cog

nize

d

thir

d T

ime

Re

cog

nize

d

>th

ird T

ime

Re

vert

ed

to

Ke

yboa

rd

Re

cog

nize

d

first

Tim

e

Re

cog

nize

d

seco

nd T

ime

Re

cog

nize

d

thir

d T

ime

Re

cog

nize

d

>th

ird T

ime

Re

vert

ed

to

Ke

yboa

rd

Subject 1 211 147 45 9 8 2 69,67% 21,33% 4,26% 3,79% 0,95%

Subject 2 267 192 39 9 6 21 71,91% 14,61% 3,37% 2,25% 7,86%

Subject 3 334 248 33 7 7 39 74,25% 9,88% 2,10% 2,10% 11,67

Subject 4 416 351 42 6 3 14 84,38% 10,10% 1,44% 0,72% 3,36%

Subject 5 488 441 28 10 2 7 90,37% 5,74% 2,05% 0,41% 1,43%

Subject 6 409 319 32 7 2 49 78,00% 7,82% 1,71% 0,49% 11,98%

Subject 7 199 158 21 5 6 9 79,40% 10,55% 2,51% 3,02% 4,52%

Subject 8 338 306 19 2 0 11 90,53% 5,62% 0,59% 0,00% 3,26%

0102030405060708090100

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA

AA

AAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAA

AAA

AAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA

AA


AAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAA

AAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAA


AAAAAAAAAAAA

AAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA

AAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

%

S ub ject 1S ub ject 2




Voice Inputs Recogn ized


12

Per Exercise

Exercises 1 to exercise 5 were done successively in one day, preceded by voice training. The sixth exercise was done at least one day later without any further voice training. During the first three exercises the performance was increasing (80,95% - 81,24% - 83,68% general aver-age: 81,22%). This effect of learning seemed to be out weighted by tiredness after one and an half hour of ATC working. The voice recognition performance fell to 79,49% and 78,91%. As shown by ‘Exercise 6’ after a day’s rest controllers achieved high performance. Detailed charts based on the performance of each subject are shown in appendix G. The related tables are in appendix F.

Nb

r. o

f Vo

ice

Inpu

ts

Re

cogn

ize

d

first

Tim

e

Re

cogn

ize

d

seco

nd

Tim

e

Re

cogn

ize

d

thir

d Ti

me

Re

cogn

ize

d

>th

ird

Tim

e

Re

vert

ed

to

Ke

ybo

ard

Re

cogn

ize

d

first

Tim

e

Re

cogn

ize

d

seco

nd

Tim

e

Re

cogn

ize

d

thir

d Ti

me

Re

cogn

ize

d

>th

ird

Tim

e

Re

vert

ed

to

Ke

ybo

ard

Exercise 1 483 391 45 10 9 28 80,95% 9,32% 2,07% 1,86% 5,80%

Exercise 2 501 407 45 15 6 28 81,24% 8,98% 2,99% 1,20% 5,59%

Exercise 3 484 405 47 3 5 24 83,68% 9,71% 0,62% 1,03% 4,96%

Exercise 4 512 407 64 10 6 25 79,49% 12,50% 1,96% 1,17% 4,88%

Exercise5 479 378 47 12 4 38 78,91% 9,81% 2,51% 0,84% 7,93%

Exercise 6 203 174 11 5 4 9 85,71% 5,42% 2,46% 1,97% 4,44%

0102030405060708090100



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAA




AAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA



AAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA



AAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

Ex ercise 1Ex ercise 2


Ex ercise 5

Ex ercise 6*

* (Ex e rc ise 6 mad e by 4 sub je cts,

min . on e day late r )

%

Voice Inpu ts Recogn ized


13

Per Voice Training

There is no real apparent relation between the number of preceding voice training runs and the quality of voice recognition of the Verbex system.

9. Analysis of the Influence of the specific Experiment setup

As the main purpose of the experiments are the workload measurements, ideally the experi-ment setup should not influence the results. Under these aspects the overall performance of the Verbex speech recognition system is shown in the following charts and tables.

Nb

r. o

f Vo

ice

Inpu

ts

Rec

og

nize

d

first

Tim

e

Rec

og

nize

d

seco

nd

Tim

e

Rec

og

nize

d

thir

d T

ime

Rec

og

nize

d

>th

ird

Tim

e

Re

vert

ed

to

Ke

yboa

rd

Rec

og

nize

d

first

Tim

e

Rec

og

nize

d

seco

nd

Tim

e

Rec

og

nize

d

thir

d T

ime

Rec

og

nize

d

>th

ird

Tim

e

Re

vert

ed

to

Ke

yboa

rd

3 Training 537 464 40 7 6 20 86,41% 7,45% 1,30% 1,12% 3,72%

4 Training 478 339 84 18 14 23 70,92% 17,57% 3,77% 2,93% 4,81%

5 Training 904 792 70 16 5 21 87,61% 7,75% 1,77% 0,55% 2,32%

6 Training 743 567 65 14 9 88 76,31% 8,75% 1,89% 1,21% 11,84%


14

Per Workload (Stress)

For the workload measurement experiments each subject should work under his maximum workload level. If a subject failed to perform several scenarios without causing conflicts then he was considered to be at his maximum workload level. He then used the traffic samples below this sample for the measured experimentation. The maximum number of aircraft in a sample was 22 (see 5.1). It may be considered that subjects working with the 22 AC samples have not reached their maximum workload level.

The influence of ‘stress’ for the voice recognition is very high (about 10% in this experiment).

Nbr

. of

Vo

ice

Inpu

ts

Re

cog

nize

d

first

Tim

e

Re

cog

nize

d

seco

nd T

ime

Re

cog

nize

d

thir

d T

ime

Re

cog

nize

d

>th

ird T

ime

Re

vert

ed

to

Ke

yboa

rd

Re

cog

nize

d

first

Tim

e

Re

cog

nize

d

seco

nd T

ime

Re

cog

nize

d

thir

d T

ime

Re

cog

nize

d

>th

ird T

ime

Re

vert

ed

to

Ke

yboa

rd

< 22 Aircraft 800 598 93 21 19 69 74,75% 11,63% 2,62% 2,38% 8,62%

22 Aircraft 1862 1564 166 34 15 83 84,00% 8,91% 1,83% 0,80% 4,46%

0

10

20

30

40

50

60

70

80

90

100





AAAAAAAAAAAA

AAAAAAAAAAAA

AAA



AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA



AAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA






AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA





AAAAAAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA



AAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA

%

Vo ice In p u ts R ecogn ized

w ith 22 A ir c r a f t

less as 22 A ir cr a f t


15

Per Traffic Sample

The degree of difficulty of the traffic samples seems to vary. A large influence on the speech recognition task can be observed (a variation of over 15% of the ‘first time’ recognition).

Nbr

. of

Vo

ice

Inpu

ts

Re

cog

nize

d

first

Tim

e

Re

cog

nize

d

seco

nd T

ime

Re

cog

nize

d

thir

d T

ime

Re

cog

nize

d

>th

ird T

ime

Re

vert

ed

to

Ke

yboa

rd

Re

cog

nize

d

first

Tim

e

Re

cog

nize

d

seco

nd T

ime

Re

cog

nize

d

thir

d T

ime

Re

cog

nize

d

>th

ird T

ime

Re

vert

ed

to

Ke

yboa

rd

Sample A 229 188 19 6 7 9 82,10% 8,30% 2,62% 3,05% 3,93&

Sample B 267 223 20 5 1 18 83,53% 7,49% 1,87% 0,37% 6,74%

Sample C 235 182 22 3 3 25 77,45% 9,36% 1,27% 1,27% 10,64%

Sample D 244 180 30 7 3 24 73,77% 12,30% 2,87% 1,23% 9,83%

Sample E 246 200 21 1 3 21 81,30% 8,54% 0,40% 1,22% 8,54%

Sample F 255 197 27 6 5 20 77,26% 10,59% 2,35% 1,96% 7,84%

Sample G 235 194 27 6 3 5 82,55% 11,49% 2,55% 1,28% 2,13%

Sample H 254 201 30 6 5 12 79,13% 11,81% 2,36% 1,97% 4,73%

Sample I 256 218 27 5 2 4 85,16% 10,55% 1,95% 0,78% 1,56%

Sample J 239 199 26 7 1 6 83,26% 10,88% 2,93% 0,42% 2,51%

Sample K 202 180 10 3 1 8 89,11% 4,95% 1,49% 0,49% 3,96%

0102030405060708090100


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAA

AAAAAAAA

AAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAA

AAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAA


AAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAA

AAAAAAAA

AA

AAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAA

AAAAAAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA


AAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAA

AAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAA

AAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAA




AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAA




AAAAAAAAAAAA


AAAAAAAAAA


AAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAA

AAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA


AAAAAAAAAAAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAAAAA


AAAAAAAA

AAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAA

AAAAAAAA


AAAAAAAAAAAAAAA


AAAAAAAAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA


AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAA

AAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAA


AAAAAAAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA


AAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA




AAAAAAAAAAAA


AAAAAAAAAAAAAAA


AAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAA


AAAAAAAAAAAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAA

Voice Inpu ts Recogn ized%

AB

C D E F G H I J K

Samples


16

Per Method of Measurement

The influence of the less complex ISA method (ERGO System used for the keyboard mode) of the speech recognition system appears to increase the measured performance.

Nb

r. o

f Vo

ice

Inpu

ts

Re

cogn

ize

d

first

Tim

e

Re

cogn

ize

d

seco

nd

Tim

e

Re

cogn

ize

d

thir

d Ti

me

Re

cogn

ize

d

>th

ird

Tim

e

Re

vert

ed

to

Ke

ybo

ard

Re

cogn

ize

d

first

Tim

e

Re

cogn

ize

d

seco

nd

Tim

e

Re

cogn

ize

d

thir

d Ti

me

Re

cogn

ize

d

>th

ird

Tim

e

Re

vert

ed

to

Ke

ybo

ard

None 686 565 56 15 13 37 82,36% 8,16% 2,19% 1,90% 5,39%

ISA (k) 505 433 45 6 6 15 85,74% 8,91% 1,19% 1,19% 2,97%

ISA (v) 494 408 50 10 3 23 82,59% 10,12% 2,02% 0,61% 4,66%

SWAT (k) 512 410 55 6 7 34 80,08% 10,74% 1,17% 1,37% 6,64%

SWAT (v) 465 346 53 18 5 43 74,41% 11,40% 3,87% 1,07% 9,25%

0

10

20

30

40

50

60

70

80

90

100





AAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA



AAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAA



AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAA





AAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA



AAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA





AAAAAAAAAAAA

AAAAAAAAAAAA

AAA



AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA



AAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA






AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA





AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA



AAAAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

%

N on eIS A (k ey b oard )

IS A (v erb al)S W A T (k ey b oard )

S W A T (v erb al)

Vo ice In p u ts R ecogn ized


17

Per Subject - Cumulative

Re

cogn

ize

dfir

st T

ime

Re

cogn

ize

dfir

st a

ndse

con

d Ti

me

Re

cogn

ize

dfir

st,

seco

nda

nd th

ird T

ime

Re

cogn

ize

da

t a

ny T

ry

Subject 1 69,7% 91,0% 95,3% 99,1%

Subject 2 71,9% 86,5% 89,9% 92,1%

Subject 3 74,3% 84,1% 86,2% 88,3%

Subject 4 84,4% 94,5% 95,9% 96,6%

Subject 5 90,4% 96,1% 98,2% 98,6%

Subject 6 78,0% 85,8% 87,5% 88,0%

Subject 7 79,4% 90,0% 92,5% 95,5%

Subject 8 90,5% 96,2% 96,8% 96,8%

0102030405060708090100




AAAAAAAAAAAA

AAA




AAAAAAAA

AA




AAAAAAAA

AA




AAAAAAAA

AA




AAAAAAAAAAAA

AAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA

AA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA

AA




AAAAAAAAAAAA

AAA




AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAA




AAAAAAAA

AA




AAAAAAAAAAAA

AAA




AAAAAAAAAAAA

AAA




AAAAAAAA

AA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAA




AAAAAAAA

AA




AAAAAAAA

AAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAA




AAAAAAAA

AAAA




AAAAAAAA

AAAA




AAAAAAAA

AAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA

AAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA



AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA





AAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA

%

S u bject 1S u bject 2




Voice Inputs Recognized(Cumulative)

first Tim ef irst + secon d Tim e

f irst + secon d + th ird T im ef irst + secon d + th ird + >th ird T im e

f irst + secon d + th ird + >th ird T im e + Referted to Key


18

10. Discussion

In general the system manufacturers claim recognition rates over 95%. During our experiments a overall mean rate of 81.2% valid recognition with the first voice input was reached. Two of our eight subjects slightly the 90% margin.

In a normal human conversation misunderstandings occurs as well. The humans increase the rate of understanding by optical (lips, hands, body, head a.s.o. movements) and other sensitive effects (smelling, feeling...). An off-the-shelf speech recognition device may increase the suc-cess rate by repetitions only. If one repetition of a command could be accepted, the cumulative overall mean rate increase to 90.9% valid recognition (with first or second input of a voice command).

Cumulative Recognition Rate (all subjects)

Considerable progress has been made by industry in speech recognition over the last five to ten years.

The size of the devices has been reduced from a small cupboard (e.g. “LOGOS 1” in 1986) to a plug in card (e.g. “VERBEX”) for a PC.

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA











































AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA




















AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA




















AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

80

85

90

95

100

Recognized f irst Time

Recognized second Time

Recognized th ird Time

Recognized > th ird Time

Revertedto Keyboard

%

%

%

%

%

or Aborted


19

The EEC report 204 noted a reaction time for the recognition of nearly one second (average 0,82...0,99s). During the experiments described here there were no specific measurements of delay, but there was no perceptible delay in recognition (estimated at 100....200 milliseconds).

The increase of the quality of the recognition is very high. The following graph shows the results (voice input recognized first time) of the EEC experiments from 1986/88 and from 1994. In 1986 the experiments subjects spoke phrases from a script - in this experiment they were controlling heavy traffic.

During voice training and measured experiments the same controller headset was used for spo-ken commands for the TRACON/Pro ATC simulator. Pilot responses were heard from a sepa-rate loudspeaker and not via the headset.

The influence of the microphone position at the speaker’s mouth is extremely important. This position always has to be the same for a subject. If the microphone was left, right, over or under the mouth was less important, but the positioning of the microphone during the training passes has to be the same for the recognition sessions. A badly positioned microphone may cause recognition rates near zero!

In our experiment no influence for the speech recognition from the background noise (e.g. pilot responses out of the loudspeaker in time with controller voice input) were noticed. The PTT (press to talk) switch of the headset was locked open during all experiments. This did not cause spurious recognition by the simulator of background noise or normal conversation of the subject with the observers of the experiments.

0

2 5

5 0

7 5

1 0 0%

%

%

%

p er E x e r c ise an d S u b j ec tf r o m E E C R e p o r t 2 0 4 ( 1 9 8 6 / 8 8 )

p e r E x e r c ise an d S u b j ec to f t h i s N o t e ( 1 9 9 4 )

R a t e s o f R e c o gn it io n


20

11. Conclusions

The ‘Method of Measurement’ of the workload and the different difficulty of the ‘Traffic Sam-ples’ seems to have had a small, but significant influence on the measured quality of the speech recognition.

Subjects working under stress (personalized maximum workload level) had less good results.

The Verbex speech recognition device shows in the implemented environment rather good per-formance.

The number of recognition voice training exercises seems not to be significant for the recogni-tion rate in our experiment. This is surprising as “Verbex” recommend to make further voice training in case of increasing lack of success.

The TRACON/Pro system (with Verbex voice recognition) would be a suitable device for training controllers in ATC specific phraseology and technique. The initial speech training stages and the requirement to repeat speech inputs would be acceptable for such self-training.

Speech inputs to pseudo pilots in a complex ATC simulator environment should be speaker independent with a very high success rate. Such a recognizer should also accept many variant forms of instructions, such as are used by controllers in real-life. The Verbex system tested here is not adequate for that purpose.


21

12. Acknowledgments

This voice recognition experiment was only possible in collaboration with the EEC ergonom-ics expert Dr. H. David and the student S. Pledger from Loughborough University (GB). They accepted me very kindly as special observer in their workload experimentation. Thank you as well for the helpful comments for this note.

13. Bibliography

1. Prosser M. & Schmit P. (February 1988) ‘Integration d’un terminal d’entree vocale dans l’environment de simulation du EEC’ Report no. 204, Eurocontrol Experimental Centre.

2. dePutron G.B., Helgeson R.D. & Warner D.E. (September 1992) ‘TRACON/Pro Radar Training System Manual’ Version 3.5, Wesson International Inc.

3. (April 1991) ‘Grammar Development Manual’ Version 3.1, Verbex Voice Systems

5. Hering H. & Coatleven G. (April 1994) ‘ERGO (Version 1) for Instantaneous Self Assess-ment of Workload’ Note no. 24, Eurocontrol Experimental Centre.

5. Pledger S. (July 1994) ‘Effects of Self-Assessment on Performing Air Traffic Control’ Eurocontrol Experimental Centre

6. Armstrong H.G. (April 1987) ‘Subjective Workload Assessment Technique (SWAT): A User’s Guide’ Aerospace Medical Research Laboratory

7. Reid G.B. Eggemeier F.T. & Nygren T.E. (1982) ‘An individual differences approach to SWAT scale development‘ Human Factors Society

8. Hart S.G. “Development of NASA-TLX (Task Load Index): Results of Empirical and The-oretical Research’ Human Mental Workload - North Holland Press


22


23

Appendix A

The Speech Recognition Interface of the TRACON/Pro ATC Simulator

For the speech recognition part of this ATC simulator, the ‘Speech Commander VAT31’ board from Verbex Voice System, New Jersey, USA is exploited. This board is plugged in an empty slot of the simulator PC (16 bit ISA/EISA bus). After initiations and down loading of required software the on-board speech recognition process is independent of the host computer (acting mainly as power supply only). Recognized speech commands from the Speech Commander board reaches the host process (ATC simulator) via an application program interface (API).

A 16-bit analog - digital converter transforms the spoken voice signals into a series of digital values. The recognition processor performs the speech recognition. Therefore the incoming series of digital values are compared with (statistical methods) data series stored during the training. The best fit is taken to be an indicator of the spoken command and send to the speech device manager (control processor) as raw data. The control processor then performs final result processing for the recognition in real time with translation, routing and response genera-tion. A spoken command may have generally two effects:

• control of the speech board only (e.g. “noise check” will start the noise recalibration procedure of the Verbex board only);

• control of the speech board and input for the TRACON/Pro ATC simulator (e.g. “TWA one fifty nine” has to be sent as a callsign to the host [response generation] and changes the dictionary level from callsign to command).

Speech Device(Microphone)

Analog/

Digital

Converter

Speech

Recognition

Processor

Speech

Device

Manager

Verbex Speech Commander VAT31

Application

Program

Interface (API)

ATC Simulator

Program

TRACON/Pro

ATC Simulator Computer


24

The generated response is given to the host via the standard AT - bus system (ISA/EISA). A TSR (Terminate and Stay Resident) program (API) monitors the host system bus. It picks up the information coming from the Verbex board and sends the information to the TRACON/Pro simulation program by emulating keystroke inputs.

The recognition processor is built around the Texas Instrument chip TI 320C31 with 6 MB of RAM (Random Access Memory) running with 40 MHz clock speed. Control processing (speech device manager) is done by an Intel 80286 processor with 12,5 MHz and 4 MB of memory.

For more technical details of the ‘Speech Commander VAT31’ board see the following Verbex data sheet.


25

Verbex Data Sheet


26


27

Appendix B

Extract of the Grammar File used for this Experiment

;*************************************************************; The main grammar & vocabulary starts here...;************************************************************* #grammar main_grammar

#main_structure“noise check”disregard .CALL_SIGN .CMD .CMD*

#end_structure

#structure .CALL_SIGN “American two thirty two” “AirFrance eight thirty seven” “American seven thirty five” “Delta eight forty three” “United one twenty three” “Delta three seventy one” “TWA one fifty nine” “TWA one forty nine” “Delta four ninety three” “JapanAir four forty seven” “Qantas one eighty five” “American one five four zero” “AirCanada eight five zero” “Qantas one eighty four” “Delta six fifty two” “AlItalia three seventy four” “AlItalia eight eighty eight” “American one four eighteen” “United five ninety two” “AirCanada nine ninety four” “Delta one zero fifty six” “USAir three fifty seven”

#end_structure


28

;************************************************;* Numbers and letters are handled in this area;************************************************ #structure .DIGIT

0 .1-9

#end_structure

#structure .1-3123

#end_structure

#structure .1-4.1-34

#end_structure

#structure .1-9.1-456789

#end_structure

#structure .0-50.1-45

#end_structure

#structure .0or505

#end_structure

#structure .0-2012

#end_structure


29

;********************************************;* This is the commands area;******************************************** #structure .CMD

roger“resume own navigation”“resume normal speed”“contact center”“radar contact”“say again”“say heading”“say speed”maintain .1-4/ .DIGIT .0or5 knots.ALTITUDE_CMD .ALT.HEADING_CMD

#end_structure

#structure .ALTITUDE_CMD“climb and maintain”“descend and maintain”

#end_structure

#structure .ALT1/ .DIGIT thousand .1-9 hundred1/ .DIGIT thousand

#end_structure

#structure .HEADING_CMD“turn left” .TURN“turn right” .TURN“fly heading” .HEAD

#end_structure

#structure .TURN.1-9 .0or5 degreesheading .HEAD

#end_structure

#structure .HEAD.0-2 .DIGIT .0or53 .0-5 .0or53 6 0

#end_structure


30

******************************************; GLOBAL keystroke definition section; (This area translates the statements; that are above and puts keystrokes to; them);****************************************** #host_response #translations

#initiator ““ #separator ““

heading ““ hundred ““ degrees ““ knots ““ “noise check” ““roger “/VR” ;adVise Roger“resume own navigation” “/CO” ;Cleared On_course“resume normal speed” “/MSN” ;Maintain Speed Normal“fly heading” “/FH” ;Fly Headingmaintain “/MK” ;Maintain Knots“contact center” “/HC” ;Handoff Center“radar contact” “/R” ;Radar_contactthousand “,” ;commadisregard “\033” ““ ;escape“turn left” “\036\064” ;l arr“turn right” “\036\066” ;r arr“climb and maintain” “\036\070” ;u arr“descend and maintain” “\036\062” ;d arr“say again” “/YG” ;saY aGain“say heading” “/YH” ;saY Heading“say speed” “/YS” ;saY Speed

“American seven thirty five” “~AAL735” “United one twenty three” “~UAL123” “Delta three seventy one” “~DAL371” “American two thirty two” “~AAL232” “AirFrance eight thirty seven” “~AFR837” “Delta eight forty three” “~DAL843” “TWA one fifty nine” “~TWA159” “TWA one forty nine” “~TWA149” “Delta four ninety three” “~DAL493” “JapanAir four forty seven” “~JAL447”


31

“Qantas one eighty five” “~QFA185”“American one five four zero” “~AAL1540” “AirCanada eight five zero” “~ACA850” “Qantas one eighty four” “~QFA184” “Delta six fifty two” “~DAL652” “AlItalia three seventy four” “~AZA374” “AlItalia eight eighty eight” “~AZA888” “American one four eighteen” “~AAL1418” “United five ninety two” “~UAL592” “AirCanada nine ninety four” “~ACA994” “Delta one zero fifty six” “~DAL1056” “USAir three fifty seven” “~USA357”

#end_grammar


32


33

Appendix C

Experiment’s Word List

Word ID Name------- -------------------------- 1 “0” 2 “1” 3 “2” 4 “3” 5 “4” 6 “5” 7 “6” 8 “7” 9 “8” 10 “9” 11 “AirCanada eight five zero” 12 “AirCanada nine ninety four” 13 “AirFrance eight thirty seven” 14 “AlItalia eight eighty eight” 15 “AlItalia three seventy four” 16 “American one five four zero” 17 “American one four eighteen” 18 “American seven thirty five” 19 “American two thirty two” 20 “Delta eight forty three” 21 “Delta four ninety three” 22 “Delta one zero fifty six” 23 “Delta six fifty two” 24 “Delta three seventy one” 25 “JapanAir four forty seven” 26 “Qantas one eighty five” 27 “Qantas one eighty four” 28 “TWA one fifty nine” 29 “TWA one forty nine” 30 “USAir three fifty seven” 31 “United five ninety two” 32 “United one twenty three” 33 “climb and maintain” 34 “contact centre” 35 “degrees” 36 “descend and maintain” 37 “disregard” 38 “fly heading” 39 “heading” 40 “hundred” 41 “knots” 42 “maintain” 43 “noise check”


34

44 “radar contact” 45 “resume normal speed” 46 “resume own navigation” 47 “roger” 48 “say again” 49 “say heading” 50 “say speed” 51 “thousand” 52 “turn left” 53 “turn right”


35

Appendix D

Script Listing for “QUICKEURO”

1. 1 8 0 maintain 1 5 5 knots “fly heading” 0 0 5 “say heading” “descend and maintain” 0 2. “Qantas one eighty four” “climb and maintain” 1 2 thousand 3. “AlItalia three seventy four” “radar contact” “turn right” 8 5 degrees “turn right” 9 5

degrees maintain 3 7 0 4. “JapanAir four forty seven” “say speed” 5. “Delta one zero fifty six” “turn left” 4 0 degrees 6. “Delta eight forty three” “contact centre” 7. “Delta six fifty two” “say again” “turn left” heading 3 3 5 8. “USAir three fifty seven” “resume own navigation” 9. “AirCanada nine ninety four” “turn left” heading 3 6 0 “turn left” 1 0 degrees 10. “AirCanada eight five zero” “climb and maintain” 1 6 thousand 9 hundred roger roger

“turn right” heading 2 1 5 11. “AirFrance eight thirty seven” “resume normal speed” maintain 4 1 0 knots 12. “AlItalia eight eighty eight” “turn left” 3 5 degrees 13. “American one five four zero” “turn left” 6 0 degrees 14. “TWA one fifty nine” “say speed” “resume normal speed” “resume normal speed”

maintain 2 2 5 knots 15. “United five ninety two” “descend and maintain” 3 thousand 5 hundred 16. “Delta three seventy one” “descend and maintain” 1 7 thousand 7 hundred 17. “United one twenty three” “fly heading” 3 6 0 “say heading” maintain 9 5 knots 18. “TWA one forty nine” roger 19. “American two thirty two” “descend and maintain” 4 thousand 4 hundred 20. “American seven thirty five” “descend and maintain” 1 8 thousand 3 hundred 21. “noise check” 22. “Qantas one eighty five” “fly heading” 3 2 5 23. “American one four eighteen” “resume normal speed” “say heading” “turn left”

heading 3 4 0 24. “Delta four ninety three” “contact centre” “fly heading” 0 9 5 “turn right” heading 0 7 5 25. “AirCanada eight five zero” “turn right” heading 1 5 5 “climb and maintain” 1 thousand

6 hundred 26. “TWA one fifty nine” “fly heading” 0 6 5 27. “AlItalia three seventy four” “climb and maintain” 5 thousand 8 hundred 28. “Delta six fifty two” “climb and maintain” 9 thousand 2 hundred 29. “AirFrance eight thirty seven” “contact centre” “turn right” 7 5 degrees 30. “American one four eighteen” “turn left” 5 0 degrees 31. “American two thirty two” “say again” maintain 6 5 knots 32. “American seven thirty five” “resume own navigation” “turn left” 2 0 degrees 33. “USAir three fifty seven” “fly heading” 2 5 5 maintain 4 4 0 knots 34. 4 5 degrees “turn left” heading 1 8 5 “turn left” 9 5 degrees maintain 2 0 35. “Delta eight forty three” maintain 2 3 0 knots 36. “Qantas one eighty five” “descend and maintain” 7 thousand 1 hundred 37. “Delta four ninety three” maintain 8 0 knots 38. “American one five four zero” “radar contact” 39. “TWA one forty nine” “resume own navigation”


36

40. “AirCanada nine ninety four” “resume normal speed” 41. “AlItalia eight eighty eight” “descend and maintain” 1 9 thousand 2 hundred 42. “Delta three seventy one” “say heading” 43. “Delta one zero fifty six” “turn left” 2 5 degrees 44. “JapanAir four forty seven” “contact centre” 45. “United five ninety two” “say again” 46. “United one twenty three” “contact centre” 47. “Qantas one eighty four” “fly heading” 0 7 0 48. “American two thirty two” “descend and maintain” 5 thousand 7 hundred 49. “TWA one forty nine” “contact centre” 50. “Qantas one eighty four” “contact centre” 51. “Delta three seventy one” “turn right” 3 0 degrees 52. “Delta eight forty three” “contact centre” 53. “American two thirty two” “climb and maintain” 2 thousand “climb and maintain”

1 0 thousand “say speed” 54. “AlItalia three seventy four” “contact centre” 55. “TWA one fifty nine” “climb and maintain” 8 thousand 6 hundred 56. “AirFrance eight thirty seven” “descend and maintain” 6 thousand 9 hundred 57. “United one twenty three” “resume normal speed” 58. “United one twenty three” “radar contact” 59. “USAir three fifty seven” roger 60. “Delta four ninety three” roger 61. “AlItalia eight eighty eight” “contact centre” 62. “American one five four zero” “fly heading” 2 6 0 63. “American seven thirty five” “climb and maintain” 1 1 thousand 4 hundred 64. “American one four eighteen” “say heading” 65. “Delta six fifty two” “resume normal speed” 66. “AirCanada eight five zero” “turn left” 8 5 degrees 67. “United five ninety two” “radar contact” 68. “JapanAir four forty seven” “say speed” 69. “noise check” 70. “AirCanada nine ninety four” “turn right” 7 0 degrees 71. “Delta one zero fifty six” “say again” 72. “Qantas one eighty five” “say speed” 73. “American one four eighteen” “radar contact” “resume own navigation” 74. “AlItalia three seventy four” “climb and maintain” 1 3 thousand 5 hundred 75. “Qantas one eighty four” “resume own navigation” 76. “American seven thirty five” “descend and maintain” 1 4 thousand 3 hundred 77. “Delta three seventy one” “climb and maintain” 3 thousand 1 hundred 78. “AirFrance eight thirty seven” “resume own navigation” 79. “American one five four zero” roger 80. “Delta eight forty three” “resume own navigation” 81. “AirCanada nine ninety four” “say heading” 82. “AlItalia eight eighty eight” “resume normal speed” 83. “United five ninety two” “turn right” 5 5 degrees 84. “TWA one forty nine” “resume own navigation” 85. “Delta six fifty two” “contact centre” 86. “TWA one fifty nine” “resume own navigation” 87. “Delta one zero fifty six” “radar contact” 88. “JapanAir four forty seven” “say again”


37

89. “Delta four ninety three” “radar contact” 90. “Qantas one eighty five” “say speed” 91. “USAir three fifty seven” roger 92. “AirCanada eight five zero” maintain 3 2 0 knots 93. “AirCanada nine ninety four” “resume own navigation” 94. “AlItalia eight eighty eight” “turn right” 1 0 degrees 95. “United five ninety two” “turn right” 6 0 degrees 96. “AlItalia three seventy four” “resume normal speed” 97. “Delta six fifty two” “say speed” 98. “TWA one forty nine” “descend and maintain” 7 thousand 8 hundred 99. “USAir three fifty seven” “fly heading” 3 6 0 100. “American one five four zero” maintain 1 6 5 knots 101. “Delta four ninety three” “radar contact” 102. “AirCanada eight five zero” “climb and maintain” 1 4 thousand 6 hundred 103. “Qantas one eighty four” “resume own navigation” 104. “United one twenty three” “descend and maintain” 1 6 thousand 105. “JapanAir four forty seven” “say heading” 106. “American one four eighteen” “resume own navigation” 107. “American seven thirty five” maintain 3 5 5 knots 108. “Delta eight forty three” “fly heading” 0 9 0 109. “AirFrance eight thirty seven” “contact centre” 110. “American two thirty two” “climb and maintain” 8 thousand 111. “Delta one zero fifty six” “radar contact” 112. “Delta three seventy one” “turn left” 1 0 degrees 113. “Qantas one eighty five” “say again” 114. “TWA one fifty nine” maintain 4 0 5 knots 115. “Delta six fifty two” “say again” 116. “AlItalia eight eighty eight” “turn left” 5 0 degrees 117. “Delta one zero fifty six” “contact centre” 118. “Qantas one eighty four” “fly heading” 3 6 0 119. “TWA one fifty nine” “radar contact” 120. “Delta eight forty three” “fly heading” 3 6 0 121. “AirCanada nine ninety four” “say again” 122. “JapanAir four forty seven” “say again” 123. “American one four eighteen” “say speed” 124. “American seven thirty five” “resume own navigation” 125. “AlItalia three seventy four” “climb and maintain” 1 7 thousand 8 hundred 126. “TWA one forty nine” “turn right” 2 0 degrees 127. “American one five four zero” “turn left” 2 5 degrees 128. “Delta three seventy one” “say heading” 129. “Delta four ninety three” “radar contact” 130. “United five ninety two” “turn right” 2 5 degrees 131. “Qantas one eighty five” “radar contact” 132. “United one twenty three” “say again” 133. “USAir three fifty seven” “say again” 134. “AirCanada eight five zero” “contact centre” 135. “AirFrance eight thirty seven” “say heading” 136. “American two thirty two” “say speed” 137. “Delta eight forty three” “say speed” 138. “American two thirty two” “say heading”


38

139. “TWA one forty nine” “contact centre” 140. “United one twenty three” “say speed” 141. “United five ninety two” roger 142. “AirCanada nine ninety four” “climb and maintain” 5 thousand 5 hundred 143. “Delta three seventy one” “descend and maintain” 1 6 thousand 5 hundred 144. “USAir three fifty seven” “resume own navigation” 145. “TWA one fifty nine” “resume normal speed” 146. “AirCanada eight five zero” roger 147. “Delta one zero fifty six” “turn right” 3 0 degrees 148. “American seven thirty five” “contact centre” 149. “AlItalia three seventy four” “say speed” 150. “AlItalia eight eighty eight” “descend and maintain” 1 thousand 6 hundred 151. “JapanAir four forty seven” “fly heading” 0 6 0 152. “Delta six fifty two” “turn left” heading 1 7 5 153. “Qantas one eighty five” “say speed” 154. “Delta four ninety three” “fly heading” 1 8 5 155. “American one five four zero” roger 156. “American one four eighteen” “descend and maintain” 1 9 thousand 4 hundred 157. “Qantas one eighty four” maintain 3 1 0 knots 158. “AirFrance eight thirty seven” “say speed” 159. “Delta one zero fifty six” “climb and maintain” 6 thousand 160. “Delta four ninety three” “descend and maintain” 5 thousand 5 hundred 161. “AirFrance eight thirty seven” “radar contact” 162. “noise check” 163. “AlItalia three seventy four” “say heading” 164. “American seven thirty five” “contact centre” 165. “American one four eighteen” “radar contact” 166. “JapanAir four forty seven” “radar contact” 167. “TWA one fifty nine” “turn right” 6 0 degrees 168. “AlItalia eight eighty eight” “climb and maintain” 1 8 thousand 169. “United one twenty three” roger 170. “Delta three seventy one” “say again” 171. “Qantas one eighty four” maintain 3 3 0 knots 172. “AirCanada nine ninety four” “climb and maintain” 1 8 thousand 3 hundred 173. “AirCanada eight five zero” “climb and maintain” 1 2 thousand 7 hundred 174. “American two thirty two” “turn right” 2 5 degrees 175. “United five ninety two” “climb and maintain” 1 6 thousand 8 hundred 176. “American one five four zero” “resume normal speed” 177. “Delta eight forty three” roger 178. “Qantas one eighty five” “turn left” 7 5 degrees 179. “USAir three fifty seven” maintain 2 7 0 knots 180. “TWA one forty nine” “say speed” 181. “Delta six fifty two” “descend and maintain” 8 thousand 2 hundred 182. “American one four eighteen” “contact centre” “turn right” heading 1 8 0 183. “AlItalia eight eighty eight” “resume normal speed” 184. “Qantas one eighty five” maintain 1 8 5 knots 185. “American seven thirty five” “fly heading” 2 6 0 186. “American one five four zero” “turn right” 5 0 degrees 187. “Delta six fifty two” maintain 4 9 0 knots 188. “AlItalia three seventy four” “radar contact”


39

189. “AirCanada nine ninety four” “fly heading” 3 0 5 190. “Delta eight forty three” “radar contact” 191. “Delta four ninety three” “say speed” 192. “Delta three seventy one” “resume own navigation” 193. “JapanAir four forty seven” “radar contact” 194. “AirCanada eight five zero” “resume own navigation” 195. “Delta one zero fifty six” “turn right” 5 0 degrees 196. “American two thirty two” “say heading” 197. “TWA one fifty nine” “say heading” 198. “Qantas one eighty four” “say heading” 199. “TWA one forty nine” “resume normal speed” 200. “USAir three fifty seven” “resume normal speed” 201. “United five ninety two” “climb and maintain” 6 thousand 8 hundred 202. “AirFrance eight thirty seven” “say again” 203. “United one twenty three” “resume normal speed” 204. “AirCanada nine ninety four” “say speed” 205. “AlItalia eight eighty eight” “say speed” 206. “Qantas one eighty five” “fly heading” 2 4 0 207. “TWA one fifty nine” “say heading” 208. “American one four eighteen” “resume own navigation” 209. “Delta one zero fifty six” “fly heading” 1 2 5 210. “United five ninety two” “fly heading” 1 7 5 211. “AirFrance eight thirty seven” roger 212. “Qantas one eighty four” “contact centre” 213. “USAir three fifty seven” “say again” 214. “TWA one forty nine” “resume normal speed” 215. “Delta eight forty three” “descend and maintain” 3 thousand 7 hundred 216. “American one five four zero” “radar contact” 217. “Delta four ninety three” “fly heading” 3 4 5 218. “JapanAir four forty seven” “resume normal speed” 219. “American two thirty two” “say again” 220. “JapanAir four forty seven” “turn left” heading 1 9 5 221. “Delta three seventy one” “say again” 222. “American seven thirty five” “radar contact” 223. “United one twenty three” “resume normal speed” 224. “AirCanada eight five zero” “say speed” 225. “Delta six fifty two” “say again” 226. “AlItalia three seventy four” “radar contact” 227. “TWA one fifty nine” “say heading” 228. “American seven thirty five” “radar contact” 229. “Qantas one eighty five” “turn left” 4 0 degrees 230. “Delta six fifty two” “say again” 231. “TWA one forty nine” “say speed” 232. “American one four eighteen” “turn left” 8 5 degrees 233. “USAir three fifty seven” “turn right” 9 0 degrees 234. “AirFrance eight thirty seven” “contact centre” 235. “Delta three seventy one” “radar contact” 236. “AlItalia three seventy four” “say again” 237. “Delta four ninety three” “say again” 238. “AlItalia eight eighty eight” “descend and maintain” 1 0 thousand 1 hundred


40

239. “AirCanada nine ninety four” “say speed” 240. “AirCanada eight five zero” “turn right” 2 5 degrees 241. “Delta eight forty three” “climb and maintain” 4 thousand 9 hundred 242. “American one five four zero” “say speed” 243. disregard 244. “American two thirty two” “radar contact” 245. “Delta one zero fifty six” “resume normal speed” 246. “Qantas one eighty four” “resume normal speed” 247. “United five ninety two” “say speed” 248. “United one twenty three” “resume own navigation” 249. “AirCanada nine ninety four” “turn left” heading 3 0 0 250. “Delta four ninety three” “contact centre” 251. “AirFrance eight thirty seven” “resume own navigation” 252. “American two thirty two” “say heading” 253. “American one five four zero” “turn right” 1 5 degrees 254. “Qantas one eighty five” roger 255. “American seven thirty five” “resume normal speed” 256. “AlItalia three seventy four” maintain 3 5 knots 257. “Delta one zero fifty six” “say again” 258. disregard 259. “United one twenty three” “climb and maintain” 5 thousand 1 hundred 260. disregard 261. “USAir three fifty seven” “fly heading” 0 2 5 262. “Delta eight forty three” “radar contact” 263. “AirCanada eight five zero” “descend and maintain” 7 thousand 2 hundred 264. “Delta six fifty two” “climb and maintain” 9 thousand 9 hundred 265. “AlItalia eight eighty eight” “climb and maintain” 1 0 thousand 5 hundred 266. “United five ninety two” “say heading” 267. “TWA one forty nine” “say speed” 268. “American seven thirty five” “turn right” heading 2 7 5 269. “TWA one fifty nine” “turn right” heading 1 6 5 270. “JapanAir four forty seven” roger 271. “American one four eighteen” “contact centre” 272. “Delta three seventy one” “turn right” 1 5 degrees 273. “Qantas one eighty four” “descend and maintain” 3 thousand 9 hundred 274. “noise check” 275. “United five ninety two” “turn left” 6 5 degrees 276. “AirFrance eight thirty seven” “climb and maintain” 2 thousand 7 hundred 277. “AirCanada eight five zero” “turn left” 8 5 degrees 278. “AlItalia three seventy four” “say speed” 279. “Delta six fifty two” “resume own navigation” 280. “Delta one zero fifty six” “fly heading” 3 1 0 281. “AlItalia eight eighty eight” “descend and maintain” 4 thousand 3 hundred 282. “American one four eighteen” “turn right” 3 0 degrees 283. “United one twenty three” “contact centre” 284. “AirCanada nine ninety four” “resume normal speed” 285. “TWA one fifty nine” “resume normal speed” 286. “Delta eight forty three” maintain 4 0 5 knots 287. “American one five four zero” “say speed” 288. “Qantas one eighty five” “fly heading” 3 6 0


41

289. “TWA one forty nine” “resume own navigation” 290. “Delta three seventy one” maintain 1 8 0 knots 291. “JapanAir four forty seven” “contact centre” 292. “USAir three fifty seven” “turn right” 3 5 degrees 293. “Delta four ninety three” “say speed” 294. “American two thirty two” “contact centre” 295. “Qantas one eighty four” roger 296. “American one five four zero” “resume own navigation” 297. “AirCanada nine ninety four” “radar contact” 298. “Qantas one eighty five” “resume own navigation” 299. “Delta three seventy one” “resume own navigation” 300. “Delta eight forty three” “climb and maintain” 1 1 thousand 3 hundred 301. “American two thirty two” “say speed” 302. “JapanAir four forty seven” “climb and maintain” 9 thousand 2 hundred 303. “AlItalia eight eighty eight” “say heading” 304. “AirFrance eight thirty seven” “resume own navigation” 305. “USAir three fifty seven” roger 306. “AlItalia three seventy four” roger 307. “TWA one fifty nine” “say speed” 308. “Delta four ninety three” “turn left” 4 0 degrees 309. “Delta one zero fifty six” “turn left” heading 2 9 0 310. disregard 311. “Qantas one eighty four” “descend and maintain” 1 2 thousand 1 hundred 312. disregard 313. “AirCanada eight five zero” maintain 2 0 0 knots 314. “noise check” 315. “United five ninety two” maintain 1 3 5 knots 316. “TWA one forty nine” “say speed” 317. “American one four eighteen” “say speed” 318. “American seven thirty five” “climb and maintain” 1 thousand 6 hundred 319. “Delta six fifty two” “resume own navigation” 320. “United one twenty three” maintain 9 0 knots 321. “AlItalia eight eighty eight” “resume normal speed” 322. “American seven thirty five” “fly heading” 2 8 5 323. “AirFrance eight thirty seven” roger 324. “AlItalia three seventy four” “turn left” 6 0 degrees 325. “United five ninety two” “say heading” 326. “Delta six fifty two” maintain 3 6 0 knots 327. “USAir three fifty seven” “turn right” 5 0 degrees 328. “American one four eighteen” “resume normal speed” 329. “American two thirty two” “resume own navigation” 330. “Delta three seventy one” “climb and maintain” 7 thousand 8 hundred 331. “AirCanada nine ninety four” “say speed” 332. “Delta four ninety three” “descend and maintain” 1 3 thousand 2 hundred 333. “TWA one forty nine” “turn right” 1 5 degrees 334. “Qantas one eighty four” “contact centre” 335. “Delta one zero fifty six” “resume own navigation” 336. “TWA one fifty nine” “resume normal speed” 337. “JapanAir four forty seven” “turn right” 7 5 degrees 338. “American one five four zero” “radar contact”


42

339. “Delta eight forty three” “descend and maintain” 4 thousand 9 hundred 340. “AirCanada eight five zero” “turn left” 3 5 degrees 341. “AirCanada nine ninety four” “turn right” heading 1 9 5 342. “Qantas one eighty five” maintain 1 2 0 knots 343. “United one twenty three” “resume own navigation”


43

Appendix E

Sample of the used Tally Table

VOICE RECOGNITION TALLY TABLE

Nbr. of Voice CommandsCorrectly InputtedIncorrectly Inputted

Voice RecognitionCorr. Recog. 1st TimeRecog. Fail.: Callsign Command DigitsRevert to Keyboard

Corr. Recog. 2nd TimeRecog. Fail.: Callsign Command DigitsRevert to Keyboard

Corr. Recog. 3rd TimeRecog. Fail.: Callsign Command DigitsRevert to Keyboard

Other....Recog. Fail.: Callsign Command DigitsRevert to Keyboard


44


45

Appendix F

Recorded Exercise Data

Sub

ject

Nb

r.

Exe

rcis

e N

br.

Nbr

. of

Tra

inin

g

Tra

ffic

Sa

mpl

e

Nb

r. o

f A

C

Me

thod

of

Me

asu

rem

ent

Nbr

. of

Vo

ice

Inp

uts

Re

cog

nize

d

first

Tim

e

Re

cog

nize

d

seco

nd T

ime

Re

cog

nize

d

third

Tim

e

Re

cog

nize

d

>th

ird

Tim

e

Re

vert

ed

to

Ke

yboa

rd

Re

cog

nize

d

first

Tim

e

Re

cog

nize

d

seco

nd T

ime

Re

cog

nize

d

third

Tim

e

Re

cog

nize

d

>th

ird

Tim

e

Re

vert

ed

to

Ke

yboa

rd

1 1 4 F 22 - 37 22 11 1 3 0 59,46% 29,73% 2,70% 8,11% 0,00%

1 2 4 G 22 ISA(v) 34 23 6 4 1 0 67,65% 17,65% 11,76% 2,94% 0,00%

1 3 4 H 22 ISA(k) 43 33 8 0 2 0 76,74% 18,60% 0,00% 4,65% 0,00%

1 4 4 I 22 SWAT(v) 49 33 12 3 1 0 67,35% 24,49% 6,12% 2,04% 0,00%

1 5 4 J 22 SWAT(k) 48 36 8 1 1 2 75,00% 16,67% 2,08% 2,08% 4,17%

SUM 211 147 45 9 8 2 69,67% 21,33% 4,27% 3,79% 0,95%

2 1 4 F 18 - 48 37 8 1 1 1 77,08% 16,67% 2,08% 2,08% 2,08%

2 2 4 C 18 ISA(k) 55 47 6 0 2 0 85,45% 10,91% 0,00% 3,64% 0,00%

2 3 4 B 18 ISA(v) 56 43 6 3 0 4 76,79% 10,71% 5,36% 0,00% 7,14%

2 4 4 E 18 SWAT(k) 53 33 12 1 2 5 62,26% 22,64% 1,89% 3,77% 9,43%

2 5 4 D 18 SWAT(v) 55 32 7 4 1 11 58,18% 12,73% 7,27% 1,82% 20,00%

SUM 267 192 39 9 6 21 71,91% 14,61% 3,37% 2,25% 7,87%

3 1 6 A 18 - 68 48 10 3 3 4 70,59% 14,71% 4,41% 4,41% 5,88%

3 2 6 B 18 SWAT(v) 55 41 5 1 1 7 74,55% 9,09% 1,82% 1,82% 12,73%

3 3 6 C 18 SWAT(k) 50 37 5 0 1 7 74,00% 10,00% 0,00% 2,00% 14,00%

3 4 6 D 18 ISA(v) 54 30 9 3 2 10 55,56% 16,67% 5,56% 3,70% 18,52%

3 5 6 E 18 ISA(k) 55 47 3 0 0 5 85,45% 5,45% 0,00% 0,00% 9,09%

3 6 6 K 18 - 52 45 1 0 0 6 86,54% 1,92% 0,00% 0,00% 11,54%

SUM 334 248 33 7 7 39 74,25% 9,88% 2,10% 2,10% 11,68%

4 1 5 K 22 - 68 62 4 1 1 0 91,18% 5,88% 1,47% 1,47% 0,00%

4 2 5 H 22 SWAT(k) 74 52 10 2 1 9 70,27% 13,51% 2,70% 1,35% 12,16%

4 3 5 G 22 SWAT(v) 65 52 10 0 1 2 80,00% 15,38% 0,00% 1,54% 3,08%

4 4 5 J 22 ISA(k) 68 57 8 1 0 2 83,82% 11,76% 1,47% 0,00% 2,94%

4 5 5 I 22 ISA(v) 79 70 8 0 0 1 88,61% 10,13% 0,00% 0,00% 1,27%

4 6 5 A 22 - 62 58 2 2 0 0 93,55% 3,23% 3,23% 0,00% 0,00%

SUM 416 351 42 6 3 14 84,38% 10,10% 1,44% 0,72% 3,37%


46

5 1 5 F 22 - 92 85 2 1 0 4 92,39% 2,17% 1,09% 0,00% 4,35%

5 2 5 J 22 SWAT(v) 91 78 7 5 0 1 85,71% 7,69% 5,49% 0,00% 1,10%

5 3 5 G 22 ISA(v) 93 87 6 0 0 0 93,55% 6,45% 0,00% 0,00% 0,00%

5 4 5 H 22 SWAT(k) 111 98 8 2 1 2 88,29% 7,21% 1,80% 0,90% 1,80%

5 5 5 I 22 ISA(k) 101 93 5 2 1 0 92,08% 4,95% 1,98% 0,99% 0,00%

SUM 486 441 28 10 2 7 90,37% 5,74% 2,05% 0,41% 1,43%

6 1 6 F 22 - 78 53 6 3 1 15 67,95% 7,69% 3,85% 1,28% 19,23%

6 2 6 B 22 ISA(k) 93 82 5 1 0 5 88,17% 5,38% 1,08% 0,00% 5,38%

6 3 6 E 22 SWAT(k) 82 68 5 0 1 8 82,93% 6,10% 0,00% 1,22% 9,76%

6 4 6 D 22 ISA(v) 88 76 9 0 0 3 86,36% 10,23% 0,00% 0,00% 3,41%

6 5 6 C 22 SWAT(v) 68 40 7 3 0 18 58,82% 10,29% 4,41% 0,00% 26,47%

SUM 409 319 32 7 2 49 78,00% 7,82% 1,71% 0,49% 11,98%

7 1 3 K 14 - 32 30 2 0 0 0 93,75% 6,25% 0,00% 0,00% 0,00%

7 2 3 G 14 ISA(k) 43 32 5 2 1 3 74,42% 11,63% 4,65% 2,33% 6,98%

7 3 3 J 14 SWAT(k) 32 28 3 0 0 1 87,50% 9,38% 0,00% 0,00% 3,13%

7 4 3 I 14 ISA(v) 27 22 2 0 0 3 81,48% 7,41% 0,00% 0,00% 11,11%

7 5 3 H 14 SWAT(v) 26 18 4 2 1 1 69,23% 15,38% 7,69% 3,85% 3,85%

7 6 3 A 14 - 39 28 5 1 4 1 71,79% 12,82% 2,56% 10,26% 2,56%

SUM 199 158 21 5 6 9 79,40% 10,55% 2,51% 3,02% 4,52%

8 1 3 A 22 - 60 54 2 0 0 4 90,00% 3,33% 0,00% 0,00% 6,67%

8 2 3 E 22 SWAT(v) 56 52 1 0 0 3 92,86% 1,79% 0,00% 0,00% 5,36%

8 3 3 B 22 ISA(v) 63 57 4 0 0 2 90,48% 6,35% 0,00% 0,00% 3,17%

8 4 3 C 22 SWAT(k) 62 58 4 0 0 0 93,55% 6,45% 0,00% 0,00% 0,00%

8 5 3 D 22 ISA(k) 47 42 5 0 0 0 89,36% 10,64% 0,00% 0,00% 0,00%

8 6 3 K 22 - 50 43 3 2 0 2 86,00% 6,00% 4,00% 0,00% 4,00%

SUM 338 306 19 2 0 11 90,53% 5,62% 0,59% 0,00% 3,25%

Su

bje

ct N

br.

Exe

rcis

e N

br.

Nb

r. o

f T

rain

ing

Tra

ffic

Sa

mp

le

Nbr

. of

AC

Me

tho

d o

f M

ea

sure

me

nt

Nb

r. o

f Vo

ice

Inpu

ts

Re

cogn

ize

d fir

st T

ime

Re

cogn

ize

d se

con

d Ti

me

Re

cogn

ize

d th

ird

Tim

e

Re

cogn

ize

d >

thir

d T

ime

Rev

ert

ed

to

Ke

ybo

ard

Re

cogn

ize

d fir

st T

ime

Re

cogn

ize

d se

con

d Ti

me

Re

cogn

ize

d th

ird

Tim

e

Re

cogn

ize

d >

thir

d T

ime

Rev

ert

ed

to

Ke

ybo

ard


47

Appendix G

Exercise Results per Subject

0102030405060708090100




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAA



AAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA



AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAA

AAAAAAAAA



Ex ercise 5

Ex ercise 6*

* (Exe rc ise 6 made by 4 su b je cts,

min. on e day late r )

%

Voice Inpu ts Recogn izedSub ject 1:

0102030405060708090100




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA



AAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAAAAAA




AAAAAAAA

AAAA



AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA



AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA



AAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAA

AAAAAAAAAAAA

Exercise 1Exercise 2


Exercise 5

Exercise 6*



%

Voice Inpu ts Recogn izedSub ject 2 :


48

0102030405060708090100




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAA


AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAAAAAA




AAAAAAAAAAAA

AAAAAA



AAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA



AAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA



AAAAAAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA



AAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA



Exercise 5

Exercise 6*



%


0102030405060708090100




AAAAAAAA

AAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAAA




AAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAAA




AAAAAAAAAAAA

AAAAAA



AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA



AAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAAA


AAAAAAAAAAAA

AAAAAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA



AAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA



Exercise 5

Exercise 6*



%



49

0102030405060708090100




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAAA




AAAAAAAAAAAA

AAAAAA



AAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA



AAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA



Exercise 5

Exercise 6*



%


0102030405060708090100




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAA

AAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA




AAAAAAAA

AAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA



AAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA



AAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAA



Exercise 5

Exercise 6*



%



50

0102030405060708090100




AAAAAAAA

AAAA




AAAAAAAAAAAA

AAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

AAAAAAAA

AAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAA



AAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA



AAAAAA


AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA



AAAAAA


AAAAAAAAAAAA

AAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA



AAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA



Exercise 5

Exercise 6*



%


0102030405060708090100




AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAA


AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA




AAAAAAAAAAAA

AAAAAAAAA




AAAAAAAAAAAA

AAAAAA

AAAAAAAA

AAAAAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA




AAAAAAAAAAAA

AAAAAA



AAAAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAA

AAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA


AAAAAAAAAAAA

AAAAAAAAAAAAA





AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAA



AAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA



AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


AAAAAAAA

AAAAAA

AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA

AAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAA


AAAAAAAAAAAA

AAAAAAAAA

AAAAAAAAAAAA

AAAAAAAAAAAA

AAAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA



AAAAA


AAAAAAAAAAAAAAAA

AAAAAAAAAAAAAAAA



Exercise 5

Exercise 6*



%


EUROPEAN ORGANISATION EUROCONTROL · EEC Task AT66 EATCHIP Task DPS.ET1.ST07 Period...

Documents

Transcript of EUROPEAN ORGANISATION EUROCONTROL · EEC Task AT66 EATCHIP Task DPS.ET1.ST07 Period...