Micro Task Interface Evaluation Method
-
Upload
mhilde -
Category
Technology
-
view
162 -
download
1
Transcript of Micro Task Interface Evaluation Method
![Page 1: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/1.jpg)
micro taskss w i p e t o s t a r t
![Page 2: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/2.jpg)
our goal: a standardized method for benchmarking control room interfaces
• Tablet tool for data collection• Can be used stand-alone or • Linked to simulator• Standardized data collection procedure• Standardized method for question generation• Standardized set of questions• Database
![Page 3: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/3.jpg)
micro tasks are for Evaluation HRA Training
![Page 4: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/4.jpg)
Efficiently and objectively benchmark innovative displays against conventional interfaces
![Page 5: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/5.jpg)
IFE process overview display on tablet
Example
IFE will conduct a micro task evaluation of this set-up in December 2015 at a U.S. training simulator
![Page 6: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/6.jpg)
Compare
![Page 7: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/7.jpg)
IFE design concept for overview displays on tablet, developed for the 2015 U.S. simulator study (screen 1 of 3)
![Page 8: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/8.jpg)
exampleinnovative vs. conventional
![Page 9: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/9.jpg)
mass balanceconventional innovative
Do the innovative displays lead to faster, more reliable identifications and decisions?
![Page 10: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/10.jpg)
mass balanceconventional innovative
Is performance (time and reliability) with innovative displays at least as good as with conventional?
![Page 11: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/11.jpg)
How to test performance benefitsof new interface solutions?
![Page 12: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/12.jpg)
Scenario-based methods.
Observational or self-report.
Qualitative insights.
Relatively few data points.
Decontextualised.
Performance based.
Quantitative data.
Large amounts of data.
flavors of t&e methods
![Page 13: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/13.jpg)
micro tasks are…
![Page 14: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/14.jpg)
Large number of questions Related to systems, components, procedures, etc.
Varying levels of difficulty Including higher-level decision making
Different display conditions E.g. innovative vs. conventional displays
We measure response time / accuracy Compare data between conditions If needed, review eye tracking recordings to understand anomalies
detection/decision tasks under time pressure
![Page 15: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/15.jpg)
Micro task tablet app linked to simulatorTablet app receives signals from simulatorTablet app can send signals / commands to simulator
Dynamic scenarios To cover monitoring / vigilance tasks
We can now record operator actions Can record operating of components, e.g. “start RCPs”
New system for aggregating data New system for managing task lists
Makes it easier to set up and manage a study
new in 2015
![Page 16: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/16.jpg)
example
![Page 17: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/17.jpg)
![Page 18: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/18.jpg)
4 operators working in the simulator individually(no communication)
![Page 19: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/19.jpg)
We control which displays are available (eg innovative, conventional)
![Page 20: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/20.jpg)
Instructions:”Please answer the questions correctly, but also as quickly as possible.
It is very important that you work as fast as you can.”
![Page 21: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/21.jpg)
Run 1s w i p e t o s t a r t
![Page 22: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/22.jpg)
How many condensate pumps are running on turbine 31?
1 2 3
s w i p e t o c o n ti n u e
![Page 23: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/23.jpg)
“How many condensate pumps are running on turbine 31?”
Average identification timeConventional 8 secInnovative
6 sec
![Page 24: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/24.jpg)
Is the subcooling margin sufficient?
Yes
s w i p e t o c o n ti n u e
No
![Page 25: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/25.jpg)
Which steam generatorsare faulty?
SG-1
s w i p e t o c o n ti n u e
SG-2 SG-3
![Page 26: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/26.jpg)
Should safety injection be stopped?
Yes
s w i p e t o c o n ti n u e
No
![Page 27: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/27.jpg)
What is the narrow-range level in steam generator 1?
s w i p e t o c o n ti n u e
%
1 2 3 4 5 6 7 8 9 0
37
![Page 28: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/28.jpg)
up to 200 questions per hourwith 4 operators in the simulator working individually, that means we can run up to 800 questions per hour
![Page 29: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/29.jpg)
Tablet can trigger events in the simulator (e.g. start a tube leak)
and receive signal from the simulator (e.g. RCP-2 was started)
![Page 30: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/30.jpg)
videohttps://vimeo.com/131387407
![Page 31: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/31.jpg)
results
![Page 32: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/32.jpg)
data sourcewhere does this data come from?
![Page 33: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/33.jpg)
6840 data points3420 response time measures
3420 accuracy measures
![Page 34: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/34.jpg)
20 operators
![Page 35: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/35.jpg)
5 hours simulator timein total for the whole data collection
![Page 36: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/36.jpg)
innovative displays are superior
![Page 37: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/37.jpg)
HS I m e a n p e rfo rm a n ce t im e s
Cu rre n t e ffe ct: F(2 , 8 ,0 0 3 )=2 1 ,8 8 9 , p = ,0 0 0 5 7
V e rti ca l b a rs d e n o te 0 ,9 5 co n fi d e n ce i n te rva l s
L S D (In n o va ti ve ) O W D (Co n ve n ti o n a l ) L S D a n d O W D
HS I d e sig n
1 5
1 6
1 7
1 8
1 9
2 0
2 1
2 2
2 3
2 4
2 5
2 6
Per
form
ance
tim
e (s
ec)
Operators were faster with innovative displays than with conventional displays
innovative
conventional
Cu rre n t e ffe ct: F(2 , 8 ,0 0 3 )=2 1 ,8 8 9 , p = ,0 0 0 5 7
V e rti ca l b a rs d e n o te 0 ,9 5 co n fi d e n ce i n te rva l s
L S D (In n o va ti ve ) O WD (Co n ve n ti o n a l ) L S D a n d O WD
HS I d e si g n
1 5
1 6
1 7
1 8
1 9
2 0
2 1
2 2
2 3
2 4
2 5
2 6
Per
form
ance
tim
e (s
ec)
Highly statistically significant difference
![Page 38: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/38.jpg)
Slightly more accurate with innovative displays than conventional displays
Percentage of correct answers
innovative conventional innovative
![Page 39: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/39.jpg)
Performance variability
![Page 40: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/40.jpg)
conclusion
![Page 41: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/41.jpg)
As reliable as conventional displays
Faster to read
innovative displays in the 2014 study are
![Page 42: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/42.jpg)
summary
![Page 43: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/43.jpg)
Generate quantitative results about performance benefits (time and reliability) of digital/innovative CR interfaces Highly efficient, objective method Compare to existing reference data Qualitative insights via eye tracking Data directly supports HRA Highly customisable and precise Generate exactly the data needed for the HRA
![Page 44: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/44.jpg)
big picture
micro task tablet app = a mobile companion for Human Factors / Human Reliability specialists
![Page 45: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/45.jpg)
Micro task tablet app
Standardized question setsDatabase
training
hra interfacedesign
gamification
evaluation of training
validation
benchmarkingwithin / across organisations
![Page 46: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/46.jpg)
what next?
![Page 47: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/47.jpg)
Benchmarking studies 2015 / 2016
![Page 48: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/48.jpg)
benchmarking studies
(1) 2015 study at U.S. training simulator analog interfaces vs IFE displays
(2) 2016 study of a partially digital control roomanalog interfaces vs. 1990s-vintage overview display vs IFE display
(3) 2016 study of a fully digital control roomMeasure operator performance and reliability in a fully digital control room and benchmark against data from study 1 and 2
![Page 49: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/49.jpg)
Micro task database Micro tasks for training
Includes gamification approach
Micro tasks for actions outside the control room (field operator, mechanical, etc)
roadmap
![Page 50: Micro Task Interface Evaluation Method](https://reader036.fdocuments.net/reader036/viewer/2022062522/58e786791a28abe7528b5eed/html5/thumbnails/50.jpg)
Micro tasks for team decisions Add secondary task capability Add workload measurement capability
roadmap