EXPERIMENTAL FRAMEWORK FOR EVALUATING COGNITIVE WORKLOAD OF USING AR SYSTEM IN GENERAL ASSEMBLY TASK

Lei Hou and Xiangyu Wang*

Faculty of Built Environment, the University of New South Wales, Australia

* Corresponding author (xiangyu.wang@unsw.edu.au)

ABSTRACT: Assembly task is an activity of collecting parts/components and bringing them together through assembly

operations to perform one or more of several primary functions. As an emerging and powerful technology, Augmented

Reality (AR) integrates images of virtual objects into a real world. Due to its self-characteristic features, AR is envisaged to

provide great potentials in guiding assembly task. In this paper, it reviews some AR applications in the area of assembly and

elaborates the great potentials of integrating animated agent with current AR technique in guiding product assembly.

Besides, in view that different assembly operations share certain common features and functions in essence, the authors

formulate the experimental framework for evaluating cognitive workload of using animated AR system in general assembly

task and make the framework general to be applied in evaluating diverse classes of AR systems for different assembly

operations.

Keywords: Augmented Reality, Cognitive Workload, Experimental Framework

1. STATE-OF-THE-ART REVIEW OF

AUGMENTED REALITY FOR PRODUCT

ASSEMBLY

As an emerging and cutting-edge technology, Augmented

Reality (AR) technology integrates images of virtual

objects into a real world. By inserting the virtually

simulated prototypes into the real environment and

creating an augmented scene, AR technology could meet

the goal of enhancing a person’s perception of a virtual

prototyping with real entities. This gives a virtual world a

better connection to the real world, while maintains the

flexibility of the virtual world. Through AR, an

assembler can directly manipulate the virtual components

while identify the potential interferences between the to-

be-assembled objects and the existing objects inside the

real environment. Therefore, in AR environment, an

assembler cannot only interact with real environments,

but also interact with Augmented Environments (AEs).

There are some critical AR applications in assembly area:

In order to obtain the optimized assembly sequence,

Raghavan et al. (1999) adopted AR as an interactive

technique for assembly sequence evaluation and

formulated the assembly planner and liaison graph. In

their work, they have addressed the issue of

automatically generating the most optimized product

assembly sequence in AEs. Besides, Salonen and his

colleagues (2007) used AR technology to conduct their

research in the area of industrial product assembly and

developed a multi-modality system based on the

commonly used AR facility, a head-mounted display

(HMD), a marker-based software toolkit (ARToolKit),

image tracking cameras, web cameras and a microphone.

Additionally, considering the utilization of AR in product

assembly design was based on the marker registration

technology, Xu and others (2008) realized a markerless-

based registration technology, for the purpose of

overcoming the inconveniences of applying markers as

the carrier in assembly design process. Nowadays, the

utilization of AR assembly has extended to a wide range

of products, e.g., furniture assembly design (Zauner et al.,

2003), toy assembly design (Tang et al., 2003), and so on.

Notwithstanding, these research works have achieved

fruitful results, there are still some issues far from being

well solved in the assembly area. For instance, previous

S19-1

625

works have not completely eliminated the assemblers’

cognitive workload when using AR as an alternative of

manuals. To most previous applications, the virtual

images of to-be-assembled objects are typically

registered for merely reflecting the bilateral or

multilateral position relations of to-be-assembled objects,

but the assembly process is dynamic and it should

include the dynamic context like displacement path,

spatial interference and so on. Accordingly, to acquire the

sequent information context such as assembly path and

fixation forms of part/component, the assemblers still

need a positive cognitive retrieval after they retrieve

these static AR clews in mind.

2. COGNITIVE FACILITATIONS OF

INTEGRATING THE ANIMATED AGENT WITH

AR IN ASSEMBLY TASK

The dynamic requirement of adopting AR in assembly

has raised a promising trade-off: integrate the dynamic

animation with the existing AR facility and make them as

a dynamic augmented agent to guide the assembly task.

This way, collaboration between information retrieval

and task operation could possibly be realized. The

following snapshots briefly present our related work,

where we developed the animated AR system to guide

LEGO toy assembly. In this work, the virtual

counterparts of physical LEGO components (Model

No.8263) and the animated process of assembly are

formulated as AR assembly guidance (Fig. 1).

(a)

(b)

Fig. 1 (a) Snapshots of 65 LEGO components (physical

view and AR views) (b) Using animated AR system in

guiding LEGO assembly.

2.1 Enhancement of information retrieval capacity

In animated AR system, it provides a dynamic

demonstration of consistent information context via

animation segments displayed in each assembly step.

Assemblers could detect the existing dimensions from

already positioned components as well as the registered

ones attached to the virtually to-be-assembled

components from see-through HMD. At the same time,

animation dynamically demonstrates the assembly

process in HMD by approaching the virtually to-be-

assembled objects to the already-assembled ones installed

in the ideal positions (Fig. 2). This enables assemblers to

mimic each assembly step and complete real assembly

operation with great ease. Through demonstrating a series

of virtual animation segments registered in real

environment, AR compensates for the mental and

cognitive gaps between individual differences of

information retrieval capacity and lowers the influences

that task difficulty imposes on individuals. Consequently,

it eases the information retrieval.

Fig. 2 Virtual arrow hints pin-hold assembly.

S19-1

626

2.2 Collaborative assembly guidance

Collaborative guiding is another characteristic feature of

animated AR system. To each step, augmented animation

dynamically and sequentially ushers the position changes

of spatial components by means of the activation of each

animation segment triggered by the assemblers

themselves. When completing each animation segment,

the system turns into a visual tool for presenting the

statically augmented component images, as well as the

attached information. In parallel, the system is

temporarily suspended for the next trigger by assemblers.

During each suspended interval after last bout of guiding

animation, the assemblers have plenty of time to pick up

the components from the rest, and position them properly.

This way, implementing assembly and retrieving

augmented guidance could be proceeded collaboratively

(Fig. 3).

Fig. 3 Step-by-step guidance with completion sequence:

left, right and front (dumper).

2.3 Stimulation of motivation

The fun of interactive experience in animated AR system

might stimulate the task motivation. As Chignell et al.,

(1997) stated that multimedia could produce a rich

sensory experience that not only conveyed information

but also increased motivation and interest to its operator

or viewer and improvement of interactivity is

contributive to the enhancement of assembly motivation,

it is believed that animated AR is a good multimedia to

increase motivation by offering lifelike assembly

guidance environment and enabling interactive operation

to assemblers.

3. EXPERIMENTAL FRAMEWORK FOR

EVALUATING AR SYSTEM IN GENERAL

ASSEMBLY

This section discusses an experimental framework of

how to evaluate cognitive workload when using AR

system for general assembly task (Fig. 4). The authors

make the framework general to be applied in evaluating

diverse classes of AR systems for different assembly

operations, considering that different assembly

operations share certain common features and functions

in essence. The framework can be applied in evaluating

the AR system presented in this paper as well as the other

types of AR systems for other assembly operations.

Fig. 4 Evaluation framework for AR systems for general

assembly.

3.1 Fore-task (Phase 1)

The intent of designing a fore-task is to distinguish the

human subjects according to their different levels of

inherent cognitive capacity. This is prior to the main

experiment and the results can be used to help analyze

the findings in the main experiment. The notion of

S19-1

627

cognitive capacity is related to a person’s ability to

mentally move into a spatial space, navigate in this

environment and manipulate the visuo-spatial imagery.

To date, mental rotation is regarded as a direct and

convenient measurement for human capacity of spatial

object cognition. The principle of mental rotation refers

to a mental processing of objects transformation, rather

than physically rotating the objects. Testers are required

to recognize those objects as mental images and rotate

them mentally, and then they should decide whether one

version of the object image is a reflected version of the

other. As Kosslyn (1994) pointed out, objects in images

first moved along continuous trajectories as they were

transformed and then came the mechanism used in visual

cognition and mental imagery. In view that the task

processing refers to the visuo-spatial input, mental

manipulation and visuo-spatial output, (needs

considerable spatial capacity and cognitive workload), it

is secure and reliable to use mental rotation to roughly

divide different levels of cognition. The fore-task uses

the 24 items testing based on testing sheet. For each item,

there is an original layout of a given spatial object and its

four rotated versions (Fig. 5). However, only two of them

are the same spatial layout of the given one while the

other two play a role in confusion to the testers. For the

testers, they have to try to figure out which two of the

four are the true reflections of the given object. During

the mental rotation task, there are some variables to

identify, e.g., practice effect, individual strategies, gender

difference, academic background and so on. Based on the

performance of mental rotation task, the human subjects

should be divided into different groups (high cognitive

capacity or low cognitive capacity).

Fig. 5 An item in mental rotation test sheet.

3.2 Main experiment and measurement (Phase 2 &3)

The aim of the main experiment is to study the human

subjects’ performance of merging the digital virtual

information (e.g., animated AR guidance) into the real

assembly workspace on the nature of a person’s

cognition as compared with merging the physical

information (e.g., manuals) into the real assembly

workspace. The concurrent task strategy (also known as

secondary task strategy) is supposed to be applied. It

reflects the level of cognitive load imposed by a primary

task. The secondary task entails simple activities that

require sustained attention, such as detecting a visual or

auditory signal, and the typical performance variables are

reaction time, accuracy, error rate and so on. Specifically,

the measurement should contain mental load (which

originates from the interaction between task and subject

characteristics. It provides an indication of the expected

cognitive capacity demands and can be considered as a

priori estimate of the cognitive load), mental effort

(which refers to the cognitive capacity that is actually

allocated to accommodate the demands imposed by the

task, thus it can be considered to reflect the actual

cognitive load) and recorded performance (which is in

terms of learner’s achievements, such as the number of

correct test items, number of errors, time consumption,

etc). According to adding concurrent cognitive task, the

susceptibility of human mental and motor performances

could be examined. This is based on the tentative that to

those who suffer less cognitive load, they may free up

their cognitive capacity to deal with interfering tasks.

The next issue is to address what is considered to be the

appropriate and plausible measurement for human

cognitive workload in assembly task, a criterion for

measuring cognitive load should be constituted. The

measurement of cognitive workload was proved to be

diverse for researchers. The mainstream of measurements

for cognitive workload includes subjective analytical

methods and empirical methods e.g., subjective data

collection and analysis (usually involves a questionnaire

comprising one or multiple semantic differential scales

where the participant can indicate the experienced level

of cognitive load), and rating scale technique (based on

the assumption that people are able to introspect on their

S19-1

628

cognitive processes and to report the amount of mental

effort expended) (Xie & Salvendy, 2000). Most

subjective measures are multidimensional in that they

assess groups of associated variables, such as mental

effort, fatigue, and frustration, which are highly

correlated. Rating scale may appear questionable,

however, it has been demonstrated that people are quite

capable of giving a numerical indication of their

perceived mental burden. What is more, physiological

perspective has also provided us some useful

measurements that are based on the assumption that

changes in cognitive functioning are reflected by

physiological variables. These techniques include

measures of heart activity, brain activity and eye activity.

Typically, the possible trade-off is combining the

subjective analytical methods (questionnaire, interviews)

and objective methods (task performance observation,

time recording), and adopting the rating scale technology

based on questionnaire, e.g., NASA Task Load Index

(Hart, 2006) (Fig. 6) and subjects’ experience evaluation

(Fig. 7), since the subjective workload measurement

techniques using rating scales are easy to use,

inexpensive, reliable, can detect small variations in

workload and provide decent convergent, construct and

discriminate validity (Gimino, 2002), meanwhile, the

objective measurement techniques (task performance

observation and time recording) are robust to conduct the

susceptibility research and enable the experimental

results of both subjective and objective analysis (Mulhall

et al ., 2004). Last but not least, a counterbalanced means

for minimizing the evaluation bias or order effects can

also be applied (see Table 1). This is formulated on the

basis of Wang’s research work (2005), where he

counterbalanced whether the Mixed Reality-based

collaborative virtual environments for pipeline layout

design were evaluated relative to the paper drawing and

vice versa. Users can define any categories of

counterbalanced evaluation in each questionnaire that

handles the evaluation of one method against the other,

and collect subjective data according to ranging scaled

technique, for example, from totally agree, agree,

disagree to totally disagree (4 scales).

Fig. 6 NASA Task Load Index based on questionnaire, a

hierarchical measurement for cognitive workload consists

of six items. Each refers to the workload of a specific

activity).

Fig. 7 Rating both methods in the six aspects based on

the six levels.

S19-1

629

Questionnaire #1 Questionnaire #2

Q1: I felt 3D interactivity in animated AR system aided assembly comprehension. Q2: Overall, compared with paper drawing, the animated AR system better facilitated assembly collaboration tasks. Q3: The animated AR system better facilitated information retrieval. Q4: The animated AR system better facilitated problem-solving. Q5: The animated AR system increased the overall quality of output from the screen view. Q6: The animated AR system better facilitated the quantity of assembly work could complete in a given amount of time. Q7: The animated AR system increased understanding of the guidance and me.

Q1: I felt that 3D interactivity in animated AR system aided assembly comprehension. Q2: Overall, compared with paper drawing, the animated AR system better facilitated assembly collaboration tasks.Q3: The animated AR system better facilitated information retrieval. Q4: The animated AR system better facilitated problem-solving. Q5: The animated AR system increased the overall quality of output from the screen view. Q6: The animated AR system better facilitated the quantity of assembly work could complete in a given amount of time. Q7: The animated AR system increased understanding of the guidance and me.

Table. 1 A counterbalanced means for evaluating whether

AR-based guidance is relative to paper drawing.

4. CONCLUSION AND FUTURE WORK

Based on the reviewed AR applications, this article

proposes some cognitive facilitations of integrating

animated agent with state-of-the-art AR technology. The

main contribution of this article is it formulates an

experimental evaluation framework of validating AR

systems for general assembly tasks. From a procedural

perspective, such a framework elaborates how to apply

the mental rotation task to divide the participants in terms

of cognitive capacity in pre-experimental stage, how to

utilize the secondary task technique to impose a cognitive

workload on assembly task in main experiment, and how

to exert the subjective and objective methods to process

the data collection and alleviate the bias after the

experiment. One of the future work/experimentation is to

investigate whether or not users who are trained under

animated AR system (compared with manuals) are

capable of gaining more usable cognitive resource and

are of more and longer mental resource rehearsal, which

might further facilitate human short-term memory.

5. REFERENCES

[1] Chignell, M. H. & Waterworth, J. A., “Multimedia”,

Handbook of Human Factors and Ergonomics, 2nd ed,

pp. 1808-1861, 1997.

[2] Gimino, A., “Students Investment of Mental Effort”,

Paper presented at the annual meeting of the American

Educational Research Association, 2002.

[3] Hart., “NASA-Task Load Index (NASA-TLX); 20

years later”, Human Factors and Ergonomics Society,

50th Annual Meeting San Francisco, CA, pp. 904—908,

2006.

[4] Raghavan, V., Molineros, J. & Sharma, R.,

“Interactive Evaluation of Assembly Sequences Using

Augmented Reality”, IEEE Transaction on Robotics and

Automation, 15(3), pp. 435-449, 1999.

[5] Salonen, T., Sääski, J. & Hakkarainen, M.,

“Demonstration of Assembly Work Using Augmented

Reality”, Proceedings of the 6th ACM international

conference on Image and video, pp. 120 – 123, 2007.

[6] Tang, A., Owen, C., Biocca, F. & Mou, W.,

“Comparative Effectiveness of Augmented Reality in

Object Assembly”, Proceedings of the SIGCHI

conference on Human factors in computing systems,

2003.

[7] Wang, X. Y., “Application of Mixed Reality in

Architecture, Engineering, and Construction:

Specification, Prototype, and Evaluation”, PhD Thesis,

Purdue University, 2005.

[8] Xie, B. & Salvendy, G., “Prediction of Mental

Workload in Single and Multiple Task Environments”,

International Journal of Cognitive Ergonomics, 4, pp.

213–242, 2000.

[9] Xu, K., Chiaa, K. W. & Cheok, A. D., “Real-time

Camera Tracking for Marker-less and Unprepared

Augmented Reality Environments”, Image and Vision

Computing, pp. 673-689, 2008.

[10] Zauner, J., Haller, M., Brandl, A. & Hartman, W.,

“Authoring of A Mixed Reality Assembly Instructor for

Hierarchical Structures”, pp. 237-246, 2003.

S19-1

630