Design of a virtual reality training system for human–robot … · proof Int J Interact Des Manuf...

Seediscussions,stats,andauthorprofilesforthispublicationat:https://www.researchgate.net/publication/271837242

Designofavirtualrealitytrainingsystemfor

human–robotcollaborationinmanufacturing

tasks

ArticleinInternationalJournalforInteractiveDesignandManufacturing(IJIDeM)·February2015

DOI:10.1007/s12008-015-0259-2

CITATIONS

9

READS

799

2authors:

Someoftheauthorsofthispublicationarealsoworkingontheserelatedprojects:

AutomationofCFRPpartmanufacturinginsmallbatchesViewproject

Low-costhigh-performanceFusedFillamentFabricationViewproject

EliasMatsas

NationalTechnicalUniversityofAthens

12PUBLICATIONS48CITATIONS

SEEPROFILE

George-ChristopherVosniakos

NationalTechnicalUniversityofAthens

151PUBLICATIONS1,637CITATIONS

SEEPROFILE

AllcontentfollowingthispagewasuploadedbyEliasMatsason06February2015.

Theuserhasrequestedenhancementofthedownloadedfile.

unco

rrec

ted

pro

of

Int J Interact Des Manuf

DOI 10.1007/s12008-015-0259-2

ORIGINAL PAPER

Design of a virtual reality training system for human–robot

collaboration in manufacturing tasks

Elias Matsas · George-Christopher Vosniakos

Received: 9 October 2014 / Accepted: 29 January 2015

© Springer-Verlag France 2015

Abstract This paper presents a highly interactive and1

immersive Virtual Reality Training System (VRTS)2

(“beWare of the Robot”) in terms of a serious game that sim-3

ulates in real-time the cooperation between industrial robotic4

manipulators and humans, executing simple manufacturing5

tasks. The scenario presented refers to collaborative handling6

in tape-laying for building aerospace composite parts. The7

tools, models and techniques developed and used to build8

the “beWare of the Robot” application are described. System9

setup and configuration are presented in detail, as well as user10

tracking and navigation issues. Special emphasis is given to11

the interaction techniques used to facilitate implementation12

of virtual human–robot (HR) collaboration. Safety issues,13

such as contacts and collisions are mainly tackled through14

“emergencies”, i.e. warning signals in terms of visual stim-15

uli and sound alarms. Mental safety is of utmost priority and16

the user is provided augmented situational awareness and17

enhanced perception of the robot’s motion due to immersion18

and real-time interaction offered by the VRTS as well as by19

special warning stimuli. The short-term goal of the research20

was to investigate users’ enhanced experience and behav-21

iour inside the virtual world while cooperating with the robot22

and positive pertinent preliminary findings are presented and23

briefly discussed. In the longer term, the system can be used24

to investigate acceptability of H–R collaboration and, ulti-25

mately, serve as a platform for programming collaborative26

H–R manufacturing cells.27

E. Matsas · G.-C. Vosniakos (B)

School of Mechanical Engineering, National Technical University

of Athens, Heroon Polytechniou 9, 15780 Zografou-Athens, Greece

e-mail: [email protected]

E. Matsas

e-mail: [email protected]

Keywords Virtual reality · Human–robot collaboration · 28

Safety · Manufacturing training · Interaction · Serious game 29

1 Introduction 30

In modern manufacturing systems an emerging need for 31

cooperation (or collaboration) and work-space sharing of 32

industrial robots and humans to execute manufacturing tasks 33

has arisen. The goal of this coexistence and collaboration is 34

to improve quality and productivity by enhancing workers’ 35

sensory-motor abilities, knowledge and dexterity in manually 36

performed tasks with the strength, accuracy and repeatabil- 37

ity of robots. In this case, rather than replacing the human’s 38

work with robots in repetitive and alienating tasks, collab- 39

orative assistance of robots can lead workers to perform 40

“value-added work” [1]. Robots are already used for teaming 41

up with, or assisting workers in assembly lines, especially 42

in automotive and aerospace industry. For instance a new 43

manipulator arm structure is presented in [2] consisting of a 44

mixture of SCARA manipulator and a parallelogram struc- 45

ture that allows the operator to move inside its workspace and 46

handle very heavy loads with an accuracy of few millimeters. 47

In [1], a mobile robotic assistant vis-à-vis human assistant is 48

evaluated in terms of performance for H–R collaboration in 49

fetch-and-deliver tasks. In the automotive industry a slow- 50

moving robot is reported to have been introduced, which 51

collaborates in proximity with a human worker to insulate 52

and water-seal vehicle doors [3]. Furthermore, a two-armed 53

collaborative manufacturing robot has been designed and 54

produced that automatically adapts to changes in its environ- 55

ment, such as obstacles, accidental contacts, dropped objects, 56

or variable conveyor speeds. It can work close to workers 57

without the need of safety cages, since it contains sensors 58

123

Journal: 12008 MS: 0259 TYPESET DISK LE CP Disp.:2015/2/2 Pages: 15 Layout: Large

Au

tho

r P

ro

of

unco

rrec

ted

pro

of


and Artificial Intelligence software that enable it to “see”59

objects, “feel” feedback forces and “understand” tasks [4].60

In a different application realm, it seems that collabora-61

tive robots are destined to provide help, offer personalized62

services and generally improve quality of life to the elderly,63

e.g. ASTRO [5] and the Giraff [6] robots. However, safety64

concerns keep human–robot (H–R) collaboration in industry65

still under-exploited, despite its seemingly offering a good66

balance between productivity, quality, flexibility and initial67

as well as running cost. For a long time now, safety of the68

human interacting with industrial robots is addressed by seg-69

regation between humans and robots in space and/or in time.70

The most critical safety aspects are the moving robot arm71

and the perceptibility of arm movements [7]. Modern safety72

techniques in H–R collaboration integrate infra-red light cur-73

tains, robotic vision with cameras, emergency-stop buttons74

and alarms, structured either in a stop-and-go fashion, or to75

slow down the robot movements. Yet, most of the above tech-76

niques usually stop the entire production when used in pro-77

duction lines and not in smaller cells. Collaborative robots are78

developing so quickly that international standards are having79

trouble to keep up. Although the International Organization80

for Standardization (ISO) has revised the ISO 10218 standard81

Part II, which allows cooperation of robot with personnel, it82

has yet to work out safety standards for collaborative robots,83

such as how much force a robot can safely apply to different84

parts of a human worker’s body [8,9].85

However, no matter how safe (physically) collaborative86

robots are, they may not be welcomed by human workers. To87

assure acceptability of such a collaboration between humans88

and robots, all important physical and “mental” safety issues89

that arise must be successfully dealt with. Physical safety90

means that the robot does not physically injure humans. Men-91

tal safety, on the other hand, can be defined as the enhanced92

users’ vigilance and awareness of the robot motion, that will93

not cause any unpleasantness such as fear, shock or surprise94

[9,10].95

Recent H–R interaction studies have shown that a fluent96

collaboration (beyond stop-and-go interaction type) requires97

awareness and anticipation of intention by both human and98

robotic agents [11]. Situation awareness can be defined as99

“the perception of the elements in the environment within100

a volume of time and space, the comprehension of their101

meaning and the projection of their status in the near future”102

[12]. To enable and facilitate this awareness and anticipation,103

it is important that both humans and robots communicate104

their status, location and intent, either explicitly through cer-105

tain cues such as audio/visual signals or implicitly via their106

actions and motion paths [1]. Serious games and highly inter-107

active and immersive Virtual, Augmented or Mixed Real-108

ity training applications are preferentially deployed in such109

cases, since they can provide enhanced training in all three110

levels of situation awareness (perception, comprehension and 111

projection). 112

In terms of functionality, Virtual Reality (VR) allows users 113

to be extracted from physical reality in order to virtually 114

change time, space, and (or) interaction type [13]. There- 115

fore, the three main features of VR are: Interaction, Immer- 116

sion and Imagination. Virtual Reality-based Training Sys- 117

tems (VRTSs) are advanced computer-assisted, interactive 118

training systems using VR technology, e.g. allowing trainees 119

to properly test and operate new equipment before it is actu- 120

ally installed [14]. VRTSs typically provide trainees impor- 121

tant perceptual cues and multi-modal feedback (e.g., visual, 122

auditory, and haptic), enabling effective transfer of virtu- 123

ally acquired knowledge to real-world operation skills. The 124

significance of VR in manufacturing training and in H–R 125

collaboration is widely pointed out in the literature, indica- 126

tively [14–18]. Furthermore, VRTSs need to have the neces- 127

sary physical fidelity to mimic the resolution of the physical 128

world in order to be effective for tasks that are characterized 129

by a significant perceptual and/or motor component [19]. 130

All VRTSs can be decomposed into three distinct functional 131

parts: (i) input devices for interaction, such as sensors, track- 132

ers, mice (ii) output devices (mainly for immersion, such as 133

Head Mounted Displays, large 3D displays and force feed- 134

back arms), and (iii) a VR engine including data and models 135

that constitute the virtual scene, interaction models, and a 136

graphical representation of the user (avatar) [20]. 137

This paper presents a highly interactive and immersive 138

VRTS (“beWare of the Robot”) in terms of a serious game 139

that simulates in real-time the cooperation between industrial 140

robotic manipulators and humans, executing simple manu- 141

facturing tasks. The scenario presented includes collabora- 142

tive handling in tape-laying for building aerospace compos- 143

ite parts. Safety issues, such as contacts and collisions are 144

mainly tackled through “emergencies”, i.e. warning signals 145

in terms of visual stimuli and sound alarms. Mental safety is 146

of upmost priority and the approach followed is that warning 147

stimuli inside a VRTS that offers immersion and real-time 148

interaction can provide to the user augmented situational 149

awareness and enhanced perception of the robot’s motion. 150

The short-term goal of the research is to investigate users’ 151

enhanced experience and behaviour inside the virtual world, 152

while cooperating with a robot, whilst the overall long-term 153

goal is to investigate the acceptability of H–R collaboration 154

and enhance the relevant terms by means of such an environ- 155

ment. 156

In Sect. 2, related work on VEs for H–R Collaboration is 157

presented. Sect. 3 provides an overview of the system and 158

details of the VE, whilst Sect. 4 explains implementation- 159

specifics relating to interaction techniques and the 3D user 160

interfaces for tracking and interaction. Preliminary findings 161

concerning user’s experience are presented and briefly dis- 162

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


cussed in Sect. 5. Finally, in Sect. 6, conclusions and future163

work are summarized.164

2 Related work165

In the literature, a deictic mode of sensory-motor control with166

symbolic gestures and a Virtual Research Head Mounted Dis-167

play (HMD) for teleoperation and teleassistance, was first168

proposed in [21], that can be useful for H–R interaction and169

control. Morioka and Sakakibara [22] developed a new cell170

production assembly system, in which physical and informa-171

tion supports are provided to the human operators to increase172

productivity. A human and a mobile robot cooperate in a173

shared workspace without stopping with the use of vision174

and speed restriction techniques. Krüger et al. [23] gives a175

survey about H–R cooperation in assembly, safety systems,176

and all available technologies that support the cooperation.177

Giesler et al. [24] presented an Augmented Reality (AR)-178

based approach wherein human and robot mostly share skills179

rather than physically collaborate. Corrales et al. [25] present180

the implementation of real-time proximity queries between181

humans and robotic manipulators, used by safety strategies182

/ algorithms for avoiding collisions. From another point of183

view, Arai et al. [26] deal with mental strains and stress of184

human operators in H–R cell production systems and pro-185

pose metrics (distance, speed) for a physiologically comfort-186

able collaboration, concluding that to reduce mental strains187

human operators need to be notified of robot motions before188

they happen. Charoenseange and Tonggoed [27] present an189

implementation of H–R collaboration in shared workspace190

for virtual assembly tasks, using AR techniques to provide191

necessary information to the human operator. An AR inter-192

face for interactive robot path planning is also shown in [28].193

Several H–R interaction methods and a Euclidean distance-194

based method are implemented, to assist the user in path195

planning, within the AR environment. In [17], a H–R collabo-196

ration experiment is conducted in an immersive Virtual Envi-197

ronment (CAVE), to evaluate the latter with respect to real-198

world applications. A markerless interface and a KinectTM-199

based interaction method that allows a human operator to200

communicate his movements to a dual robotic manipulator201

during an object manipulation task is presented in [29]. Wang202

et al. [30] report a novel, vision-guided method of real-time203

active collision avoidance for H–R collaboration in an aug-204

mented environment, where the system can alert the operator,205

stop or modify the robot’s trajectory away from an approach-206

ing operator. Qiu et al. [31] propose a real-time driving model207

for personalized Virtual Human modeling, and integrate the208

complete model into a VR environment for interactive assem-209

bly. Zaeh and Roesel [32] present a safety-based smart H–R210

system, which perceives its environment with multiple sen-211

sors and integrates a robot with reactive behavior and an oper-212

ator. Neuhoefer et al. [33] developed an immersive VR/AR 213

simulation system for H–R cooperation, with a virtual robot 214

and real-time physics simulation capabilities. Usability eval- 215

uation between the AR and the VR configuration showed 216

users’ preference towards the former. Weistroffer et al. [34] 217

presented an immersive VR simulating environment to study 218

how people perceive robots during collaboration tasks, as 219

well as the acceptability of H–R collaboration. To summa- 220

rize, a large number of authors have thoroughly tackled H–R 221

collaboration safety aspects, as well as tools and methods for 222

safe collaboration. Many authors have also studied VR or AR 223

environments for teleoperation and teleassistance. Very few 224

studies focus on mental safety issues and acceptance of H–R 225

collaboration using immersive and interactive VR systems. 226

3 VRTS overview 227

In this section, the tools, models and techniques exploited in 228

building the “beWare of the Robot” application are described. 229

System setup and configuration are presented in detail, as 230

well as user tracking and navigation issues. Special emphasis 231

is given to the interaction techniques used to facilitate virtual 232

H–R collaboration implementation. 233

3.1 Use case 234

A use case scenario was developed, representing H–R collab- 235

orative tape laying for building aerospace fabric reinforced 236

composite parts. Conventional tape laying typically involves 237

carbon fiber reinforcement in the form of profiled fabric lay- 238

ers (patches/cloths) being stacked manually by an operator 239

inside a die successively, one on top of the other, until the 240

desired thickness is reached. In the H–R collaborative sce- 241

nario, the robotic manipulator is assigned the picking and the 242

transfer of the patches towards the human. Once the user takes 243

the patch from the robot and places it in the right position 244

inside the metallic die located in front of him, the robot pro- 245

ceeds to feed the next patch. The process is repeated until all 246

different patches are properly placed in the die by the human, 247

hereby represented by the avatar’s hands. These direct H–R 248

collaborative tasks are kept as simple as possible, but they 249

involve close proximity between human and robot and they 250

are performed in parallel, as a “hand-to-hand” collabora- 251

tive manipulation scenario. The latter was selected to rep- 252

resent typical “hand-to-hand” H–R collaborative manufac- 253

turing scenarios, to ensure further exploitation in the future. 254

The robot arm is attached upside-down to a metallic struc- 255

ture next to the user and the workbench on which the die lies, 256

see Fig. 1. The robotic arm takes a carbon patch (cloth) from 257

a workbench on which the patches are stacked, and handles 258

it towards the user. When the robot moves, the user is capa- 259

ble of both virtually seeing and hearing it. In addition, the 260

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


Fig. 1 The shop-floor

environment and the main VE

components as seen from a third

person shooter inside Unity

3dTM

robot’s work envelope is always visible by the user as a semi-261

transparent, red, toroidal type surface. The latter is an exact262

representation of the real robot workspace, and its projection263

on the floor is also depicted in Fig. 1. When the user enters264

the robot’s workspace, the red surface is blinking and a sound265

alarm is turned on, to warn the user of the robot’s proximity266

and a possible collision. The use of these visual and audio267

warning stimuli might be argued to be imprecise or unrealis-268

tic and even to compromise physical fidelity, however such269

exaggerated triggers can prove very effective for tasks that270

are characterized by a significant perceptual component [19],271

and can provide the user enhanced situational awareness and272

alertness of a potentially hazardous workspace [35]. More-273

over, according to Bowman et al. [36], 3D user interfaces274

should deliver only as much realism as is needed for the par-275

ticular application.276

3.2 System setup and configuration277

“BeWare of the robot” VRTS is a standalone build devel-278

oped on the Unity 3dTM game engine platform that can279

run on a typical PC with WindowsTM operating system.280

The system platform consists of: the “beWare of the robot”281

application, a PC running Windows XPTM equipped with an282

nVidia QuadroTM FX1700 graphics card, an eMagin Z800283

3DVisorTM HMD with stereo ear buds, a Microsoft KinectTM284

sensor, keyboard and mouse. Furthermore, in order to be able285

to reproduce what the user sees through the HMD and to286

record the user with a video camera, a projector is normally287

used, cloning the displays of the HMD on a wall behind the288

user. KinectTM sensor and headtracker (input) data are trans-289

mitted to the PC over USB cables. Communication between290

Unity 3dTM and the KinectTM sensor is implemented with the291

OpenNI framework. Figure 2 shows a diagram of the system292

and data flow between its main components.293

Shop-floor environment and its components were devel- 294

oped using RhinocerosTM and 3ds MaxTM software, for 3D 295

part design, mesh creation and editing, and avatar’s biped 296

design. The skinned model of the avatar was created online 297

in the Evolver website [37]. 3D parts and models are exported 298

in “.fbx” file format for increased compatibility with Unity 299

3dTM software. Unity 3dTM game engine was used for assem- 300

bly, rendering, lighting, shading, physics and collision mod- 301

elling, simulation, programming and building the compila- 302

tion. 303

The VRTS incorporates (i) the assembly of original 3d 304

models forming the virtual model of a composites hand lay- 305

out work-cell, (ii) the model of a StäubliTM RX90L robotic 306

manipulator, (iii) the skinned model of an avatar with a biped 307

attached to it, (iv) real-time data exchange with the tracking 308

devices, (v) interaction scripts in C# and Unity’s JavaScript 309

mainly concerning collision and ray-casting, child/parenting 310

functions and skeletal tracking of avatar joints, (vi) real-time 311

shadows, rendering and lighting, and (vii) image, video and 312

audio textures from real working spaces. The virtual scene 313

of the shop-floor, its main components 3d models and the 314

avatar are depicted in Fig. 1. 315

3.3 The virtual scene—simulating environment 316

During execution and testing of the “beWare of the robot” 317

application, the user is asked to wear the HMD and to stand 318

straight at a distance of 2–3 m towards the KinectTM sensor. 319

The VE is rendered in real time in frame-sequential stereo- 320

scopic vision in the HMD, and a monoscopic duplicate of it 321

is shown typically through a wall projector behind the user. 322

When the system starts the user sees the initial screen of the 323

VE and a calibration prompt, see Fig. 3. 324

Calibration of the user against the avatar skeleton is 325

achieved by the user raising hands with both elbows bent at 326

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


Fig. 2 System setup

Fig. 3 The initial view of the VE as seen from the user (first person

shooter/egocentric view)

90◦ thereby standing in “Y” posture towards the Kinect sen-327

sor, see Fig. 4a. After calibrating, users are usually allowed328

some time to familiarise themselves with system’s tracking,329

navigating and travelling capabilities and to explore the dif-330

ferent objects of the VE as well as interaction with them.331

When the user moves or turns in real space, the avatar’s body332

and the virtual viewpoint change accordingly. When the user333

walks around in the real space, the avatar follows the same334

path in the VE, though with some spatial restrictions and lim-335

its, defined by the HMD cables length and the sensor tracking336

range. When the user turns his head, the avatar’s head and337

the first person camera attached to it respond accordingly.338

The user can also collide or interact with rigid bodies in the339

virtual scene, bend his body in every direction, move his340

hands and handle selected objects, see Fig. 5, noting that 341

the clone of the VE projected on the background wall was 342

for the benefit of the researchers. Stereoscopic vision offered 343

by the HMD, combined with body and head tracking capa- 344

bilities, provide the user an enhanced feeling of immersion 345

and involvement. Moreover, immersion and presence feel- 346

ings are enriched with 3d sound sources, i.e. robot motion 347

sound, alarm, shop-floor sounds, real-time shadows, and the 348

video of a real manufacturing work-cell which is reproduced 349

on the background wall, in form of a video texture, see Fig. 1. 350

Once immersed in the virtual scene and familiarized with 351

navigation and tracking, the user has to push a green button, 352

located on the workbench, to start the training, see Fig. 4b. 353

Apart from the green (start) button, a red, emergency stop but- 354

ton is also located on the workbench. When the user pushes 355

the green start button, a collision is detected between the cap- 356

sule colliders of the avatar hands and the sphere collider that 357

envelops the button. Colliders are invisible components of the 358

Unity physics library, defining the shape of an object for the 359

purpose of physical collisions, elaborated on further in Sect. 360

4.2 below. This collision makes the robotic arm start mov- 361

ing from its default position towards the workbench with the 362

carbon cloths (patches) stacked in the desired order. When 363

the robot’s end-effector approaches the first patch in the right 364

distance and orientation, their ray-casting colliders interact 365

with each-other and the end-effector becomes parent of the 366

cloth, i.e. the cloth is virtually attached to it and follows its 367

movements. The robot then moves again and hands the patch 368

over to the user (avatar) with the proper orientation. 369

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


Fig. 4 A storyboard visualising the different tasks of the use-case scenario

At the same time, the user has to approach the robotic370

arm, see Fig. 4c, preferably with only one arm extended, and371

take the patch from the end-effector, minding not to collide372

with the robot’s body. A ray-casting collider is attached to373

the avatar’s index fingers, allowing the user to take the patch374

from the robot, when the fingers touch the patch, see Fig. 4d.375

The avatar fingers are now parents of the patch. If the user376

enters the robot’s workspace when trying to grab the patch,377

the red surface representing the workspace starts flashing and378

an alarm sound is turned on, to warn the user to be aware of379

the robot’s proximity and of a potential hazard.380

Afterwards, the user has to move towards the workbench 381

where the metallic die lies and place the cloth on the high- 382

lighted (rendered with red color) surface of the die, see 383

Fig. 4e. The appropriate, different part of the die is high- 384

lighted, depending on the patch being handled, in order to 385

guide the user as to the position, boundary and orientation of 386

the patch. 387

The patch is not released from the avatar’s hand until it is 388

properly placed in the indicated area of the die. After that it 389

is released (destroyed) and appears laid in its final position 390

on the die, see Fig. 4f–n. Thereafter, the robotic arm moves 391

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


Fig. 5 Hands tracking example-avatar’s hands following the user’s arm movements (left), and avatar’s hand moving a patch and placing it onto

the red-coloured surface of the die (right)

again to the workbench with the carbon cloths, in order to392

fetch the next patch to the user. This process is repeated until393

all different patches are properly placed in the die (Fig. 4o)394

and there are no patches left on the workbench.395

The completion of the above tasks, including calibration,396

for four different patches normally takes about 6-8 minutes.397

A graphical representation of the system activities workflow398

is given in the UML activity diagram in Fig. 6. In the latter,399

task-sharing between the user, the tracker and the robot is400

presented in three vertical swim-lanes respectively.401

3.4 Avatar402

An avatar is very seldom used in H–R collaboration appli-403

cations reported in literature irrespective of how immersive404

or interactive the latter are. Most of them use either a fixed405

camera, or a simple first person shooter to abstractly simu-406

late the user’s vision and navigation. Aiming at a more fluent407

and realistic collaboration between the user and the robot, it408

was decided to involve a skinned and fully functional male409

avatar model. The inclusion of a realistic and functional vir-410

tual body may enhance the sense of presence, which in turn411

has a positive effect on spatial knowledge acquisition and412

usage [36,38]. Furthermore, Usoh et al. [38] showed the413

importance of user association with the virtual body, Thus,414

a highly detailed avatar was created consisting of 11,000415

polygons. In addition, when the user is looking down, or is416

simply facing his arms, he can see a realistic representation417

of his tracked virtual hands and arms. This is considered to418

be essential for the sense of presence in such a first-person419

shooter application, where the user is wearing a HMD and420

the real world is completely blocked from his view.421

The avatar shown in Fig. 7 was created as a textured mesh422

model, with a custom face (clone) and a body animation rig.423

A biped was attached to the avatar mesh model, making it424

kinematically functional, i.e. the different body parts mesh,425

the skin and cloth textures follow the biped’s kinematics. The 426

representation of the avatar object in Unity game engine has 427

a straightforward tree hierarchy, with the mesh model as a 428

parent and 3D Transforms ordered as children according to 429

the biped’s kinematic chain. 430

The main camera is attached to the avatar’s head as a child 431

of the head transform, at eye height and with ±60◦ field of 432

view about the local vertical axis. A first person shooter is 433

created in this way, offering to the user the perspective of 434

the avatar, and a view of his hands, if combined with head 435

and body tracking. The first person perspective was preferred 436

to the third person perspective because it arguably offers an 437

augmented presence, realism and immersion feeling; every 438

action taking place in the VE can be experienced by the user 439

as if performed by him/her. 440

A capsule collider bounds the avatar’s torso and two 441

smaller capsule colliders are attached to its hands. The col- 442

liders are used by the physics engine of Unity game engine 443

for collision detection and for event triggering as explained 444

in Sect. 3.3. Rigid body physics is also added in the avatar’s 445

hands, head and torso. The 11 avatar tracking points (Unity 446

Transforms) that are used for skeletal tracking of the user are 447

depicted in Fig. 7 on the right. 448

3.5 Robot 449

The industrial robot manipulator arm used in the H–R col- 450

laboration scenario is a StäubliTM RX 90L with six rotational 451

joints. It can lift objects with a maximum weight of 12 kg 452

and a maximum reach of 1185 mm. The virtual robot end 453

effector is a simplified vacuum based custom design for fast 454

handling of the lightweight carbon patches. 455

The virtual robot arm is suspended from a structure next to 456

the workbench, so that it can easily collaborate with the user, 457

as shown in Fig. 8. The robot object in Unity has a straightfor- 458

ward tree hierarchy, allowing proper robot motion according 459

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


Fig. 6 UML activity diagram for the system workflow

to robot open kinematic chain, starting from the base joint460

(J1) being just a parent and successively moving to the last461

one (J6) being just a child. In the VRTS, the robotic arm is462

controlled by forward kinematics scripts only, i.e. joint val-463

ues are only used to control the robot motion and determined464

position and orientation of the end-effector.465

A combination of mesh and primitive-shape colliders466

is used in the robot structure to increase execution speed;467

primitive-shape colliders such as box, sphere or capsule col-468

liders are faster in collision detection calculations, compared469

to complex mesh colliders. Capsule colliders are attached to470

the base, the arm and the forearm of the robot, and box collid-471

ers to the shoulder and the elbow. In the end-effector, though,472

where extra accuracy is needed to avoid false-collisions with473

the avatar’s hand, a mesh collider was preferred.474

The robot work envelope is depicted as a penetrable, semi- 475

transparent toroidal-like surface form, an exact copy of the 476

real robot workspace, with a mesh collider and the alarm 477

audio source attached to it. 478

4 Implementation 479

4.1 Unity VR engine 480

Unity is a platform for game creation including a game 481

engine and an Integrated Development Environment (IDE). 482

All the necessary resources (models, textures, sounds etc.) 483

are imported in a Unity project as assets, which are included 484

in GameObjects (classes) [39]. Components and scripts are 485

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


Fig. 7 The skinned avatar

inside the VE with the capsule

collider attached to its left hand

(middle), and, the 11 skeletal

tracking points in the avatar’s

biped (right)

Fig. 8 The robotic arm and the different colliders attached to it

then attached to GameObjects to control their behaviour.486

Components, or combination of components, usually add487

functionality to GameObjects, and they can be transforms,488

cameras, colliders, particles, rays, rigid bodies, audio sources489

etc.490

Scripts are essential elements of every VR application,491

and they are treated by the game engine as original com-492

ponents, written by the developer. Scripts can be used to493

respond to inputs, trigger events or wait for events to run,494

or even to create special physical or kinematical behaviours.495

Three programming languages are supported in Unity: (i) C#,496

(ii) UnityScript which is modelled after JavaScript, and, (iii)497

Boo, a language similar to Python. All original scripts devel-498

oped for the “beWare of the Robot” application are written499

in C#. Two functions are defined inside every class in Unity500

scripts: (i) the Start function, which is called once by Unity,501

before the play mode begins, and where all properties are set502

to their initial values, and, (ii) the Update function, which 503

is handled during the frame update in play mode, and which 504

usually includes movements, actions triggering, responses to 505

user actions, and whatever needs to be handled over time. 506

A very useful Unity asset type is the Prefab. Prefabs are 507

instances of a GameObject, cloning the complete original one 508

with its components and its properties. Any changes made 509

to a prefab are immediately applied to all its instances. The 510

advantage of Prefabs is that they are re-usable and they can be 511

easily instantiated or destroyed throughout the runtime, with 512

a few lines of code. In the scenario implemented composite 513

patches creation, selection and placement are implemented 514

with prefabs. 515

Transforms are the essential kinematic components in 516

Unity; transforms are the equivalent components to 3D 517

frames of other VR engines. Every GameObject has a trans- 518

form attached to it, determining its position, rotation and 519

scaling, relatively to the transform’s parent. 520

4.2 Interaction techniques 521

In VR, an interaction technique is a means or a method 522

including both hardware (I/O devices) and software com- 523

ponents for accomplishing the desired interaction task, via a 524

user interface (e.g. manipulating, moving, selecting etc) [36]. 525

The most preferable, successful and tried-and-true interac- 526

tion techniques attempt to simulate, or adapt from, the real, 527

physical world, referring to real behavioural schemes. If these 528

schemes are difficult to implement, 3D interaction metaphors 529

can prove very effective [13]. Such metaphors or a combi- 530

nation of them, form the fundamental mental model of a 531

technique, a symbolic image and a perceptual manifestation 532

of what users can do by using these techniques [36]. 533

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


The typical generic interaction tasks in VEs are the fol-534

lowing: (i) selection, (ii) manipulation, (iii) navigation (travel535

and way-finding), and, (iv) system control [36]. All of the536

above tasks are performed during the runtime of ‘BeWare537

of the Robot’ application, and they are accomplished with538

the use of a wide variety of natural interaction techniques539

(schemes). No metaphors or “magic” (i.e. overcoming human540

or real world limitations) techniques are employed. Many of541

the techniques used for some tasks are used in combina-542

tion with interaction techniques for other tasks not directly543

related to them (cross-task techniques), e.g. the ray-casting544

technique is used for selection and for system control, or, the545

direct virtual hand technique is used for manipulation and546

activation of controls.547

Ray-casting and collision-based pointing techniques are548

used for selection, and direct-virtual-hand, real hand ges-549

tures, collision, and child/parenting techniques are used for550

manipulation. a A ray casting script is attached to the avatar’s551

index fingers and the robot end-effector, which casts a small,552

invisible ray in a specific direction against all colliders, and553

returns a true Boolean value when the object with the desired554

tag is hit. If this value is true during H–R collaboration, the555

cloth is attached to the avatar’s hand, i.e. it becomes a child556

of the hand’s transform. According to the parenting tech-557

nique, the child inherits the movement and rotation of its558

parent. Although there are two interfering rays hitting the559

patch (when the user tries to grab the patch from the robot),560

the user is favoured because the ray cast from the avatar’s561

fingers is longer than the ray cast from the end-effector. The562

user then has to properly position and lay the cloth on the red-563

coloured area of the die. This task is accomplished in the VE564

with a combination of interaction techniques. First, the box565

collider of the patch collides with the red-coloured part of566

the die and a trigger event is sent destroying the patch object;567

the patch thus appears released from the avatar’s hand. At the568

same time, a prefab of the laid patch, which was not visible569

before, is instantiated and rendered, appearing as if it were570

automatically placed, laid and shaped in its final position of571

the die.572

Collision detection and triggering is the main interaction573

technique, and it is mostly implemented using sphere, box574

and cylinder covering techniques. The use of primitive shape575

colliders (or combination of them) is preferred to the use of576

complex mesh colliders, for increased execution speed and577

for maintaining a steady frame rate. During typical tests run-578

time, a steady frame rate, greater than 25 frames per second is579

obtained, which balances affordability and quality. Collision580

detection can be used either for solid object collision behav-581

iour, and thus, initiate rigid body physics actions (onCollisio-582

nEnter function), or just as a trigger event (onTriggerEnter583

function) that allows other colliders to pass through, and sim-584

ply triggers an event or calls another function. For example,585

when a game object tagged as “avatar” collides with the trig-586

Fig. 9 The visual alarm is enabled (flashing), warning the user that

he/she has entered the robot workspace

ger mesh collider of the robot workspace, the OnTriggerEnter 587

function enables the audio alarm to start playing, and makes 588

the renderer of the robot workspace start flashing, see Fig. 9. 589

Navigation and travel are implemented with simple phys- 590

ical motions and (real) walking techniques, since head and 591

body tracking is used. Navigation techniques based on track- 592

ing and walking are natural and instinctive, allowing there- 593

fore the user to carry out other, more important tasks without 594

being distracted even by thinking about how to move around. 595

Another advantage is that the hands are not used at all for 596

navigation, thus being entirely free to be devoted to the pur- 597

poses for which they are used in reality. In addition, walking 598

technique provides to the user vestibular cues that help him 599

understand the size of the environment [36]. However, walk- 600

ing is not always feasible due to real space limitations; in the 601

current case real walking is nevertheless an appropriate and 602

effective technique, as the size of the active working envi- 603

ronment is equal to the range of the Kinect tracker, and the 604

cabling length of the I/O devices allows freedom of move- 605

ment in the tracked area. 606

The system control task is crucial since it allows the user to 607

control the interaction flow between other tasks of the appli- 608

cation; a command is issued to request the system to perform 609

a function, or to change the system state. In this case, neither 610

Graphical User Interfaces (GUIs: buttons, menus etc), nor 611

physical tools are used for system control. The main tech- 612

nique used is the tracking-driven virtual hand that interacts 613

(selects, manipulates) with specific objects, and that in turn 614

triggers an event or a change in the system state/interaction 615

mode. For example, when the user’s virtual hand interacts 616

with the green button, e.g. the user walks to, points with his 617

hand to, approaches and finally touches/collides with it, a 618

function is triggered that moves the robotic arm and makes 619

it fetch the first patch. Similarly, patch selection, grabbing, 620

placement and release, call a function and change the system 621

state or request it to perform an action (parenting, render, 622

change colour etc). However, some conventional system con- 623

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


trol techniques are also used, such as the postural commands624

technique during the calibration process. Finally, mouse and625

keyboard are still used, but for minor system control tasks,626

such as the exit-on-escape-button function, which terminates627

the application, and parameter value typing in the settings628

window at the beginning of the application.629

4.3 3D user interfaces for interaction and tracking630

3D User Interfaces (UIs) involve 3D interaction, in which631

user tasks are performed directly in a 3D spatial context [36].632

In “beWare of the Robot” application, two main 3D UIs are633

employed: an eMagin Z800 3DVisor HMD and a Microsoft634

KinectTM sensor.635

The HMD is used as an output device for visual display,636

stereoscopic vision and 3D sound, and as an input device for637

head motion tracking. The Kinect sensor is used as an input638

device for skeletal tracking and positioning of the user.639

The Z800 HMD uses two OLED microdisplays to accom-640

plish the stereoscopic effect, when a frame-sequential 3D sig-641

nal is transmitted by the graphics card. Although the Z800642

HMD has three integrated gyroscopes and accelerometers to643

detect head motion along the X, Y and Z axis, HMD head644

tracking is limited to the head’s pitch angle, i.e. head turn-645

ing up or down (vertical head tracking) in the range ±60◦.646

Data from the other two gyroscopes are not exploited, since647

a tracking conflict arose between the tracking data of the648

Kinect on the one hand, and the data of the HMD gyroscope649

on the other hand. More specifically, the conflict was caused650

by the simultaneous horizontal tracking of the head (yaw651

rotation angle) by the HMD, and of the horizontal skeletal652

tracking of the shoulders and the neck by the Kinect sen-653

sor; for example, the avatar (camera) exhibits intensely and654

irregular trembling when the shoulders turn in one direction655

and the head turns in the opposite direction, thus making the656

immersion experience unbearable. This is the reason for not657

using the Kinect tracking data of the upper body (head and658

neck joints).659

The HMD headtracker, which comes bundled with a660

mouse-emulating software, translates the detected motion661

into smooth and accurate mouse motion on the screen. Spe-662

cialized settings in the bundled software and the character663

controller settings in Unity are also used to fine-tune mouse664

emulation. In Unity VR engine, the main camera is attached665

to the head transform, which is child of the character con-666

troller and the avatar. A “Mouse Look” script is attached to667

the main camera, mapping the mouse translational move to668

rotational move of the camera.669

3D skeletal tracking and positioning using Microsoft670

KinectTM is conducted by continuous processing of RGB671

and depth images of the user [29], captured by the sensor672

that is fixed in front of him. The sensor contains two cam-673

eras (one infrared and one standard video camera) and an674

infrared projector that produces a grid of dots (structured IR 675

light at 830 nm) that measure the distance of objects from the 676

Kinect, and compose a “depth map” of the image [40]. The 677

output images are downsampled and transmitted to the host 678

device via the USB cable, at a frame rate of 9–30 Hz. Kinect 679

software takes the computed depth map and infers body posi- 680

tion using machine learning and “known skeletons”, thereby 681

transforming the body parts image into a skeleton. 682

The OpenNI framework was chosen as the middleware for 683

Kinect, both for skeletal tracking, as well as a basic driver. 684

Tracking data of twenty body joints were available. How- 685

ever, data corresponding to just eleven of the joints were 686

sufficient and were therefore matched to the correspondent 687

biped control points, see Fig. 7. In particular, head, neck and 688

lower leg tracking, including feet, is not carried out. As men- 689

tioned above, head tracking is performed by the HMD. In 690

addition, the absence of neck, leg and feet tracking by the 691

KinectTM sensor does not affect the overall tracking qual- 692

ity or the immersion feeling in such a first person shooter 693

application, since the overall response of the biped to the 694

movements of the tracked joints was accurate enough. On 695

the contrary, it was noticed that lack of feet tracking made 696

the avatar’s movement smoother and steadier in the vertical 697

direction. When the user looks down he/she could see his/her 698

tracked virtual hands and arms, and probably the untracked 699

feet. However, given the fact that the feet are always oriented 700

in the hips direction, feet tracking absence is not noticeable 701

and was not described by any user as a distraction. 702

As far as Kinect’sTM tracking limitations and problems 703

are concerned, observations in [35] revealed that almost half 704

of the users sometimes “lost control” of their hands, even 705

instantaneously. That is because experiment tasks required 706

user body rotation towards the sensor, and at the extremes 707

hands tracking is confusing: KinectTM sensor cannot easily 708

distinguish the left from the right hand. In addition when 709

users turn their body more than 90◦ to the Kinect sensor 710

axis, skeletal tracking is sometimes lost, and avatar remains 711

“frozen” for a while in the last detected posture, although the 712

user may continue moving. 713

5 User experience and discussion 714

In interactive VEs, the human user becomes a real partici- 715

pant in the virtual world, interacting and manipulating virtual 716

objects. Human performance in training VEs highly depends 717

on the degree of presence that the user experiences, i.e. the 718

subjective experience of being in one place or environment 719

even when one is situated in another. Indeed, presence is the 720

most important measure for situation awareness in an inter- 721

active VE [41]. Prothero et al. [42] claim that “presence” 722

and “situation awareness” are overlapping constructs. It is 723

generally agreed that the two necessary conditions to expe- 724

rience presence are immersion and involvement. This work 725

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


Table 1 Test results (SD: strongly disagree, D: strongly agree, N: neutral, A: agree, SA: strongly agree)

Question SD D N A SA

I feel like I was really moving in the scene 0 1 3 5 21

Wearing the HMD I feel present and involved in the virtual activity 2 1 5 15 7

I lost my concentration, even instantaneously, during the experiment 28 0 0 2 0

Wearing the HMD I feel immerged in the virtual environment 1 0 7 13 9

I feel like I was really moving an object with my hand, despite the fact that

the object did not have physical mass

1 7 8 12 2

I did not easily perceive the red transparent surface that represented the robot

workspace

29 0 0 0 1

Audiovisual stimuli (alarms) helped me being aware of a potentially

hazardous workspace

0 1 1 8 20

I did not encounter any difficulties during the initial calibration process with

the Kinect sensor

0 3 3 6 18

I feel like the avatar followed my body and head movements precisely 1 2 12 14 1

It was easy to take the patches form the robot’s end-effector 0 0 9 13 8

It was easy to navigate/move in the virtual world (ease of movement, restraint) 0 3 12 12 3

I feel like my behavior in the virtual world didn’t change, compared to my

behavior in the real world

3 10 9 8 0

I feel more like I was participating in an amusing game 0 2 5 10 13

After the experiment I went through, I believe that H–R training tasks can be

more attractive with the use of “serious games”

1 0 2 8 19

focuses more on “presence by involvement” [43], i.e. both726

involvement and immersion are thought to be necessary for727

experiencing presence.728

BeWare of the Robot was tested by a group of 30 pre-729

final year mechanical engineering students and the detailed730

findings are published in [35]. A summary of the main results731

is shown in Table 1.732

The results revealed that the system scored great in “pres-733

ence by involvement” and in navigation issues; a vast major-734

ity of subjects were feeling as if they were really moving in735

the scene, “involved and present” in the virtual activity, with-736

out losing their concentration at all during the test. It was also737

found that participants felt immerged in the VE when wear-738

ing the HMD and a large number of them felt as if they were739

really manipulating an object with their hands, despite the740

fact that the object did not have physical mass.741

During the H–R collaboration procedure almost all sub-742

jects answered that they easily perceived the red transparent743

surface that represented the robot’s workspace, and 70 % of744

subjects managed to complete the H–R collaborative tasks745

without entering into the workspace (which was the poten-746

tially hazardous area). Most of the subjects (93 %) favourably747

accepted the use of visual and auditory stimuli (alarms) for748

the robot’s workspace awareness. Note that, as already men-749

tioned in Sect. 3.1, the employment of the red surface (robot750

workspace) and of the visual and audio warning stimuli might751

seem contradictory with fidelity and realism of VEs, but752

realism and level-of-detail are not always the ultimate goal.753

In VRTSs learning procedure for example, exaggeration or 754

deformation of real events is “authorized” for understanding 755

complex situation [44], for enhancing situation awareness 756

[36], and for alerting in a potentially hazardous workspace 757

[35]. 758

Concerning usability and tracking quality, very few sub- 759

jects reported encountering any difficulties during the initial 760

detection and calibration process by the Kinect sensor. Most 761

of the participants considered that the avatar was following 762

their (tracked) movements precisely. On the other hand, 43 % 763

of the subjects felt that their movements were altered com- 764

pared with the real ones, and 23 % of them pointed out a slight 765

“vibration” in the avatar’s hands movements. Nevertheless, 766

70 % of the subjects answered that they easily managed to 767

pick-up the cloth patches from the robot’s end-effector, while 768

the majority said that it was easy to navigate (travel) in the 769

virtual world. 770

According to [38] immersion requires that there is a 771

match between the user’s proprioceptive feedback about 772

body movements and the information generated on the dis- 773

plays of the HMD. This is obviously favoured by a detailed 774

body mapping. In this work, where tracking is implemented 775

seamlessly, contactlessly, and without time delay, the consis- 776

tency between proprioceptive information and sensory feed- 777

back is high. 778

As to user motivation, 76 % of the participants replied that 779

during the experiment they were feeling more as if they were 780

participating in an amusing game, and 90 % of the subjects 781

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


believe that training tasks requiring H–R collaboration can be782

more attractive with the use of interactive “serious games”.783

Eventually, the results suggest a positive prospect for the784

use of VR for training on H–R collaboration. Users experi-785

ence a high level of involvement and immersion, and there-786

fore, presence. The inclusion of a realistic avatar, that is the787

user’s own kinematic representation, strengthens the sense788

of presence. Stereoscopic vision combined with the body789

and head tracking capabilities provide the user an enhanced790

immersion feeling and have a positive effect on spatial knowl-791

edge. Moreover, first-person perspective was preferred to792

the third-person perspective, because it offers an augmented793

presence feeling; every action taking place in the VE can be794

experienced by the user as if performed by him/her. In addi-795

tion, moving around the scene and using real gestures (skele-796

tal tracking) to manipulate objects and to complete the tasks797

makes learning more active and impressive for learners than798

conventional, non-interactive training methods. Ultimately,799

the more control the user has in interacting with the VE and800

over the system, the higher his/her experience of presence801

and awareness, and the greater the success of the VRTS.802

6 Conclusion and future work803

A novel interactive VRTS named “beWare of the robot” has804

been presented, in terms of a serious game for H–R collabora-805

tion. This short-term aim of the research was to construct this806

system as a platform for experimentation. Thus, this paper807

presented an extended overview of the system, functional808

details of the Virtual Environment, interaction techniques and809

tools used, as well as 3D user interfaces for interaction, and810

especially for tracking.811

The medium-term aim of the research concerns the accept-812

ability of H–R collaboration, in regard to safety issues that813

arise during the execution of collaborative manufacturing814

tasks in a shared workspace. To establish the acceptability of815

the collaboration, mental safety issues, i.e. human’s aware-816

ness and vigilance of the moving robot, have to be tack-817

led. The line adopted in this work is that immersive serious818

games simulating H–R collaboration are the most pertinent819

tool to facilitate situation awareness, since they can provide820

enhanced training at all three levels of situation awareness821

(perception, comprehension and projection).822

H–R collaborative tape-laying for composite parts was823

implemented as a relatively simple use case, in which the824

virtual world includes a shop floor environment, an industrial825

robot and an avatar, a metallic die, carbon fibre patches, and826

several auxiliary machine shop objects. Both the VE model827

and the training scenario were designed to have the necessary828

fidelity and to be as close as possible to reality.829

What distinguishes this work is primarily the wide “blend-830

ing” of different interaction techniques that were used. More831

than 25 original scripts were deployed to accomplish all kinds 832

of interaction tasks: selection, manipulation, navigation, and, 833

system control. The main interaction techniques that were 834

used are: ray-casting, collision detection, child/parenting, 835

tracking-driven virtual hand, and, real walking technique. 836

Using markerless body and head tracking, human-system 837

interaction is natural, replicating real and inherent move- 838

ments. A second novelty of this work is the simultaneous 839

employment of the KinectTM sensor together with the HMD, 840

for skeletal and head tracking respectively. The HMD is 841

used as an output device for 3D visual display and as an 842

input device for head tracking. Head-tracking quality and 843

presence feeling are therefore higher, because by using one 844

device for both activities, the data generated on the output 845

displays correspond precisely to the input tracking data, and 846

the match between proprioception and sensory feedback is 847

considerable. A further novelty is the inclusion of a high qual- 848

ity avatar with the first person shooter, allowing the user to 849

observe his/her tracked hands, and to experience any activ- 850

ity as if it was really performed by him. This first-person 851

enhanced interactive experience is difficult, or even danger- 852

ous to be naturally performed in the real world involving a 853

real robot. Consequently, the VRTS can be seen as a tool to 854

obtain knowledge, and transferring (returning) this knowl- 855

edge back to the real world. 856

Overall, testing results are positive and suggest a good 857

prospect in the use of such applications for H–R collabora- 858

tion simulation or acceptability testing. It would be difficult 859

to claim, of course, that results can be generically applicable 860

to all aspects of H–R collaboration; the findings however sug- 861

gest that the immersive experience combined with warning 862

stimuli, can help the user to experience the feel of presence 863

as a situation awareness phenomenon, which is, eventually, 864

the intent of the research. 865

The long term-aim of the research is to consider task-based 866

or even safety-based robot motion programming according 867

to situation awareness and anticipation of intention by both 868

human and robot. For instance, the human could wait for 869

the robot to finish its task, move away or move with cau- 870

tion, or finally perform an alternative task. Conversely, the 871

robot would change its posture, work on another possible 872

task (patch), slow down or move with priority to a specific 873

joint. 874

Concerning tracking quality and experience newer I/O 875

devices would be considered, such as the KinectTM for Xbox 876

One sensor for tracking, the Leap MotionTM device for pre- 877

cise hand/finger tracking as well as immersion devices offer- 878

ing wider field of view and extensive headtracking (e.g. the 879

Oculus RiftTM). 880

Acknowledgments This research has been co-financed by the Euro- 881

pean Union (European Social Fund—ESF) and Greek national funds 882

through the Operational Program “Education and Lifelong Learning” of 883

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


the National Strategic Reference Framework (NSRF)—Research Fund-884

ing Program: Heracleitus II. Investing in knowledge society through the885

European Social Fund. The authors would also like to thank Dimitrios886

Batras, MSc, for his generous help in programming, his motivation and887

his helpful comments during the development of the application.888

References889

1. Unhelkar, V.V., Siu, H.C., Shah, J.A.: Comparative performance890

of human and mobile robotic assistants in collaborative fetch-891

and-deliver tasks. In: Proc. 2014 ACM/IEEE Int. Conf. Human–892

robot Interact—HRI ’14, pp. 82–89 (2014). doi:10.1145/2559636.893

2559655894

2. de Santos, P.G., Garcia, E., Sarria, J., Ponticelli, R., Reviejo, J.: A895

new manipulator structure for power-assist devices. Ind. Robot. An.896

Int. J. 37(5), 452–458 (2010). doi:10.1108/01439911011063272897

3. Knight, W.: Smart Robots Can Now Work Right Next to Auto898

Workers. http://goo.gl/EUbRNX (2013). Accessed 22 April 2014899

4. Rethink robotics: baxter robot overview. http://goo.gl/fRUqou900

(2014). Accessed 22 April 2014901

5. Coradeschi, S., Cesta, A., Cortellessa, G., Coraci, L., Gonzalez, J.,902

Karlsson, L., Furfari, F., Loutfi, A., Orlandini, A., Palumbo, F., Pec-903

ora, F., von Rump, S., Stimec, A., Ullberg, J., Otslund, B.: Giraff-904

Plus: combining social interaction and long term monitoring for905

promoting independent living, 2013. In: 6th Int. Conf. Hum. Syst.906

Interact, pp. 578–585 (2013). doi:10.1109/HSI.2013.6577883907

6. Cavallo, F., Aquilano, M., Bonaccorsi, M., Mannari, I., Carrozza,908

M.C., Dario, P.: Multidisciplinary approach for developing a new909

robotic system for domiciliary assistance to elderly people. In:910

Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol.911

Soc. Annu. Conf., pp. 5327–5330 (2011). doi:10.1109/HSI.2013.912

6577883913

7. Helander, M.G.: Ergonomics and safety considerations in the914

design of robotics workplaces: a review and some priorities for915

research. Int. J. Ind. Ergon. 6(2), 127–149 (1990). doi:10.1016/916

0169-8141(90)90018-W917

8. Working with robots: our friends electric. http://goo.gl/v8DhAK918

(2013). Accessed 22 April 2014919

9. Ogorodnikova, O.: Human robot interaction? The safety challenge920

(an integrated frame work for human safety). PhD dissertation,921

Budapest University of Technology and Economics, Faculty of922

Mechanical Engineering, Budapest (2010)923

10. Alami, R., Bicchi, A., Bischoff, R., Chatila, R., De Luca, A., De924

Santis, A.: Safe and dependable physical human–robot interaction925

in anthropic domains? State of the art and challenges. In: Bicchi,926

A., De Luca, A. (ed.) Procceedings IROS Workshop on pHRI—927

Physical Human–Robot Interaction in Anthropic Domains (2006)928

11. Hoffman, G., Breazeal, C.: Effects of anticipatory action on human-929

robot teamwork efficiency, fluency, and perception of team. Pro-930

ceeding ACM/IEEE Int. Conf. Human-robot Interact. - HRI ’07.931

1–8 (2007). doi:10.1145/1228716.1228718932

12. Wickens, C.D.: Situation awareness: review of Mica Ends-933

ley’s 1995 articles on situation awareness theory and mea-934

surement. Hum. Factors. 50(3), 397–403 (2008). doi:10.1518/935

001872008X288420936

13. Fuchs, P., Moreau, G., Moreau, G., Burkhardt, J.-M., Coquillart, S.:937

Le traité de la réalité virtuelle, vol. 2. L’interfaçage, l’immersion et938

l’interaction en environnement virtuel. Les Presses de l’Ecole des939

Mines de Paris, Paris (2006)940

14. Lin, F., Ye, L., Duffy, V., Su, C.-J.: Developing virtual environments941

for industrial training. Inf. Sci. (Ny) 140(1—-2), 153–170 (2002).942

doi:10.1016/S0020-0255(01)00185-2943

15. Shukla, C., Vazquez, M., Frank Chen, F.: Virtual manufacturing: an 944

overview. Comput. Ind. Eng. 31(1–2), 79–82 (1996). doi:10.1016/ 945

0360-8352(96)00083-6 946

16. Burdea, G., Coiffet, P.: Virtual Reality Technology, 2nd edn. Wiley- 947

Interscience, New York (2003) 948

17. Duguleana, M., Barbuceanu, F.G., Mogan, G.: Virtual and mixed 949

reality—new trends. In: Shumaker, R. (ed.) Virtual and mixed 950

reality—new trends. Springer, Berlin, pp. 164–173 (2011). doi:10. 951

1007/978-3-642-22021-0_19 952

18. Zhu, J., Ong, S.K., Nee, A.Y.C.: A context-aware augmented reality 953

system to assist the maintenance operators. Int. J. Interact. Des. 954

Manuf. (2014). doi:10.1007/s12008-013-0199-7 955

19. Nathanael, D., Vosniakos, G.-C., Mosialos, S.: Cognitive task 956

analysis for virtual reality training. In: Proceedings of the 28th 957

Annual European Conference on Cognitive Ergonomics—ECCE 958

’10. ACM Press, New York, p. 241 (2010). doi:10.1145/1962300. 959

1962350 960

20. Burkhardt, J.-M., Bardy, B., Lourdeaux, D.: Immersion, Réalisme 961

et Présence dans la conception et l ’ évaluation des Environnements 962

Virtuels. Psychol. Française. 48(2), 35–42 (2003) 963

21. Pook, P.K., Ballard, D.H.: Deictic human/robot interac- 964

tion. Rob. Auton. Syst. 18, 259–269 (1996). doi:10.1016/ 965

0921-8890(95)00080-1 966

22. Morioka, M., Sakakibara, S.: A new cell production assembly sys- 967

tem with human–robot cooperation. CIRP Ann. Manuf. Technol. 968

59, 9–12 (2010). doi:10.1016/j.cirp.2010.03.044 969

23. Krüger, J., Lien, T.K., Verl, A.: Cooperation of human and 970

machines in assembly lines. CIRP Ann. Manuf. Technol. 58, 628– 971

646 (2009). doi:10.1016/j.cirp.2009.09.009 972

24. Giesler, B., Steinhaus, P., Walther, M., Dillmann, R.: Sharing skills: 973

using augmented reality for human–robot collaboration. In: Woods, 974

A.J., Merritt, J.O., Benton, S.A., Bolas, M.T. (eds.) Proc. SPIE 975

5291, Stereoscopic Displays and Virtual Reality Systems XI, pp. 976

446–453 (2004). doi:10.1117/12.526689 977

25. Corrales, Ja, Candelas, Fa, Torres, F.: Safe human–robot inter- 978

action based on dynamic sphere-swept line bounding volumes. 979

Robot. Comput. Integr. Manuf. 27(1), 177–185 (2011). doi:10. 980

1016/j.rcim.2010.07.005 981

26. Arai, T., Kato, R., Fujita, M.: Assessment of operator stress induced 982

by robot collaboration in assembly. CIRP Ann. Manuf. Technol. 983

59(1), 5–8 (2010). doi:10.1016/j.cirp.2010.03.043 984

27. Charoenseang, S., Tonggoed, T.: Human–robot collaboration with 985

augmented reality. In: Stephanidis, C. (ed.) HCI International 986

2011—Posters’ Extended Abstracts. Springer, Berlin, pp. 93–97 987

(2011). doi:10.1007/978-3-642-22095-1_19 988

28. Fang, H.C., Ong, S.K., Nee, A.Y.C.: A novel augmented reality- 989

based interface for robot path planning. Int. J. Interact. Des. Manuf. 990

8(1), 33–42 (2014). doi:10.1007/s12008-013-0191-2 991

29. Du, G., Zhang, P.: Markerless human–robot interface for dual robot 992

manipulators using Kinect sensor. Robot. Comput. Integr. Manuf. 993

30(2), 150–159 (2014). doi:10.1016/j.rcim.2013.09.003 994

30. Wang, L., Schmidt, B., Nee, A.Y.C.: Vision-guided active collision 995

avoidance for human–robot collaborations. Manuf. Lett. 1(1), 5–8 996

(2013). doi:10.1016/j.mfglet.2013.08.001 997

31. Qiu, S., Fan, X., Wu, D., He, Q., Zhou, D.: Virtual human modeling 998

for interactive assembly and disassembly operation in virtual reality 999

environment. Int. J. Adv. Manuf. Technol. 69(9–12), 2355–2372 1000

(2013). doi:10.1007/s00170-013-5207-3 1001

32. Zaeh, M., Roesel, W.: Safety aspects in a human–robot interaction 1002

scenario: a human worker is co-operating with an industrial robot. 1003

In: Kim, J.-H., et al. (eds.) Progress in robotics. Springer, Berlin, 1004

pp. 53–62 (2009). doi:10.1007/978-3-642-03986-7_7 1005

33. Neuhoefer, J.A., Bernhard, K., Schlick, C.M.: Embedded aug- 1006

mented reality training system for dynamic human–robot coop- 1007

eration. NATO RTO-MP-HFM-169. RTO Human Factors and 1008

123


Au

tho

r P

ro

of

unco

rrec

ted

pro

of


Medicine Panel (HFM) Workshop, RDECOM, pp. 1–14, Orlando,1009

Florida, USA (2009)1010

34. Weistroffer, V., Paljic, A., Callebert, L., Fuchs, P.: A methodology1011

to assess the acceptability of human-robot collaboration using vir-1012

tual reality. In: Proceedings of the 19th ACM Symposium on Virtual1013

Reality Software and Technology—VRST ’13. ACM Press, New1014

York, p. 39 (2013). doi:10.1145/2503713.25037261015

35. Matsas, E., Batras, D., Vosniakos, G.: Beware of the robot: a1016

highly interactive and immersive virtual reality training applica-1017

tion in robotic manufacturing systems. In: Emmanouilidis, C.,1018

Taisch, M., Kiritsis, D. (eds.) Advances in Production Manage-1019

ment Systems. Competitive Manufacturing for Innovative Products1020

and Services. Springer, Berlin, pp. 606–613 (2013). doi:10.1007/1021

978-3-642-40352-1_761022

36. Bowman, D.A., Kruijff, E., LaViola, J.J., Poupyrev, I.: 3D User1023

Interfaces: Theory and Practice. Addison Wesley Longman Pub-1024

lishing Co., Inc, Redwood City (2004)1025

37. Evolver 3D digital avatar website. www.evolver.com. Accessed 231026

January 20121027

38. Usoh, M., Arthur, K., Whitton, M.C., Bastos, R., Steed, A., Slater,1028

M., Brooks, F.P.: Walking, walking-in-place, flying, in virtual1029

environments. In: Proc. 26th Annu. Conf. Comput. Graph. Inter-1030

act. Tech.—SIGGRAPH ’99, pp. 359–364 (1999). doi:10.1145/1031

311535.3115891032

39. Unity online manual. http://goo.gl/kH1Hv8. Accessed 25 July 1033

2014 1034

40. Miles, R.: Start here! Learn microsoft kinect API. O’Reilly Media 1035

Inc, Sebastopol (2012) 1036

41. Vora, J., Nair, S., Gramopadhye, A.K., Duchowski, A.T., Melloy, 1037

B.J., Kanki, B.: Using virtual reality technology for aircraft visual 1038

inspection training: presence and comparison studies. Appl. Ergon. 1039

33(6), 559–70 (2002). doi:10.1145/311535.311589 1040

42. Prothero, J.D., Parker, D.E., Furness III, T.A., Wells, M.J.: Towards 1041

a robust, quantitative measure for presence. In: Garland, D.J., End- 1042

sley, M.R. (eds.) Conference on Experimental Analysis and Mea- 1043

surement of Situational Awareness. Embry-Riddle Aeronautical 1044

University Press, Daytona Beach, pp. 359–366 (1995) 1045

43. Schuemie, M.J., van der Straaten, P., Krijn, M., van der Mast, C.A.: 1046

Research on presence in virtual reality: a survey. Cyberpsychol. 1047

Behav. 4(2), 183–201 (2001). doi:10.1089/109493101300117884 1048

44. Burkhardt, J.-M., Lourdeaux, D., Fuchs, P.: Conception d’un sys- 1049

tème de RV pour la formation des agents de conduite aux opérations 1050

en milieu ferroviaire. In: Richards, A., Grumbach, N. (eds.) Col- 1051

loque Réalité Virtuelle et Cognition, pp. 123–132. ENST Paris, 1052

France (1999) 1053

123


Au

tho

r P

ro

of

View publication statsView publication stats

Design of a virtual reality training system for human–robot … · proof Int J Interact Des Manuf...

Documents

Transcript of Design of a virtual reality training system for human–robot … · proof Int J Interact Des Manuf...