Received September 25, 2019, accepted October 15, 2019, date of publication October 22, 2019, date of current version November 4, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2948879

A Tele-Operated Display Witha Predictive Display AlgorithmYEONGTAE JUNG , KYUTAEK HAN , AND JOONBUM BAE , (Member, IEEE)Department of Mechanical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan 44919, South Korea

Corresponding author: Joonbum Bae (jbbae@unist.ac.kr)

This work was supported in part by the 2019 Research Fund (1.190011.01) of the Ulsan National Institute of Science and Technology(UNIST), and in part by the National Research Foundation of Korea (NRF) Grant funded by the Korean Government (MSIT)(No. NRF-2019R1A2C2084677). Preliminary results of this research were presented in the doctoral thesis of Yeongtae Jung [1].

ABSTRACT Tele-operated display systems with head mounted displays (HMD) are becoming popularas visual feedback systems for tele-operation systems. However, the users are suffered from time-varyingbidirectional delays caused by the latency and limited bandwidth of wireless communication networks.Here, we develop a tele-operated display system and a predictive display algorithm allowing comfortableuse of HMDs by operators of tele-operation systems. Inspired by the kinematic model of the human head-neck complex, we built a robot neck-camera system to capture the field of view in any desired orientation.To reduce the negative effects of the time-varying bidirectional communication delay and operation delay ofthe robot neck, we developed a predictive display algorithm based on a kinematic model of the human/robotneck-camera system, and a geometrical model of a camera. Experimental results showed that the systemprovide predicted images with high frame rate to the user.

INDEX TERMS Tele-operateion, head mounted display, delay compensation.

I. INTRODUCTIONSites of disasters such as natural disasters, radioactive acci-dents and chemical accidents, which can pose problems interms of human access, are increasing [2], [3]. Tele-operationsystems have been researched to work in such disaster siteswith human user’s intelligence [4]–[7]. In the tele-operatedsystems, the user typically obtains the information of the sitesfrom two-dimensional images transmitted to monitors bycameras attached to the tele-operated robot. Visual data is themost intuitive form of information when observing environ-ments. However, the performance of a tele-operated system islimited if three-dimensional (3D) stereoscopic images are notprovided, because the user cannot gauge distances betweenobjects.

Recently, head mounted displays (HMDs) affordingimmersive 3D visual feedback have been used to observeenvironments and control tele-operated robots to performmanipulations [8]–[10]. Most HMDs feature integrated iner-tial measurement units (IMUs) that measure the orientationof the user’s head. The head orientation is used to capture thefield of view (FOV) in the direction in which the user looks by

The associate editor coordinating the review of this manuscript and

approving it for publication was Zheng Chen .

employing a robot neck-camera system or a panoramic cam-era of tele-operated robots. However, it is difficult to providereal-time images in such tele-operated situations; the latency,limited bandwidth of the tele-communication network, andthe large sizes of stereoscopic images impose delays andpacket loss. Such delays and loss of data induced by unstablenature of wireless tele-communication network trigger time-varying delays in image presentation, causing simulator sick-ness [11]–[13]. The operation delays imposed by the physicallimitations of robot neck-camera system increase the delayfurther.

Delays in tele-operation system have been researched fordecades, especially to ensure stability and transparency ofbilateral haptic systems. Many control algorithms, such aswave-variable based passivity control algorithms [14], [15]and modified algorithms to address constant [16], [17] andtime-varying delays [18]–[20] have been researched. Adap-tive and robust control algorithms have been studied todeal time-delayed systems with nonlinearities and uncertain-ties [21]–[24]. Despite of the achievements, it is difficult toapply such methods in visual feedback systems because ofdifferent nature.

The use of point cloud data or images to construct virtualworlds (rather than images recently captured by the robot

VOLUME 7, 2019 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ 154447

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

camera) may mitigate the delay effect problems [9], [25],[26]. However, such methods impose large computationalburdens or require expensive sensors. The delay effect couldbe reduced if the user’s head motion is predicted, and theremotely controlled robot moves ahead of the user. Predictionof user’s head motion have been researched to reduce thedelay of HMDs for displaying virtual reality [27], [28]. How-ever, it is difficult to apply such predictions, because largedelays and packet loss are expected in the communicationsunder disaster situations. Model-based predictions have beenresearched to mitigate the effect of delays in tele-operationsystems. The position and orientation of robot manipula-tors were predicted by kinematic models of the robots andreflected in the user-side display, to avoid long task timeinduced by ‘‘move and wait’’ strategy [29], [30] or increasemanipulation precision [31]. However, user’s head motionwas not involved in the systems, because the display systemsconsisted of ordinary monitors. A predictive display methodwas suggested in [32] to provide the image in actual direc-tion of a tele-operated vehicle utilizing the dynamics of thevehicle, but it was not implemented in real system.

Here, we developed a tele-operated display systemand a practical predictive display algorithm that compen-sates for the bidirectional network and operation delays.We constructed a robot neck-camera system based on a kine-matic model of the human head-neck complex. The displayalgorithm predicts images in the direction of user’s headorientation by employing delayed image and analyzing thedifference between the delayed robot neck orientation and thecurrent user head orientation. We compensated for the bidi-rectional communication and operational delays by translat-ing and rotating the delayed images using kinematic modelsof the human neck and the robot neck-camera system, and ageometrical model of the camera. The delays were addressedby predicting images which correspond to the current user’shead orientation, utilizing the delayed images, robot neckorientation, and current user’s head orientation.

The remainder of this paper is organized as follows.Section II provides an overview of our tele-operated displaysystem. Section III contains a detailed analysis of the cam-era and the predictive display algorithm. The experimentalsetup and the results afforded by the proposed algorithm areshown in Section IV. Section V presents the conclusions anddescribes planned future work.

II. OVERVIEW OF THE DISPLAY SYSTEMWe developed an intuitive display system for tele-operatedrobots. As shown in Figure 1, the human head-neck complexwas modeled as a simplified series linkage system featuringthree revolute joints to represent the rotational head motion.The HMD [33] attached to the user’s head measures headorientation, allowing presentation of 3D stereoscopic images.The robot neck had the same kinematic structure as the humanmodel. Two cameras [34] were placed on top of the robotneck to capture stereoscopic view of the work site. To pre-vent user’s sickness caused by time differences between the

FIGURE 1. Design of a robot neck-camera module based on the kinematicmodel of human head-neck complex. (θ1, θ2 and θ3 are the rotation angleof each rotational joints that represents yaw, pitch and roll motion of thehead, respectively).

TABLE 1. Specifications of the tele-operated display system.

stereoscopic cameras, the cameras were synchronized by dig-ital signals; both captured images at the same moment. Thedetailed specifications of the tele-operated display system arelisted in Table 1.

Figure 2 is a schematic of the system. The orientation ofthe user’s head (oH (t)) is measured by an IMU inside theHMD, with a measurement delay of 1IMU . The measuredorientation [oH (t − 1IMU )] is delivered to the robot neck-camera through a wireless network with a communicationdelay of 1CM , and the robot neck then follows the user’shead orientation [oH (t −1IMU −1CM )] with an operationaldelay 1OP caused by physical limitations of the actuators.The cameras captures stereo images in the direction of theirorientation oR(t), which is the same as the human neck ori-entation but delayed by 1IMU , 1CM and 1OP; the delayedorientation is denoted as oH (t −1IMU −1CM −1OP). Thecaptured images [Image(oR(t))] and the orientations of imagecapture [oR(t)] are delivered to the user with an additionalcommunication delay of 1CM .

154448 VOLUME 7, 2019

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

FIGURE 2. Overview of the tele-operated display system. oH and oR are the orientation of user’s head and robot neck-camera module, respectively.1IMU , 1CM and 1OP represent the measurement delay of user’s head orientation, communication delay of the wireless network and operationaldelay of the robot neck, respectively.

The images can be delivered directly to the user viathe HMD. However, as shown in Figure 2, the deliveredimages [Image(oR(t − 1CM )) = Image(oH (t − 1IMU −

21CM − 1OP))] contain the measurement delays of theIMU [1IMU ], bidirectional communication delay [21CM ],and the operational delay of the robot neck [1OP]. If theimage is provided directly to the user, the user may sufferbecause of the delays. Especially, bidirectional communica-tion delay is time-varying in nature and can trigger simu-lator sickness [11]–[13]. Also, random loss of image data,which is frequent because of the size of stereoscopic images,increases variation in 1CM . Such delays and image loss areinevitable in wireless networks, unless the communication isexceptionally well-controlled. 1OP is also time-varing, as itdepends on user’s head motion. 1CM and 1OP are the pre-dominant delays;1IMU is both relatively small and constant.To reduce the undesirable effects of 1CM and 1OP (suchas sickness [11]–[13]), we develop a predictive compensa-tion algorithm. The algorithm modifies the delivered images[Image(oR(t − 1CM ))] using delayed robot neck orienta-tions [oR(t −1CM )] and the measured user head orientation[oH (t − 1IMU )] before providing the images. In this way,the image which is supposed to be in the direction of user’shead is predicted; thus, the provided image instantly reflectsuser’s head motion despite the existence of time-varyingdelays. The algorithm compensates for1CM and1OP, whichare the dominant time-varying delays of the entire system thataffect to the sickness of users.

III. THE PREDICTIVE DISPLAY ALGORITHMA. DERIVATIONWe first analyzed camera geometry. In this analysis,we assumed that the camera is a pinhole camera without lensdistortion. Also, the image sensor was assumed to be square.

The camera was analyzed by dividing robot neckmotions intoyaw, pitch motion (θ1 or θ2 in Figure 1, respectively) and rollmotion (θ3 in Figure 1) of the robot neck. Figure 3 showsthe simplified geometry; the camera features an aperture andan image sensor. Figure 3a shows image formation on theimage sensor and the position change of the formed image byyaw or pitch motion of the robot neck. If a subject is placedin the direction of θ from the perpendicular line of the sensor,the distance from the center of the image sensor to the formedsubject image [op] can be approximated as follows:

op = dtanθ ≈ dθ (1)

where d is the distance between the aperture and the cameraimage sensor. If the camera is rotated by θ ′, the distancechanges to op′, as follows:

op′ = dtan(θ + θ ′) ≈ d(θ + θ ′) (2)

Using (1) and (2), the positional change in a formedimage [pp′] can be calculated as:

pp′ = op′ − op = dθ ′ (3)

Note that pp′ is proportional to the translation distance of thesubject in the captured image (in pixels, δpixel) as follows:

δpixel = αdθ ′ (4)

where α is a conversion factor used to transform the distancechange of the subject image on the image sensor to that in thecaptured image in pixels. Figure 3b shows image formationof a subject on the image sensor and its position change dueto the roll motion by θ ′. In such a case, the image sensor alsorotates by θ ′, as does the subject in the captured image.

The relationship between changes in camera orientationand positional change of the subject in captured images canbe used to predict future images based on current images.

VOLUME 7, 2019 154449

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

FIGURE 3. Simplified camera geometry and image position change of asubject by camera motions.

If the camera joints corresponding to θ1, θ2 and θ3 of thehuman kinematic model are rotated by θ ′1, θ

′

2 and θ′

3 respec-tively, the future image can be predicted by translating thecurrent image by αdθ ′1 in the horizontal direction and αdθ ′2in the vertical direction, and by rotating the image through θ ′3.Similarly, the scene in the direction of the current user headorientation can be predicted using that current orientation ofuser’s head [oH (t −1IMU )], the camera orientation when thedelayed image was captured [oR(t −1CM )], and the delayedimage [Image(oR(t −1CM ))].

The image prediction can be implemented by introducing acropped area, and translating or rotating it by the orientationdifference between the robot and human for yaw, pitch androll motion, respectively. An example of image manipula-tion in the case of yaw motion (rotation of θ1) is shownin Figure 4a. Here, a single axis-rotation case is considered,for the clarity of the explanation. The camera captures animage in the direction of its current orientation, as measuredby the robot neck system, and both the image [Image(oR(t))]and the robot neck orientation [oR(t)] are delivered to the userside with a delay of 1CM . If the predictive display algorithm

is not applied, the delivered image does not change until a newimage arrives. Thus, the delay generates an orientation differ-ence between the user head orientation [oH (t − 1IMU )] andthe currently displayed image’s camera orientation [oR(t −1CM ) = oH (t −1IMU − 21CM −1OP)].

However, the predictive display algorithm provides amanipulated image (not an original) to the user. A rectangularcropping area imposed on the delivered image [Image(oR(t−1CM ))] with horizontal and vertical margins of Mh and Mv,respectively. If the user head orientation changes, the croppedarea is translated by αdθ ′ pixels in the direction of headrotation, where θ ′ is the difference between the current userhead orientation [oH (t −1IMU )] and the delayed robot neckorientation [oR(t − 1CM )]. The translated cropped image isthen provided to the user through the HMD; the extent oftranslation is determined by the camera model of (4).

Similarly, the algorithm can deal with pitch and roll motionof the user head by applying the manipulation sequentially(Figure 4b). The cropped areas are translated horizontally andvertically by αdθ ′1 and αdθ

′

2, respectively, and rotated by θ ′3,where θ ′1, θ

′

2 and θ′

3 are the yaw, pitch and roll components ofthe difference between oH (t−1IMU ) and oR(t−1CM ). As thealgorithm predicts images in the direction of current userhead orientation [oH (t−1IMU )], the effect of1CM and1OPis compensated. Thus, image immediately change followinguser head rotation feature a delay of only1IMU , which is bothconstant and much smaller than the time-varying delays1CMand 1OP. Therefore, the predictive display algorithm canreduce the discomfort feelings or the sickness of the displaysystem caused by delays.

A delay compensation algorithm was proposed inEdwards’ patent with a similar approach [35]. However, thatmethod yields an image smaller than or equal to the FOV ofthe HMD, with negative or zero Mh and Mv values. Thus,the image shown in the HMD looks like a scene througha window that is translated and rotated when the image isupdated. This may negatively affect user immersion, limitingthe user-side FOV. Although the concept is similar to ours,there was no analysis about the camera geometry and imageposition change; our analyses support the validity of ouralgorithm. Also, the extent of margins were not considered;we discuss these margins in Section III.B.

B. MARGIN ANALYSISThe margins must be set considering the rotational speedof the user’s head and the image delays caused by 1CMand 1OP. If the cropping area is overlapped with the out-side of image, the user no longer receives a square image,but rather a clipped image, because information is lackingoutside the image. This compromises task performance ofthe tele-operation system by reducing user immersion. Thus,the margins Mh and Mv must be large enough to make thecropped area do not attain the image edges. However, largemargins reduce the image FOV in turn reducing the informa-tion imparted. Given this trade-off, the margins must be as

154450 VOLUME 7, 2019

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

FIGURE 4. Implementation procedure of the predictive display algorithm.

small as possible but sufficiently large to prevent overlappingwith cropped areas the outside the image.

The required margins can be calculated from the imagedelay, and the rotational speeds of the user’s head in yaw (θ̇1),pitch (θ̇2) and roll (θ̇3). Figure 5 shows the translation androtation of the cropped area. The four corners are denoted A,B, C , and D. As one of these points will be the first point toreach the edge of the image when the cropped area beginsto overlap with and area outside the image, it is necessaryto ensure that all points stay within the image. Margin anal-ysis commences at point A with initial coordinates (xA, yA).The distances between A and closest horizontal and verticalimage edges are denoted Mh,A and Mv,A, respectively. Thecoordinates of point A prior to translation and rotation of thecropped area are:

xA = lcosθi (5)

yA = lsinθi (6)

where l is the distance from the center of the image to A, andθi is the angle between the horizontal line and OA, calculated

using the margins and image size, as follows:

l =√(Lh/2−Mh)2 + (Lv/2−Mv)2 (7)

θi = atan(Lv/2−Mv

Lh/2−Mh

)(8)

Before manipulation of the cropped area, Mh,A and Mv,A areidentical to Mh and Mv. As the area is translated and rotated,the coordinates of point A change:

x ′A = lcos(θi + θ ′3)+ αdθ′

1 (9)

y′A = lsin(θi + θ ′3)+ αdθ′

2 (10)

Assuming an image delay of dt and user rotational speeds ofθ̇1, θ̇2, θ̇3 (yaw, pitch and roll axis, respectively), the positionalchange of pointA caused by usermotion during the delay timedt can be calculated as follows:

x ′A = lcos(θi − θ̇3dt)− αd θ̇1dt (11)

y′A = lsin(θi − θ̇3dt)+ αd θ̇2dt (12)

VOLUME 7, 2019 154451

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

FIGURE 5. Margin changes by the translation and rotation of the croppedarea.

The positional change varies the margins Mh,A and Mv,A asfollows:

M ′h,A = Lh/2− x ′A (13)

M ′v,A = Lv/2− y′A (14)

To ensure that the cropped area does not attain the edges ofthe image,Mh,A andMv,A must satisfy the following criteria:

M ′h,A,M′v,A > 0 (15)

Similarly, the coordinates and margins of the other threepoints after manipulation of the cropped area are:

x ′B = lcos(θi + dotθ3dt)− αddotθ1dt (16)

y′B = −lsin(θi + dotθ3dt)+ αddotθ2dt (17)

x ′C = −lcos(θi − dotθ3dt)− αddotθ1dt (18)

y′C = −lsin(θi − dotθ3dt)+ αddotθ2dt (19)

x ′D = −lcos(θi + dotθ3dt)− αddotθ1dt (20)

y′D = lsin(θi + dotθ3dt)+ αddotθ2dt (21)

M ′h,B = Lh/2− x ′B (22)

M ′v,B = Lv/2− y′B (23)

M ′h,C = Lh/2− x ′C (24)

M ′v,C = Lv/2− y′C (25)

M ′h,D = Lh/2− x ′D (26)

M ′v,D = Lv/2− y′D (27)

Mh,B, Mv,B, Mh,C , Mv,C , Mh,D and Mv,D are also required tosatisfy following criteria:

M ′h,B,M′v,B,M

′h,C ,M

′v,C ,M

′h,D,M

′v,D > 0 (28)

FIGURE 6. Experimental setup for the parameter identification.

such that the cropped area does not reach to the edge of theimage. Using these criteria, the minimal required marginscan be calculated by applying the maximum angular velocityof the user’s head (θ̇1,max , θ̇2,max and θ̇3,max) and maximumimage delay (dtmax) into (11)-(12) and (16)-(21). On the con-trary, the delay that can be dealt with a set of margins can beobtained by assigning the margins to (7)-(8) and calculatingdt that satisfies (15) and (28).

The problem here is that the image delay changes sig-nificantly as tele-communication network conditions vary.To address this problem, margins can be calculated andapplied in realtime, because our predictive display algorithmdoes not impose a large computational burden. However,margin variation also changes the image FOVs, which maydisturb the user. Also, there are limitations in the margins,since the image have limited size. If the user’s head motionis fast enough to escape the margins in the given networkdelay, reaching to the outside of image is unavoidable. Thus,we determined the margins empirically in the following per-formance tests. The margins were first set to zero, and thenwere increased until the cropped area does not reach to theedge of image during the experiments so that the margin cri-teria can be satisfied under given delay and user’s rotationalspeed in a test operation.

IV. EXPERIMENTSA. PARAMETER IDENTIFICATIONTo apply the predictive display algorithm, the camera param-eterαd of (4) was identified by an experiment. Figure 6 showsthe experimental setup. The robot neck-camera module wasfixed, and a subject was placed in front of the camera at a dis-tance of 1m, occupying the center of the imagewhen the neckof the robot was in the initial position. The robot neck wasrotated (in yaw or pitch) from −30◦ to 30◦ in 1◦ steps withthe camera operating. The distance between the subject in thecaptured image and the image center was recorded in pixels;the αd parameter was identified by linear fitting of αd to (4),applying the least squares method to the recorded positionand rotational angle. Thus, αd was identified as αd = 5.5pixel/deg for yaw, and αd = 5.9 pixel/deg for pitch with rootmean square (RMS) errors of 3.3 pixel and 4.1 pixel respec-tively. Since there was no significant outlier, the identifiedmodel showed good agreement with the measured data with

154452 VOLUME 7, 2019

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

FIGURE 7. Identification result of camera parameter αd by yaw and pitchmotions.

FIGURE 8. A random network delay.

the least squares method. Figure 7a and Figure 7b show themeasured positions of the subjects and the linearly fittedmod-els for yaw and pitch of the robot neck, respectively. As thegraphs show, the measured data and camera model of (4)were in good agreement. The parameters differed slightly interms of both yaw and pitch motion, caused by lens distortionor/and image sensor asymmetry. The parameters depend onthe physical specifications of the lens and the image sensorof camera, such as the refractive index of the lens and the sizeand position of the image sensor, which are not time-varyingspecifications. Thus, pre-identified camera parameters wereused in the implementation of predictive display algorithm.

B. IMPLEMENTATION OF THE PREDICTIVE DISPLAYALGORITHMThe predictive display algorithm was tested in a real-worldtele-operation experiment. The experimental setup was thesame as that of Figure 2. The user wore the HMD androtated the head; head orientation was measured and deliv-ered to the robot neck-camera module through a wirelessnetwork; the robot neck followed the user’s motion while astereo camera captured environmental images at a resolutionof 554 × 413 pixels. The captured images and robot neck

FIGURE 9. Captured images in the tele-operation experiment. Focus onchanges of image edges (e.g., the robot on the right side), or the movingsubject (i.e., the human). Check the attached video file for more details.

orientations when the images were taken were delivered tothe user side. A random delay was intentionally added to thewireless network to reflect network conditions in practicaltele-operated applications. The captured image and robotneck orientations were sent to the user side after randomdelay with a maximum magnitude of 230 ms and a minimumof 70 ms, representing a communication delay in the tele-operated display system. Both images (i.e., those subjectedto and not subjected to predictive display algorithm) wererecorded. The margins of the predictive display algorithmwere set to appropriately tuned values (Mh = 90, Mv = 50)as in Section III, or inappropriate values (Mh = 0, Mv = 0

VOLUME 7, 2019 154453

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

FIGURE 10. Experimental results with inappropriate margins. ((a): Zeromargins, (b): Negative margins) Check the attached video file for moredetails.

and Mh = −90, Mv = −50) as suggested in Edwards’patent [35] Given that the frame rate of most digital videos is30 frames per second (FPS), image prediction proceeded atin 30 Hz.

The recorded results are attached as a supplementary videofile. The bidirectional network delay is shown in Figure 8.The minimum, average, maximum, and standard deviationof the delay were 145, 245, 361, and 48 ms, respectively.Figure 9 shows sets of updated images from scenes (1) to(3) without/with the predictive algorithm operating underthe same experimental conditions with appropriate margins.In the absence of the predictive display algorithm, the cap-tured images do not change until new images arrive. However,the images modified by the predictive algorithm change con-tinuously as the user’s head rotates. As shown in the videofile, the delay-compensated images are much smoother andmore natural than those presented in the absence of the algo-rithm. The desired frame rate (30 FPS) was maintained evenin the presence of unpredictable time-varying delays. Theresults imply that the proposed algorithm allows operators oftele-operation systems to comfortably use their HMDs.

A feature analysis was performed to verify the performanceof proposed algorithm. In this analysis, the coordinate differ-ences of keypoints between the images before and after thearrival of new image were compared, as shown in Fig. 11a.A fast library for approximate nearest neighbors (FLANN)matcher in openCV was used in the analysis. The averagevalues of coordinate differences are plotted in Fig. 11b. Theaverage difference was 11.0 pixels and 1.9 pixels for theimages without the predictive display algorithm and withthe algorithm, respectively. The coordinate difference wassignificantly reduced when the predictive display algorithmwas applied; i.e., the video with proposed algorithm is morecontinuous, providing a predicted image similar to the origi-nal image to be updated.

However, the algorithm cannot deliver the change of envi-ronment instantly, as reflected in Figure 9c and the videowith a moving subject. The motion of a moving subject isnot reflected until the image is updated in (3). This is a fun-damental limitation of the algorithm, because the algorithmutilizes delayed images for the prediction. Also, inappropriatemargins negatively affect user immersion as we concerned inSection III, and as shown in the last part of the supplemen-tary video and Figure 10, limiting the user-side FOV. Thus,

FIGURE 11. Feature analysis results.

the margins must be selected to satisfy the criteria introducedin Section III.

V. CONCLUSIONIn this research, a display system with a predictive displayalgorithm was developed. A robot neck-camera module wasdesigned and manufactured based on a kinematic model ofthe human head-neck complex. The image delay during thetele-operation was analyzed; this may trigger simulator sick-ness. To deal with this issue, a predictive display algorithmwas developed based on the camera geometry and humanand robot kinematics. The image for the current user headorientation was predicted and provided to the user, exploitingthe difference between the current user head orientation andthe delayed camera orientation. Thus, the time-varying delaysin communication and operation were compensated, and con-tinuous images were thus provided to the user. The proposedalgorithm is simple to implement in real-time tele-operationsystems without expensive sensors or computational burden,yet powerful as demonstrated in the experiments. The featureanalysis result shows that the proposed algorithm is effectivein predicting images.

However, some challenges remain. The modeling uncer-tainties of camera geometry, such as tangent approximationand certain aspects of the pinhole camera model, shouldbe reflected in the algorithm. The cropping margin deci-sion strategy requires refinement for use under various tele-communication network conditions. If the user’s motion isfast enough to escape the margins, the proposed algorithmcannot provide properly predicted image, as in [35]. Also,

154454 VOLUME 7, 2019

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

the predictive display algorithm does not instantly reflecta change in the environment, because the algorithm useddelayed images. Nevertheless, the proposed algorithm is asignificant practical utility because it is simple and imposesa low computational burden. User evaluation in various sit-uations is essential; we will quantitatively evaluate improve-ments in user comfort and employ these data to guide ourfuture research.

REFERENCES[1] Y. Jung, ‘‘A wearable control interface for Tele-operated robots.’’ Doctoral

thesis, Ulsan Nat. Inst. Sci. Technol., Ulsan, South Korea, Feb. 2019.[2] L. Bevere, R. Sharan, S. Barrett, and C. Honegger, ‘‘Natural catastrophes

and man-made disasters in 2016: A year of widespread damages,’’ Sigma,vol. 2, pp. 1–43, Feb. 2017.

[3] D. Guha-Sapir, P. Hoyois, P. Wallemacq, and R. Below, ‘‘Annual disasterstatistical review 2016: The numbers and trends,’’ Centre Res. Epidemiol.Disasters, Brussels, Belgium, Oct. 2017, pp. 1–80.

[4] J. Lim et al., ‘‘Robot system of DRC-HUBO+ and control strategy of teamKAIST inDARPA robotics challenge finals,’’ J. Field Robot., vol. 34, no. 4,pp. 802–829, 2017.

[5] S.-J. Yi, S. G. McGill, L. Vadakedathu, Q. He, I. Ha, J. Han, H. Song,M. Rouleau, B.-T. Zhang, D. Hong, M. Yim, and D. D. Lee, ‘‘TeamTHOR’s entry in the DARPA robotics challenge trials 2013,’’ J. FieldRobot., vol. 32, no. 3, pp. 315–335, 2015.

[6] M. Johnson et al., ‘‘Team IHMC’s lessons learned from the DARPArobotics challenge: Finding data in the rubble,’’ J. Field Robot., vol. 34,no. 2, pp. 241–261, 2017.

[7] S. Gray, R. Chevalier, D. Kotfis, B. Caimano, K. Chaney, A. Rubin,and K. Fregene, ‘‘An architecture for human-guided autonomy: TeamTROOPER at the DARPA robotics challenge finals,’’ J. Field Robot.,vol. 34, no. 5, pp. 852–873, 2017.

[8] C. L. Fernando, M. Furukawa, T. Kurogi, S. Kamuro, K. Sato,K. Minamizawa, and S. Tachi, ‘‘Design of TELESAR V for transferringbodily consciousness in telexistence,’’ in Proc. IEEE/RSJ Int. Conf. Intell.Robots Syst. (IROS), Oct. 2012, pp. 5112–5118.

[9] M. Schwarz, T. Rodehutskors, D. Droeschel, M. Beul, M. Schreiber,N. Araslanov, I. Ivanov, C. Lenz, J. Razlaw, S. Schüller, D. Schwarz,A. Topalidou-Kyniazopoulou, and S. Behnke, ‘‘NimbRo rescue: Solvingdisaster-response tasks with the mobile manipulation robot momaro,’’J. Field Robot., vol. 34, no. 2, pp. 400–425, 2017.

[10] S. Park, Y. Jung, and J. Bae, ‘‘An interactive and intuitive control inter-face for a tele-operated robot (AVATAR) system,’’ Mechatronics, vol. 55,pp. 54–62, Nov. 2018.

[11] J. Golding and H. Markey, ‘‘Effect of frequency of horizontal linearoscillation onmotion sickness and somatogravic illusion,’’Aviation, Space,Environ. Med., vol. 67, no. 2, pp. 121–126, 1996.

[12] E. L. Groen and J. E. Bos, ‘‘Simulator sickness depends on frequency ofthe simulator motion mismatch: An observation,’’ Presence, TeleoperatorsVirtual Environ., vol. 17, no. 6, pp. 584–593, 2008.

[13] M. E. S. Pierre, S. Banerjee, A. W. Hoover, and E. R. Muth, ‘‘The effectsof 0.2 Hz varying latency with 20–100 ms varying amplitude on simu-lator sickness in a helmet mounted display,’’ Displays, vol. 36, pp. 1–8,Jan. 2015.

[14] G. Niemeyer and J.-J. E. Slotine, ‘‘Using wave variables for system analy-sis and robot control,’’ in Proc. IEEE Int. Conf. Robot. Autom., Apr. 1997,pp. 1619–1625.

[15] G. Niemeyer and J.-J. E. Slotine, ‘‘Stable adaptive teleoperation,’’ IEEEJ. Ocean. Eng., vol. 16, no. 1, pp. 152–162, Jan. 1991.

[16] D. Lee and M. W. Spong, ‘‘Passive bilateral teleoperation with con-stant time delay,’’ IEEE Trans. Robot., vol. 22, no. 2, pp. 269–281,Apr. 2006.

[17] A. Aziminejad, M. Tavakoli, R. V. Patel, and M. Moallem, ‘‘Trans-parent time-delayed bilateral teleoperation using wave variables,’’IEEE Trans. Control Syst. Technol., vol. 16, no. 3, pp. 548–555,May 2008.

[18] D. Sun, F. Naghdy, and H. Du, ‘‘Wave-variable-based passivity con-trol of four-channel nonlinear bilateral teleoperation system under timedelays,’’ IEEE/ASME Trans. Mechatronics, vol. 21, no. 1, pp. 238–253,Feb. 2016.

[19] D. Sun, F. Naghdy, and H. Du, ‘‘Neural network-based passivity controlof teleoperation system under time-varying delays,’’ IEEE Trans. Cybern.,vol. 47, no. 7, pp. 1666–1680, Jul. 2017.

[20] C. Yang, X. Wang, Z. Li, Y. Li, and C.-Y. Su, ‘‘Teleoperation con-trol based on combination of wave variable and neural networks,’’IEEE Trans. Syst., Man, Cybern., Syst., vol. 47, no. 8, pp. 2125–2136,Aug. 2017.

[21] Z. Chen, Y.-J. Pan, and J. Gu, ‘‘Integrated adaptive robust controlfor multilateral teleoperation systems under arbitrary time delays,’’Int. J. Robust Nonlinear Control, vol. 26, no. 12, pp. 2708–2728,2016.

[22] H. Wang, P. X. Liu, and S. Liu, ‘‘Adaptive neural synchronization con-trol for bilateral teleoperation systems with time delay and backlash-like hysteresis,’’ IEEE Trans. Cybern., vol. 47, no. 10, pp. 3018–3026,Oct. 2017.

[23] P. M. Kebria, A. Khosravi, S. Nahavandi, P. Shi, and R. Alizadehsani,‘‘Robust adaptive control scheme for teleoperation systems with delay anduncertainties,’’ IEEE Trans. Cybern., to be published.

[24] Z. Chen, F. Huang, W. Chen, J. Zhang, W. Sun, J. Chen, J. Gu, andS. Zhu, ‘‘RBFNN-based adaptive sliding mode control design for delayednonlinear multilateral tele-robotic system with cooperative manipulation,’’IEEE Trans. Ind. Informat., to be published.

[25] K. Theofilis, J. Orlosky, Y. Nagai, and K. Kiyokawa, ‘‘Panoramicview reconstruction for stereoscopic teleoperation of a humanoidrobot,’’ in Proc. IEEE-RAS 16th Int. Conf. Hum. Robots, Nov. 2016,pp. 242–248.

[26] M. Jagersand, A. Rachmielowski, D. Lovi, N. Birkbeck,A. Hernandez-Herdocia, A. Shademan, D. Cobzas, and K. Yerex,‘‘Predictive display from computer vision models,’’ in Proc. Int. Symp.Artif. Intell., Robot. Automat. Space, 2010, pp. 673–680.

[27] R. Azuma and G. Biship, ‘‘A frequency-domain analysis of head-motionprediction,’’ in Proc. SIGGRAPH, vol. 95, 1995, pp. 401–408.

[28] J. R. Wu and M. Ouhyoung, ‘‘On latency compensation and its effects onhead-motion trajectories in virtual environments,’’ Vis. Comput., vol. 16,no. 2, pp. 79–90, 2000.

[29] W. S. Kim and A. K. Bejczy, ‘‘Graphics displays for operator aid intelemanipulation,’’ in Proc. IEEE Int. Conf. Syst., Man, Cybern., 1991,pp. 1059–1067.

[30] T. Yuichi and M. Uchiyama, ‘‘Predictive display of virtual beam for spaceteleoperation,’’ in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS),1996, pp. 1544–1549.

[31] F. Richter, Y. Zhang, Y. Zhi, R. K. Orosco, and M. C. Yip, ‘‘Aug-mented reality predictive displays to help mitigate the effects of delayedtelesurgery,’’ in Proc. Int. Conf. Robot. Autom. (ICRA), May 2019,pp. 444–450.

[32] M. J. Brudnak, ‘‘Predictive displays for high latency teleoperation,’’ inProc. NDIA Ground Vehicle Syst. Eng. Technol. Symp., 2016, pp. 1–16.

[33] Oculus VR. (2018). Oculus Rift. [Online]. Available: https://www.oculus.com/

[34] FLIR. (2018) Flea3. [Online]. Available: https://www.flir.com/[35] E. C. Edwards, ‘‘Method and system for time/motion compensation for

head mounted displays,’’ U.S. Patent 7 312 766 B1, Dec. 25, 2007.

YEONGTAE JUNG received the B.S. degree inmechanical and advanced materials engineeringand the Ph.D. degree in mechanical engineer-ing from the Ulsan National Institute of Scienceand Technology (UNIST), Ulsan, South Korea,in 2013 and 2019, respectively. He is currently aSenior Researcher with Korea Shipbuilding andOffshore Engineering, Ulsan. His current researchinterests include modeling, design and controlof human–robot interaction systems, teleoperation

systems, and actuation mechanisms. He received the Korean GovernmentMinister Awards from the Ministry of Science, ICT and Future Planning,in 2017. He was a recipient of the Global Ph.D. Fellowship supported by theNational Research Foundation of Korea, from 2013 to 2018.

VOLUME 7, 2019 154455

Y. Jung et al.: Tele-Operated Display With a Predictive Display Algorithm

KYUTAEK HAN received the B.S. degree inmechanical system design engineering from theSeoul National University of Science and Tech-nology, Seoul, South Korea, in 2016, and theM.S. degree in mechanical engineering from theUlsan National Institute of Science and Technol-ogy (UNIST), Ulsan, South Korea, in 2019. Heis currently a Researcher with the Korea CultureTechnology Institute. His current research inter-est includes imaging technology and platform forimmersive experience.

JOONBUM BAE (S’07–M’12) received the B.S.degree (summa cum laude) in mechanical andaerospace engineering from Seoul National Uni-versity, Seoul, South Korea, in 2006, and the M.S.degree inmechanical engineering, theM.A. degreein statistics, and the Ph.D. degree in mechanicalengineering from the University of California atBerkeley, Berkeley, CA, USA, in 2008, 2010, and2011, respectively.

In 2012, he joined the Department of Mechan-ical Engineering, Ulsan National Institute of Science and Technology(UNIST), Ulsan, South Korea, where he is currently the Director of theBio-Robotics and Control (BiRC) Laboratory. His current research interestsinclude modeling, design, and control of human-robot interaction systems,soft robotics, and biologically inspired robot systems.

Dr. Bae received Korean Government Minister Awards from the Min-istry of Public Safety and Security and the Ministry of Science, ICT andFuture Planning, in 2016 and 2017, respectively. He also received theYoung Researcher Award from the Korea Robotics Society, in 2015. Hewas a finalist of the semi-plenary Paper Award at the American Society ofMechanical Engineers Dynamic Systems and Control Conference, in 2012,and a finalist of the Best Poster Paper Award at the International Federationof Automatic Control World Congress, in 2008. He was a recipient of aSamsung Scholarship during his Ph.D. studies.

154456 VOLUME 7, 2019