Available online at ScienceDirecte-navigation.kr/down/2-Jaeyong OH.pdf · 2020-01-22 · Available...

Available online at www.sciencedirect.com

ScienceDirect International Journal of e-Navigation and Maritime Economy 5 (2016) 021 – 031

21

Original article

Advanced Navigation Aids System based on Augmented Reality*

Jaeyong OH 1, Sekil PARK 2, Oh-Seok KWON 3† 1 Korea Research Institute of Ships and Ocean Engineering, Korea, [email protected] 2 Korea Research Institute of Ships and Ocean Engineering, Korea, [email protected]

3† Dept. of Computer Engineering, Chungnam National University, Korea, [email protected], Corresponding Author

Abstract

Many maritime accidents have been caused by human-error including such things as inadequate watch keeping and/or mistakes in ship handling. Also, new navigational equipment has been developed using Information Technology (IT) technology to provide various kinds of information for safe navigation. Despite these efforts, the reduction of maritime accidents has not occurred to the degree expected because, navigational equipment provides too much information, and this information is not well organized, such that users feel it to be complicated rather than helpful. In this point of view, the method of representation of navigational information is more important than the quantity of that information and research is required on the representation of information to make that information more easily understood and to allow decisions to be made correctly and promptly. In this paper, we adopt Augmented Reality (AR) technologies for the representation of information. AR is a 3D computer graphics technology that blends virtual reality and the real world. Recently, this technology has been widely applied in our daily lives because it can provide information more effectively to users. Therefore, we propose a new concept, a navigational system based on AR technology; we review experimental results from a ship-handling simulator and from an open sea test to verify the efficiency of the proposed system.

Keywords: navigational aids system, augmented reality, IBS(Integrated Bridge System)

Copyright ⓒ 2016, International Association of e-Navigation and Ocean Economy. Hosting by Elsevier B.V.

This article is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/). Peer review under responsibility of Korea Advanced Institute for Intelligent Maritime Safety and Technology.

* This is a revised version presented at the 3rd Ai-MAST held at Rivera Hotel in Daejeon, Korea, November 12-14, 2015.

OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality

22

I. Introduction

The marine traffic environment has recently witnessed significant increases in traffic and trends

toward larger and faster ships, resulting in a greater risk of marine accidents. Data from the Korean

Maritime Safety Tribunal has revealed a steady rise in marine accidents over the last five years, with

80% of such accidents being caused by errors during operation. Among all accidents, collision

accidents were highest in number and mainly caused by human error such as negligence, and the

number of marine accidents continues to increase along with the resulting damages (Ministry of

Oceans and Fisheries, 2015). In order to prevent or help mitigate this, real-time support is provided

to ship navigators through the Vessel Traffic Services System (VTS) and the Aids to Navigation

System, and related legislation around the world is being carried out to prevent marine accidents.

Additionally, navigation systems with advanced IT technologies are being developed to provide

various data to navigators for safer operation; however, such complex equipment could be more of a

burden for navigators and interfere with safe operation. It cannot be assumed that all navigation

equipment provides useful information to navigators, and thus there is a need to analyze the current

limitations of different types of navigation equipment and solutions to such limitations from the

perspective of the navigators.

According to the bridge operations analysis results by STCW (The International Convention on

Standards of Training, Certification and Watchkeeping for Seafarers), the largest problem with

currently operating navigation equipment is the provision of excessive and unnecessary information

or inappropriate methods of data provision. In particular, the alert functionality with respect to

emergency situations is currently provided through RADAR and ECDIS, but such alert messages

are delivered through text messages or alarm sounds which are ineffective and not intuitive, so this

aspect requires further improvement. Additionally, an effective means of information delivery for

limited screen space needs to be further studied (Jeong J.S, 2012). In this respect, this paper

proposes an AR (Augmented Reality) technology-based navigational aid system that can be utilized

in bridge environments as an intuitive means of information delivery. Experimentation was

conducted to examine the application potential of the proposed system.

II. Analysis of Previous Study

As mentioned above, the complexity of the navigation equipment in the bridge can interfere

with safe operation; thus, navigational aid systems developed in the future need to support the

decision-making process of the navigator through effective information provision methods rather

than by simply increasing the amount of information delivered. Moreover, the intuitiveness of the

navigation data provided is a critical element that can resolve issues caused by excessive

information. This paper aims to improve upon this by utilizing augmented reality technology.

Augmented Reality is a field of VR (Virtual Reality) that fuses virtual objects or information

to the real environment through computer graphics technology so that virtual objects appear to

originate from the existing environment. This technology allows the display of information by

overlapping the information upon images of the real world. Considering that data obtained


23

through watch-keeping takes up a significant portion of the ship operation data provided, this

method of utilizing augmented reality as a method of information provision in a bridge may be

very effective.

Various studies are currently being carried out regarding such augmented reality technologies

along with the recent development of mobile devices, and applications to a diverse range of

industries including aerospace, automobile, and ships are increasing rapidly. The augmented

reality display technology, HUD (Heads-Up Display) system is being used in fighter jets and

commercial airliners to contain the time of the pilot and enhance the capabilities of the pilot (Kim

K.H. 2008). Currently, attempts to apply this HUD technology to the automobile industry are

being carried out in order to minimize visual interference to the driver while providing essential

information while driving. Generally, ships are relatively slower than aircraft or automobiles.

However, navigational equipment has been diversified with the development of new technologies.

Therefore, ships officers are requesting the efficient display of information on the bridge, and

various navigational aids systems are being researched and developed using augmented reality

technology as a means of displaying data.

In the European e-Navigation project, ACCSEAS, a wearable navigational aids system was

developed using HMD (Head-Mounted Display), as shown in Figure 1 (a), in order to enhance the

target recognition speed of the navigator. Though simulation experiments were conducted, further

research using the position data of the ship and navigator is still necessary for accurate registration

(ACCSEAS, 2015). Additionally, various navigation support applications have been developed

recently for mobile devices, as shown in Figure 1 (b), and such applications are being used in the

navigation of small ships, like yachts (B&G, 2015).

Figure 1: User Interfaces of AR Navigation System

(a) ACCSEAS Project, (b) Mobile application of SeaNav, B&G Source: ACCSEAS (2015), B&G (2015)

However, these systems all have different user interfaces, different methods of displaying

information, and different types of data expressed with no agreed upon standard, causing

inconveniences to the user and restrictions in their usage. Also, the indiscreet provision of

inaccurate data can result in serious problems such as marine accidents, further necessitating

research on what kind of data is to be shown and how such data will be delivered (Olivier Hugues,

2010). In this paper, a survey of navigators was conducted to determine key elements necessary

for navigation, and we then propose an augmented reality-based user interface which can

effectively express such data. Moreover, a navigational aid system to implement the proposed


24

interface is detailed, and we review experimental results from ship-handling simulator and an

open sea test to verify the efficiency of the proposed system.

III. Navigational Aids System based on AR

3.1. User Requirements As the first step in designing and implementing an augmented reality-based navigational aid

system, a survey was carried out to analyse user requirements. This survey was composed of

questions to determine necessary navigation data, effective methods of presenting such data, and

the arrangement or location of the presentation of the data. The survey was completed by 20

experienced ship officers. Analysis of the survey results revealed that most respondents preferred

navigation data to be presented and arranged in ways similar to how they are presented in current

navigation equipment and desired new functionality including the display of distance to a fairway.

Table 1: List of the Maritime Service Portfolios (MSPs) – taken from Annex 7 of NCSR 1/28)

Items Details Unit Type

GPS UTC/LAT/LON/SoG/CoG date/time/degrees/knots/degrees $xxRMC

heading heading degrees $xxHDT

STW speed through water knots $xxVHW

AIS AIS messages - $xxVDM

echo Sounder depth meter $xxDPT

tidal Current tidal current direction/speed degrees/knots $xxVHW

wind wind direction/speed degrees/knots $xxVWR

RoT rate of turn degrees/min $xxROT

rudder indicator rudder indicator degrees $xxRSA

RADAR target RADAR tracked target - $xxTTM

CCTV camera color image, pan/tilt degrees/degrees JPEG

Ownship attitude roll/pitch degrees/degrees -

Figure 2: User Interface of AR Navigation System


25

In addition, respondents showed positive responses toward the augmented reality system being

developed, stating that it could be used effectively in the entry and departure of ports. On the

other hand, responses showed that integration with the ECDIS system is necessary for the

provision of accurate navigation data and that data regarding fairways, anchorage, land area, and

planned routes are necessary for safe operation. Table 1 shows the necessary navigation data for

the augmented reality navigational aid system obtained from the survey results.

3.2. Design of User Interface

The information for the navigational aids system were categorized into 1) data regarding the

ownship in operation, 2) data regarding traffic ships, and 3) ECDIS data based on the survey

response analysis results. As shown in Figure 2, data regarding the ship in operation are mainly

displayed through the ODD (Overhead Data Display) in the bridge, composed of the location,

bearing, speed, and rate of turn (RoT), and displayed on the upper portion of the augmented

reality display using an intuitive interface, such as a graph. Data of traffic ships are displayed as

overlays on their corresponding positions, where detailed data was additionally displayed when

the user selected the data. The detailed data was generated based on the AIS data, and the targets

of RADAR and AIS were distinguished using symbols. The ECDIS data was displayed

selectively on the location of the ship in operation and followed the standard of ENC (Electronic

Navigational Charts) symbol. Additionally, the user was able to select whether to display each

data, and it was designed so that the user is able to adjust the data display color.

3.3. System Configuration

The hardware of the system is composed of a PTZ (Pan/Tilt/Zoom) camera, AHRS (Attitude

and Heading Reference System), NMEA Combiner, and user console with an additional joystick

device for camera control. The software system is composed of a data manager module, user

interface module, registration module, and augmented image rendering module. Figure 3 shows

the data flow between each module.

The first stage of the system collects navigational data through the NMEA combiner, and the

ENC data and planned route data are extracted through the ECDIS integration module. Following

this, the user interface module uses the collected navigation data to generate the display

component elements, processes user input such as through the joystick, and finally generates the

augmented image in real time through the registration and rendering modules for provision to the

navigator. In this experiment, a simulator system was also used to conduct tests in various

maritime conditions, and a simulator integration module was included for this. The detailed

specifications of each module are as follows.


26

Figure 3: Configuration of Navigation Aids System based on AR

(a) Data Manager Module

The data manager module processes navigation data collected from various navigation devices

of the bridge and performs integration and management functions. The collected navigation data

in our experiment followed the NMEA-0183 standard, was synchronized based on the GPS time

(UTC, Universal Time Coordinated), and was shared with other modules through Shared Memory.

And, multi-threading was used for performance enhancement to allow parallel processing of the

input navigation data.

(b) ECDIS Module

ENC are vector type charts that conform to International Hydrographic Organization (IHO)

specifications, as contained in Publication S-57. S-57 includes various feature object classes and

their attributes. The proposed system extracts and uses elements necessary for the augmented

reality navigational aids system from the ENC data of ECDIS. Data determined through analysis

of the requirements is extracted from ENC, and the extracted data is then stored in a separate

database system and displayed selectively depending on the location of the ownship in operation.

In addition, the distance to the next waypoint and the estimated arrival time data are calculated

from the route plan inputted through ECDIS.

(c) Unser Interface Module

The user interface module is the user interface (GUI) provided to the navigator and performs

the task of user input processing through mouse and joystick operation. Additionally, display

components are generated using the collected navigation data, and the option of whether to

display each component along with its colour is reflected.

(d) Augmented Image Rendering Module

In the augmented image rendering module, an augmented image is generated by integrating

the collected navigation data and ENC data. In order to generate the augmented image, the

augmented elements are arranged in a virtual 3-dimensional space and integrated with the real

world image in real time. In this study, a 3-dimensional graphics engine (Unity3D) that supports

GPU acceleration was used to enhance the performance of the image rendering. Additionally, the


27

registration process of matching the position of the virtual object with the position of the real

world image needs to be performed in advance for the virtual reality system. In this module,

registration is carried out by positioning the camera in the virtual space according to the ownship

position (latitude and longitude) and attitude (pitch and roll) data obtained in real time through the

GPS and AHRS sensors installed on the ship.

(e) Simulator Integration Module

A ship-handling simulator was used in this study in order to assess the applicability of the

implemented augmented reality-based navigation support service and system for various maritime

environments. The simulator integration module virtually generated a camera image and sensor

data, and the generation period and types of data used were identical to the actual ship

environment.

IV. Experiments

In this study, a prototype of the AR navigational aid system was manufactured based on the

implemented technology. Using the prototype, simulated and real ship application experiments

were performed. For each experiment, each function of the prototype was tested for normal

operations, application feasibility to the bridge environment was evaluated, and usability as

navigation equipment was assessed. Table 2 shows the specifications of the prototype used in the

experiment.

Table 2: Specification of Prototype system

Specification

PC system

CPU Intel i7 3.0GHz

RAM 16GB

GPU NVIDIA GTX750

Operating System Windows 7 pro 64bit

Monitor LCD monitor, 23 inch

PTZ Camera AXIS P5534E (image resolution : 1280×720 pixels)

4.1. Experiment Conditions The subjects of all the experiments were experienced ship officers. After being sufficiently

informed of the system instructions before the experiments were carried out, the navigator

subjects were instructed to use the system for port entry and departure situations. After the

experiment was finished, opinions on the usability of the system and additional requirements were

collected through surveys and interviews. For the simulator experiment, we utilized an FMB (Full

Mission Bridge) class ship-handling simulator system from the Korea Research Institute of Ships

and Ocean Engineering (KRISO). In order to make the experiments as realistic as possible, the

experiment scenario was designed using marine traffic analysis data from Busan North Port (Safe

Tech Research, 2013). For the experiment at open sea, the prototype system was installed on the


28

training ship, “Saenuri” of Mokpo National Maritime University, and the experiment was carried

out using entry and departure conditions at Mokpo Port. The detailed experiment conditions are

shown in Table 3, and the experiment environment is shown in Figure 4.

Figure 4: Experiments for usability test (a) AR navigation system with ship-handling simulator system, (b) Saenuri at Mokpo harbour,

(c) prototype of AR navigation system

Table 3: Experiments condition

Simulation test Sea test

target harbor Busan port entry Mokpo port entry

ownship Tanker SAENURI

traffic-ships 15 vessels > 30 vessels

weather dense fog light fog

# of subjects 12 5

4.2. Experiment Results The experiment included the process of identifying problems during the operation of the

prototype system along with each function. Experiment results showed that the hardware and

software systems operated in a stable manner in the simulator and real ship environment, and the

average augmented image refresh rate, which determines the performance of the entire system,

was measured to be above 30Hz. In addition, the data manager module processed all the received

data without data loss. Figure 5 shows the user screens of the simulator and real ship experiment.

The ownship information was displayed as a graph in the upper portion of the screen by

integrating the navigation data as shown in Figure 6 (a), and the azimuth circle data shown below

the graph operated according to the bearing of the ship in operation. The traffic ship information

was directly displayed on the location of the corresponding ship by integrating the RADAR and

AIS data as shown in Figure 6 (b). This feature is more intuitive than existing navigation

equipment in determining information of traffic ships; however, there were cases when the

overlay on the position of the other ships actually made recognition difficult. The ECDIS data was

displayed the ENC symbols as shown in Figure 6 (c). Since the same size symbols were used, it

was found that there were difficulties in identifying its distance. Meanwhile, the bearing and

distance information to the next waypoint calculated from the route plan data were displayed

according to the ownship information.


29

Figure 5: Screenshot of Experiments (a) simulation test, (b) sea test

Figure 6: Results of experiment (a) ownship information, (b) trafficship information, (c) ECDIS information

4.3. Analysis of User Survey After the simulator experiment, a survey was carried out on the officers with regard to the

satisfaction level for each function of the augmented reality system. The survey questions were

composed of 5 scale items and miscellaneous opinion items.

Analysis of the survey responses reveals that levels of satisfaction regarding the ownship

information was highest in items asking the satisfaction level for each function, as shown in

Figure 7, while the level of satisfaction of the traffic ship information was relatively low.

Additionally, the level of satisfaction for the entire system showed high points for efficiency and

effectiveness, while legibility and clarity obtained lower than average points. The relatively low

level of satisfaction for legibility and clarity was thought to be due to the difficulty in

distinguishing information, as the information overlapped in multitudes when other ships were

concentrated near the horizon during port entry and departure.

The officers who participated in the experiment were interviewed. They gave mostly positive

feedback with regard to the effectiveness of the augmented reality navigation equipment; however,

they also expressed concern with regard to the additional workload necessary to use the new

navigational equipment. Additionally, for the information display method in the augmented reality

image, responses varied depending on the inclination or preference of the respondent. It is evident

from the data that development of user interfaces optimized for each user is necessary.


30

(a) (b)

Figure 7: Survey results of AR navigation system (a) satisfaction of functions, (b) satisfaction of AR system

V. Conclusion

Although various types of navigation equipment with advanced technology are available to

help enhance operation safety for navigators, further research on the method of efficiently

displaying and servicing information rather than indiscriminately providing a bounty of

information is necessary. In this regard, this paper proposed a navigation aid system based on

augmented reality technology, and the proposed system provides various overlaid navigation

information on images from cameras to support swift and accurate decision-making by officers.

The ship-handling simulation and real vessel application experiments were carried out to verify

the functionality of the system. Through our experiments, stable operation of the prototype

hardware and software in a real ship environment was verified. In a survey conducted after the

experiment, most participants gave positive feedback with regard to the augmented reality

navigation aid system but expressed concern with the increase in workload from the new

navigation equipment. There were also cases when registration errors occurred during augmented

image registration due to location errors from the GPS, and elements that need improvement were

identified, including difficulty in distinguishing data due to excessive amount of information

displayed on the horizon.

As research and development continue, various navigation aid systems based on transparent

displays will be further developed and utilized in the bridge to maximize the effect of augmented

reality technologies. Such use of augmented reality technology is expected to prevent marine

accidents and increase ship operation efficiency in the future.

Submitted: March 8, 2016 Accepted: June 25, 2016

VI. Acknowledgements

The Contents of this paper are the results of the research project of Ministry of trade, industry

& energy (10041790, Development of Advanced Ship Navigation Supporting System based on

Oncoming International Marine Data Standard).


31

References

B&G (2015), http://www.bandg.com Jeong, J. S. (2012), “A Study on the User Requirements for the Support of a Safe Navigation of the

Incoming Vessels to Port Waterways and the Outgoing Vessels”, Research report of Korea Research Institute of Ships and Ocean Engineering, pp. 73-83.

Jung M. (2015), “Analysis of User Requirement for the Improvement of ECDIS to Enhance Navigational Safety and Work Efficiency”, Journal of Navigation and Port Research, Vol. 39, No. 3, pp. 141-147.

Jung M. (2016), “User Requirement Analysis of ECDIS for the Development on S-Mode Guideline”, Journal of Navigation and Port Research, Vol. 40, No. 3, pp. 89-95.

Kim K. H. (2008), “Application of Head-Up-Display Technology to Telematics”, Electronics and Telecommunications Trends, Vol. 23, No. 1, pp. 153-162.

Mads Bentzen (2015), “ACCSEAS Final Report” Ministry of Oceans and Fisheries (2015), “Causal analysis of marine accident”, http://www.mof.go.kr Olivier Hugues (2010), “An Experimental Augmented Reality Platform for Assisted Maritime

Navigation”, ACM. Virtual Reality International Conference, pp. 87-92. SafeTechResearch (2013), “Maritime Safety Audit for New Marina Terminal of Busan North Port”

There is no conflict of interest for all authors.

Available online at ScienceDirecte-navigation.kr/down/2-Jaeyong OH.pdf · 2020-01-22 · Available...

Documents

Transcript of Available online at ScienceDirecte-navigation.kr/down/2-Jaeyong OH.pdf · 2020-01-22 · Available...