Download - Augmented Reality with Phones for Way Findingcourses.cecs.anu.edu.au/courses/CSPROJECTS/19S1/... · 2.1 Augmented Reality Augmented Reality (AR) is an interactive technology concerned

Augmented Reality with Phones for Way Finding

Zhibo Zhang

U6015337

COMP4560

The Research School of Computer Science, CECS

Australian National University

28 May 2019

The Australian National University | 2

Acknowledgment

Finishing this report takes a long time in coding and evaluation. I am writing to extend

my sincere gratitude to those people who helped me with this project.

Firstly, I have my best wishes to my supervisor, Henry Gardener, and my tutor, Wanqi

Zhao. They took a long time to help me to get familiar with AR environment and

Unity3D. They also provided lots of instructions about project management and report

writing skills. I also want to thank Penny who gives a lot of ideas about how to evaluate

an AR application.

Finally, I want to thank my parents gave me chance to have this great experience in

ANU and encouraged me during the project. Also, I want to say thanks to my friend

Yukun who was willing to be my listener by my side.


Abstract

Nowadays, many kinds of map applications can be found in phones. A map application

helps people to navigate in the real world. With the development of Augmented Reality

(AR), it is possible to combine such applications with three dimensional graphical

overlays on top of a real image. Such AR technology offers the prospect of making the

map applications more convenient to use.

This report is about designing and implementing an application on an Android phone

for wayfinding by implementing Augmented Reality using Unity3D. Current research

about Augmented Reality and wayfinding system are outlined, including how

Augmented Reality technology has developed and what other researchers have done to

combine it with wayfinding problems. An AR phone application prototype on Android

is then implemented, using Unity3D, ARKit, and Mapbox, to find directions to

locations outside and inside of buildings. Based on the AR map prototype, a Heuristic

Evaluation is performed, and its results are used to improve the prototype.


Content

1 Introduction ......................................................................................................... 9

1.1 Outline of the project ................................................................................... 9

1.2 Motivation ................................................................................................... 9

1.3 Structure of the report ................................................................................ 10

2 Background ....................................................................................................... 11

2.1 Augmented Reality .................................................................................... 11

2.2 GPS ............................................................................................................ 13

2.3 AR in wayfinding ...................................................................................... 14

3 Project Plan ....................................................................................................... 18

4 Implementation ................................................................................................. 19

4.1 Development environment ........................................................................ 19

4.2 Import Mapbox-plugin .............................................................................. 20

4.3 Match GPS and initialize the map ............................................................. 21

4.4 Navigation in 2D map................................................................................ 22

4.5 Navigation outside the building ................................................................. 23

4.6 Navigation inside the building ................................................................... 24


4.7 NavMesh .................................................................................................... 25

4.8 User interface design ................................................................................. 28

5 Evaluation ......................................................................................................... 30

5.1 Heuristic Evaluation Principles ................................................................. 30

5.2 Heuristic Evaluation Process ..................................................................... 32

5.3 Improvement based on evaluation ............................................................. 42

6 Limitations and Future Work ............................................................................ 43

6.1 Limitations ................................................................................................. 43

6.2 Future Work ............................................................................................... 44

7 Conclusion ........................................................................................................ 45

8 Bibliography ..................................................................................................... 46

9 Appendix 1 Product Screenshot ........................................................................ 48

10 Appendix 2 Contract ..................................................................................... 57

11 Appendix 3 Evaluation .................................................................................. 61


Figure Content

Figure 1: 2D mobile map and AR map (Google ARMap) ........................................ 10

Figure 2: Span in Milgram's Reality-Virtuality (Milgram et al., 1995) ....................... 12

Figure 3: How Virtual Information Display(Report et al., 2016) ................................ 13

Figure 4: Narzt’s program (Narzt et al., 2006) ............................................................ 15

Figure 5: Narzt's program workflow(Narzt et al., 2006) ............................................. 15

Figure 6: Kim & Jun's Program(Kim and Jun, 2008) .................................................. 16

Figure 7: Kim & Jun program workflow ..................................................................... 17

Figure 8: Mapbox Setup in Unity3D ........................................................................... 21

Figure 9: GeoToWorldPosition Method. ...................................................................... 22

Figure 10: Screenshot from 2D map ............................................................................ 23

Figure 11: Workflow for 2D and AR outdoor map ...................................................... 24

Figure 12: AR indoor map workflow ........................................................................... 25

Figure 13: NavMesh without path rendering ............................................................... 26

Figure 14: NavMesh with rendering path .................................................................... 27

Figure 15: Screenshot of indoor map prototype .......................................................... 29

Figure 16: Task device and materials ........................................................................... 39


Figure 17: Check point when during task .................................................................... 39

Figure 18: Screenshot of the new prototype ................................................................ 42

Figure 19: Screenshot of new prototype with reference check point ........................... 43

Figure 20: Workspace in Unity3D ............................................................................... 48

Figure 23: Original prototype screenshot 1 ................................................................. 49





Figure 28: New prototype screenshot 1 ....................................................................... 54




Table Content

Table 1: Development environment ............................................................................. 20

Table 2: Usability Principles for AR Application (Ko et al., 2013) ............................. 31

Table 3: HCI principles(Dünser et al., 2007) ............................................................... 32

Table 4: Principles used in evaluation ......................................................................... 33

Table 5: Serious problems found from the prototype .................................................. 37

Table 6: Problem list from task-based evaluation ........................................................ 40

Table 7: Feedbacks on principles ................................................................................. 41


1 Introduction

1.1 Outline of the project

The difficulty in finding directions from a 2D-map, and matching these directions to

the real world, has been a big problem in wayfinding for personal navigation. To solve

this problem, a prototype application using Augmented Reality (AR) technology is

presented in this report. Not just outdoor navigation, this application will also support

wayfinding inside of a building. Heuristic Evaluation and expert-based evaluation will

be done based on the prototype.

The prototype application will be designed on the Android system and will be run on a

Google Pixel Phone. Unity3D will be used as the main platform to implement the

application and the Unity3D Mapbox package will be used as a map database.

The prototype application is based on a map of Australian National University campus

and provides navigation around and inside of the Hanna Neumann building (building

145). People can use this prototype to find offices and meeting rooms in an area of the

first and second floor of the Hanna Neumann building.

1.2 Motivation

Nowadays, it is common to use a 2D map for people who use the navigation system.

However, there are always some troubles in finding directions when people use a 2D

map. Besides, hardly any map application provides an indoor map navigation service

when people are more likely to get lost inside a building.

The potential of Augmented Reality to enhance navigation is shown in Figure 1.

Compared with the 2D map shown on the left of this Figure, the “AR map” on the right

side is much easier to use to find a direction. It takes time to find the correct direction


to go in a 2D map navigation system. However, with implementing AR on a map, the

direction is shown obviously, restaurants and shops are marked. In addition to the

“outside” case shown in Figure 1, a 2D map does not support navigation inside of a

building and people are often likely to get lost inside of large buildings because of the

complex building structure. Thus, this project aims to design an AR map application

includes inside and outside building and evaluate how it helps people for wayfinding in

the real world. The challenge in this project will be a low accuracy of GPS service

inside a building. Finding a good solution will be main problem for indoor map.

Figure 1: 2D mobile map and AR map (Google ARMap)

1.3 Structure of the report

There will be five parts to this report. Firstly, related works are outlined: what other

researchers have done about Augmented Reality technology, how they implemented it

on devices and problems in wayfinding will be discussed. The project plan will be

illustrated in the second part; detailed plans will be explained, and this will be an order

of implementation. The third part describes implementation, according to the project

plan; what technologies are used in AR map and how it was implements will be written

here. Evaluation will be discussed in the next part; some Heuristic evaluation and


expert-based evaluation was performed, and their results will be described. Based on

the result of the evaluation, some improvements were made to the prototype. The last

part will be future work; what other techniques can be used to improve the map will be

discussed here.

2 Background

2.1 Augmented Reality

Augmented Reality (AR) is an interactive technology concerned with combining data

from the real world with computer generated data. Typically, objects and information

from the real world are augmented by virtual objects and other information generated

by a computer (Schueffel, 2017). This technology makes it possible to interact with the

virtual world by matching it with the real world directly, rather than simply display

information in the screen, relevant information or objects can be labeled and highlighted,

and this makes interacting more convenient. Augmented Reality is shown in Milgram's

Reality-Virtuality Continuum as a span between the real world and virtual world which

is showed in Figure 2 (Milgram et al., 1995). As shown in this figure, Augmented

Reality is much closer to the real world while Augmented Virtuality is closer to the

virtual world. Virtual information is brought to users' real-world and surroundings,

which enhances the perception of the real world and provides a more convenient way

of life to the user(Carmigniani et al., 2011).


Figure 2: Span in Milgram's Reality-Virtuality (Milgram et al., 1995)

Figure 3 (Report et al., 2016) shows how AR displays information on an object. Three

ways are shown in the figure: (1) Spatial AR projects information onto the real object

by a projector. It shows all information in the real-world. In spatial AR people do not

need to wear or hold a device; simple examples are teaching-used projector and

holographic display. (2) The second way is to display information in a hand-held device

such as a phone, all information is shown in a screen. The real object will be detected

by camera, then the information is processed by a phone or computer and then display

the information in the screen. (3) The third way is to have a head-attached device such

as a Microsoft HoloLens (ref). All information is displayed on the glass lenses of such

a display. In the present project, only the second way will be discussed and used.


Figure 3: How Virtual Information Display(Report et al., 2016)

2.2 GPS

To implement a navigation system, the most important part is to locate device and

update location when the device is moving. To obtain location, Global Positioning

System (GPS) is used. GPS is a system provides position, navigation and timing

services based on space. It uses satellites and ground monitoring stations to provide

signals for people to track, measure and locate(Hegarty, 2017). GPS technology has

been applied in many fields such as vehicles and aircraft navigation and mobile phone

track. In this project, mobile phone is a GPS receiver, location information is obtained

to implement a navigation system.


2.3 AR in wayfinding

Nowadays, AR has been applied in many different navigation systems. A vehicle based

AR navigation system was designed by Narzt et al. (2006); they pointed out the

advantage of AR in navigation is that user can easily perceive virtual information and

the real-world because they are in the same place. They also pointed out some AR

design concepts such like the path line should be in a semi-translucent color to avoid

covering the road.

The workflow of this program is shown in Figure 5. The program uses map data to

calculate a planning route, then uses location information to track vehicles for further

route planning. Finally, applications render an AR path aligned with a real-life scene

captured by a camera with video input. As shown in Figure 4, the AR renderer calculates

information from route, GPS and topography to render a virtual 3D model path, then

matches it with a live view. Finally, a navigation path is shown in the screen of a vehicle

navigation device, augmented reality navigation is implemented.


Figure 4: Narzt’s program (Narzt et al., 2006)

Figure 5: Narzt's program workflow(Narzt et al., 2006)


An indoor AR wayfinding application was designed by Kim and Jun (2008) with using

a head-wear device. Three different positioning ways were introduced, including GPS-

based, Sensor-based and RFID tag-based. They found that while GPS-based works not

well and Sensor-based needs extensive wiring, the RFID tag-based was possible to use

well.

In Kim and Jun’s paper, indoor location was calculated by marker detection and image

matching. As shown in Figure 6, their application shows a wayfinding path is quite

different from other AR navigation application. As many AR navigation application use

a line match road, such as the example in Figure 4, while his application uses a small

2D map and arrow to implement wayfinding shown in Figure 7. Furthermore, they use

a marker detection to fix position solve the problem that GPS has low accuracy indoors.

Figure 6: Kim & Jun's Program(Kim and Jun, 2008)


Figure 7: Kim & Jun program workflow


3 Project Plan

The plan for this project was divided into 4 parts:

1. The first part was to become familiar with the Unity3D platform and to match

locations from the virtual world to real-world. Unity3D is an excellent platform

to develop an AR environment and is widely used by researchers in AR. It

includes many AR Application Program Interfaces (APIs) such as Mapbox and

ARCore from Google and Vuforia respectively. This stage was thought to take

2 to 3 weeks.

2. In order to develop a map, matching the user’s present location is important.

Stage 2 of the project combined an outdoor 2-D map with an AR map. Although

this project focuses on an AR map, when the user chooses where they want to

go, an AR map by itself will not provide a specific location, so a 2D map is

necessary for the user to confirm their destination. This stage was thought to

take 4 to 5 weeks.

3. Stage 3 of the project involved the construction of an indoor AR map. This map

provides navigation inside of a building. Because of the complexities of a

building structure, it is sometimes difficult to reach a specific room or reception.

This part of the project implemented navigation from the ground in front of a

school building on the ANU Campus to an office inside the building. This stage

was thought to take 4 to 5 weeks

4. The final stage of the project was evaluation and improvement based on the

results of that evaluation. This stage was thought to take 3 to 4 weeks.


4 Implementation

This section introduces the implementation of this project, according to its project plan

and describes the technology and tools. Unity3D is a cross-platform game engine

released in June 2005 developed by Unity3D Technologies(Contributors, 2019). Three-

dimensional and two-dimensional games can be created by Unity3D, and by using an

API called ARCore develop by Google(Amadeo, 2017). AR can also be implemented

in Unity3D. The Unity3D API Mapbox is used in wayfinding in this project, Mapbox

provides custom online maps data(The New Cartographers, 2013). In this project the

map prototype is design based on the frame in Mapbox named ‘WorldScaleAR’.

Additional function will be added to this frame for an outdoor map. As for indoor map,

the indoor map will be rendered manually, navigation will be implemented by using

NavMesh in Unity3D. Different map will use different camera in Unity3D as different

camera will have different layer view levels.

4.1 Development environment

All the development environment and software version are listed in Table 1 below.


Name Version Note

Unity3D 2018.2f or

later

Game Engine, support Google AR

Core, main development platform in

this project

Unity Mapbox Plug-in v.1.4.5 or later Map database, contains Google AR

Core, contains function to get device

GPS location, contains function to

transfer GPS location to coordinate in

Unity

OS Windows 10 Main operation system

Android SDK 24 or later Support AR Core

Android System 7.0 or later Support AR Core

Microsoft Visual Studio 2017 Platform to edit code in Unity3D

Google Pixel Phone 3 Device to run and test prototype, it

should support AR Core

Table 1: Development environment

4.2 Import Mapbox-plugin

The Mapbox plug-in (v.1.4.5) is downloaded from Mapbox Unity(Mapbox.com, 2019),

then import the package in Unity. After importing the package, in Mapbox setup menu,

an access token should be input to Mapbox and then obtain permission to get map data.

Access token can be obtained from Mapbox website after registering an

account(Mapbox.com, 2019). As shown in Figure 8, after install Mapbox package and

input the access code obtained from website, a program frame can be setup.


Figure 8: Mapbox Setup in Unity3D

4.3 Match GPS and initialize the map

This AR Map matches location by using the GPS service in Unity3D. It gets GPS

information from a device by using script on GameObject called ‘LocationProvider’.

This script uses build-in class named ‘LocationService’ to obtain GPS information from

device(Unity - Scripting API: LocationService, 2019). After getting GPS information,

a function called ‘GeoToWorldPosition’ will be used. This transfers the GPS location

to Unity3D world location and then initializes map by setting this GPS location as the

world center and then the program gets map data from Unity3D’s Map Box. As shown

in Figure 9, this method takes the latitude and longitude map’s GPS information and a

scale (in meters) as input and returns a 2D vector as the location in the Unity3D world.

This initializes the map at Unity3D’s location refPoint and sets this point with known

longitude and latitude.


Figure 9: GeoToWorldPosition Method.

4.4 Navigation in 2D map

2D map workflow is shown in Figure 11. Firstly, program gets location information

from device by using LocationService in Unity3D. Then the 2D map will be initialized

based on start and end points. The start point will be set default by the location of the

device, and the endpoint can be set by a user using the screen or inputting a GPS

location in the program in advance. Then the location of these points will be converted

from function introduced in 4.3. Then these locations will be used to translate these

points to their location in Unity3D world. After initializing map, the possible path will

be calculated based on map data from Mapbox. Then a navigation line will be drawn

from start point to destination on walkable roads and shown on the screen. The line will

be drawn by the Unity3D in-built function LineRenderer. This function inputs start and

end positions and will draw a line from start to end position. The line type and color

can be set at the LineRenderer setting interface in Unity3D. From Figure 10, a blue line

from device to the destination can be seen. Implementing a 2D map lets people confirm

their destination, because in an AR map only surrounding environment can be seen on

the screen, when users is far away from the destination, it will be hard to confirm the

destination.


Figure 10: Screenshot from 2D map

4.5 Navigation outside the building

In the prototype, AR outdoor navigation will be implemented based on a 2D map. Based

on the 2D map data, building models will be rendered according to their location by

Mapbox. For those building which are new and not contained in Mapbox database, an

API called Points of Interest (POIs) will make it possible for a designer to render a

building or add a tag to building (POI placement | Maps SDK for Unity | Mapbox, 2019).

In the prototype, the Hanna Neumann building (Building 145) at ANU was built by POI

builder in this way. The navigation path will be the same as the navigation line rendered

in the 2D map because it is based on the same map data. Besides, this reduces repeating

calculation and rendering. The workflow of the AR outdoor map is shown in Figure 11.


Figure 11: Workflow for 2D and AR outdoor map

4.6 Navigation inside the building

To fix low accuracy of GPS problem, the application will use button to synchronize

location. It is implemented by moving camera to the check points to match the location

with real world.

In the application, the AR map inside a building is based on the Unity API called

NavMesh. NavMesh can let Unity render and set walkable areas and obstacles and

navigation can be implemented(Unity - Manual: NavMesh building components, 2019).

By rendering the model based on blueprint of the building, an indoor map is formed.

The floor areas are set to be “walkable” and walls are set to be “obstacles”. The

destination location of offices and rooms in building are set with tags. As Unity3D

employs a first-person game-playing paradigm, it is necessary to create a “player” and

to add components called NavMesh Agent for making the player as a start point for this

navigation. To make sure that the path line can be rendered and remaining for

pathfinding, it is necessary to set the player's speed to zero. Then a destination point

can be set by clicking the button. After setting the start point and destination point,

LineRenderer will be used to render the path. For multiple floor navigation, multiple

players will be set for each floor. Direction arrows will be rendered under the player to

make user know the direction they should go. The workflow of AR indoor map is shown

in Figure 12.


Figure 12: AR indoor map workflow

4.7 NavMesh

NavMesh is a singleton Class in Unity3D, it helps to achieve pathfinding and

walkability tests in a 3D space(Unity - Manual: NavMesh building components, 2019).

To build a NavMesh pathfinding system, a walkable plane must be set as a pathfinding

area, then it must be set to static and walkable. Several destination points are set on the

plane. According to the building structure in the real-world, walls are rendered and set

to be static and unwalkable. To set the walkability of objects, Navigation page will be

used, it can be opened by select menu bar- Window -> Navigation. After rendering all

staffs in the map, a player should be set on the plane, and then add a component called

‘NavMeshAgent’ on this player. This makes the player be a start point in NavMesh

system. Before the program be tested, the map should be baked in Unity3D in the bake

bar in Navigation page. The player size can also be set here by setting height and radius

by selecting Agent bar. As shown in Figure 13, after baking the map, there will be a

blue plane as walkable area formed according to the walkable place and player’s size.

As shown in Figure 14, when the program runs, a blue line is rendered from the player

to destination.

To achieve pathfinding, there must be a script adding on the player to set its destination.

Destination can be set by obtain an object’s location on the plane and then set it as

destination with build-in function ‘NavMeshAgent.SetDestination()’. After setting


destination, when the program is initialized, the player will move to the destination on

walkable area. In order to render a pathfinding line, all corner points on a path are gotten

from NavMeshAgent and use LineRenderer, a class in Unity3D for rendering line, to

render line from corner point to corner point. The player’s movement speed should be

set as 0 to remain a complete path line. As a result, there will be a path line on the plane

from player to the destination.

Figure 13: NavMesh without path rendering


Figure 14: NavMesh with rendering path


4.8 User interface design

Controls for the user interface are divided into four parts:

(i) Map type selection,

(ii) Set checkpoint,

(iii) Location debug,

(iv) Set destination.

As shown in Figure 15, controls to set a checkpoint are shown inside the red boundary,

controls to set the destination are shown inside the yellow boundary, the green boundary

contains controls debugging and the blue boundary contains controls to select different

types of map. In the prototype there are three options for map type: ARO for AR map

outdoor, ARI for AR map indoor and Map for 2D map. Buttons are designed by code

and each button is linked to target function, and button location is based on the width

and height of the screen. Buttons in red boundary will enable user to link the present

device location with target points for in the real-world. Buttons in the green boundary

will rotate the screen in order to debug direction. Buttons in the yellow boundary will

set a destination to a target point. As shown in Figure 15, the room “MSI-235” can be

seen, and a blue line as a wayfinding path for navigation.


Figure 15: Screenshot of indoor map prototype


5 Evaluation

Usability principles proposed by Ko et al. (2013), and Human Centered Interaction

(HCI) principles proposed by Dünser et al. (2007) will be described in section 5.1.

Based on these principles, Heuristic Evaluation was designed and conducted to evaluate

the prototype application construction described above in section 5.2. Both design

process and the evaluation will be discussed. Reflecting on the results of Heuristic

Evaluation, several adjustments were made to improve the prototype application, and

these are also discussed in section 5.3.

5.1 Heuristic Evaluation Principles

Ko et al. (2013) categorized several AR usability principles into five groups, which are

User-information, User-cognitive, User-support, User-interaction, User-usage. Each

group has several usability principles making up 22 principles in total as shown in Table

2 below.

Group Name Principle Name Principle Definition

User-information Defaults The application should be easy for a user to use and

provide an interaction interface that includes input

buttons and that shows corresponding output.

Enjoyment Use colourful and interesting interface components,

such as colourful buttons, icons, or text.

Familiarity Use familiar interface layout, as well as familiar icons

and familiar language.

Hierarchy Divide up large information display into different

levels to make it easier for a user to process.

Multi-modality Provide different modalities for information content

by combining visual display with sounds, feelings.

Visibility Properly design graphic features so that they can be

easily understood and recognized.


User-Cognitive Consistency Interface and terms use should be consistent to

prevent confusion.

Learnability Application functions and use should be easy and

effective for the user to learn

Predictability How a user reacts to the interface should be

predictable.

Recognition Necessary information should be provided to avoid

user using short-term memory.

User-Support Error

management

Errors occur while using application should be

prevented where they can be and have ready solutions

to fix when they cannot

Help and

documentation

Proper help should appear to help the user, such as

guidelines or tips

Personalization The user should be able to modify the interface style,

such as changing icon shape or text font and color.

User control The user should feel that they are controlling the

system.

User-Interaction Direct

manipulation

Graphical controls should be directly amenable to

user actions.

Feedback System state and process sequence should be shown

to the user visibly, immediately and consistently.

Low physical

effort

Minimize the effort of operating the application and

the tiredness of users.

Responsiveness React quickly to users’ action.

User-Usage Availability Rapid initialization, previous working state, and

options should remain when re-operating an

application.

Context-based UI design should adapt to a different environment.

Exiting The application should be easy to stop and exit.

Navigation Navigate freely in the application.

Table 2: Usability Principles for AR Application (Ko et al., 2013)

Dünser et al. (2007) also applied other some Human Computer Interaction (HCI)

principles to AR systems that help to evaluate an AR application. These principles

include affordance, reducing cognitive overhead, low physical effort, learnability, user

satisfaction, flexibility in use, responsiveness and feedback and error tolerance. Clearly


some of these principles are quite similar to the principles in Table 2. Those HCI

principles which are different from principles in Table 2 are listed in Table 3.

Principle name Description

Affordance Affordance in an AR system focuses on the subject-object

relationship model. In other words, use the functional shape of

the icon or button to make metaphor and let a user know the

function of the button.

Reducing cognitive

overhead

Reduce unnecessary cognitive features on the interface. Too

much non-automatic cognitive efforts can lead to users’

distraction and inefficiency.

User satisfaction Users’ feeling about interaction is important.

Table 3: HCI principles(Dünser et al., 2007)

5.2 Heuristic Evaluation Process

The principles shown in Table 4 will be used to perform a heuristic evaluation as

follows:

a) Task based evaluation will be undertaken by independent experts and by the

developer. During task-based evaluation, the evaluator will perform some

scripted tasks using the interface and make notes when and where they find

difficulties.

b) A static audit will be performed by examining and interacting with at least two

representative screenshots of the application. Once again, the developer and two

independent experts will undertake this evaluation.


Principle name Description Task-Based or Static-

Audit

Defaults The application should be easy for a user to

use and provide an interaction interface

that includes input buttons and that shows

corresponding output.

Task-based and static

audit

Hierarchy Divide up large information display into

different levels to make it easier for a user

to process.

Static audit

Visibility Properly design graphic features so that

they can be easily understood and

recognized

Static audit

Learnability Application functions and use should be

easy and effective for the user to learn

Task based.

Error

management

Errors occur while using application

should be prevented where they can be and

have ready solutions to fix when they

cannot

Task based and static

audit

Help and

documentation

Proper help should appear to help the user,

such as guidelines or tips

Static audit

User control The user should feel that they are

controlling the system.

Task based

Feedback System state and process sequence should

be shown to the user visibly, immediately

and consistently.

Task based

Responsiveness React quickly to users’ action. Task based

Affordance Affordance in an AR system focuses on the

subject-object relationship model. In other

words, use the functional shape of the icon

or button to make metaphor and let a user

know the function of the button.

Static audit

Table 4: Principles used in evaluation


5.2.1 Static audit Heuristic Evaluation

Based on principles in Table 4, principles marked with ‘static audit’ will be explained

and applied in this section, evaluation is based on several screenshots from prototype.

Defaults

Defaults needs application provide interface with input buttons and output display in

order to make it easy for users to use. For an AR wayfinding system, buttons should be

showed in screen and can be easily found by users. The output should be displayed with

matching the real-world and then show it in screen.

By applying this principle in the prototype, as shown in Figure 15, buttons are easily

found in screen and information for pathfinding can be seen matching with the real

building path.

Hierarchy

Hierarchy deals with large number of information shown on screen. An AR wayfinding

application including both the indoor map and the outdoor map will have much

information about locations both inside and outside a building.

To apply this principle, different maps selection is implemented in the prototype.

Information on an indoor and outdoor map is divided into different levels, the user can

select to use indoor or outdoor map. As shown in Figure 15, a selection module is in

the blue frame. By implementing hierarchy, it is much easier for the user to utilize map

and find target information they want.


Visibility

Visibility requires that graphic factors be designed properly, including using

understandable modeling and easily recognizable staffs. AR system shows all

information on the screen matched with the real world, so it is important to make

information understandable for the user. An AR map needs to match virtual information

with the real world, so it is important to render a building or landmark to help people

find the correct information and locate themselves.

By implementing visibility for the outdoor AR map, building models are rendered based

on real building models and an important building is tagged with that building's name.

Because the same model is shown in the device and real world and important building

will be tagged, users can easily find their destination. For the indoor AR map, there is

a flag called ‘synchronize point’ on the wall at each checkpoint for a user to synchronize

location, rooms, and offices are tagged with a name. All tags are visible and

recognizable easily. As shown in Figure 15, room name and navigation path are visible

for the user and it is easily recognized.

Error management

Some errors can be anticipated. AR wayfinding applications work based on GPS

services, and for a short-distance movement, the GPS coordinates might be incorrect as

there is an error associated with them. Besides, when a map renders a building and tags

with its name, this name might appear at the wrong location if the GPS coordinates are

stuck. Thus, implementing error management is important for AR way-finding systems.

Error management is applied by designing function to prevent and fix errors in the

application.

In the prototype, errors are fixed by adjusting the information location. To make sure

that information is at the right location, a rotate function is designed to rotate the camera


in Unity3D world. As shown in Figure 15, buttons in the green frame are used to fix

GPS errors.

Help and documentation

A guideline or instruction should be appeared when user use the application in order to

help user learn and use the application. For an AR system, a guideline should appear

when user open the application to teach user every function in the application.

Instructions should appear when complex function is used.

After applying this principle, there is no features that reflect this principle and this

problem will be listed in next section in Table 5.

Affordance

Affordance matches button’s function with its shape to make metaphor and lets a user

know the function of the button easily. In an AR system, this principle is linked with

button shape. For example, in the famous AR application game Pocket Mongo, when

player find a Pokémon, there is a Pocket ball on the screen and player can click and

move the ball on the Pokémon and then catch it. The pocket ball is linked with its

function.

By applying this principle, there is no features that reflect this principle and problem

will be listed in Table 5.

5.2.2 Problems found from static audit heuristic evaluation

Problems found during evaluation were majorly caused by limited or missed

application of some principles. Both description and reason of the problems and related

principles are listed in Table 5 below.


Problem Description Reason Related

Principles

Limited

information

There is no

information about

which button links

to which target

function

A user may get confused

because they do not know

which button to push to

implement their

requirements. This will

make it difficult for a user

to learn.

Defaults,

Learnability

Useless

buttons

The indoor map

button is showed on

all types of map. In

other maps, it will

not work and not

give feedback.

When a user uses the

outdoor map or 2D map, it

may be confusing because

these buttons do not work,

and the user may use it and

have no feedback.

Hierarchy,

Feedback, User

control,

Affordance

No guidelines

or

instructions

There are no

guidelines and

instructions telling

the user how to use

application

People who use the

application will get

confused about how to use

this map.

Help and

documentation

Limited error

management

No reference object

for a user to

confirm the debug

result.

To debug errors in the

prototype, the user has to

rotate information in the

device to match the real

world, but there is no

reference object to confirm

whether it is correct.

Error

management,

Help, and

documentation

No feedback

from the

system

There is no

feedback after user

push buttons.

A user may get confused

when they push buttons but

get no feedback, even the

prototype has finished the

button function.

Feedback, User

control,

Responsiveness

Table 5: Serious problems found from the prototype


5.2.3 Heuristic task-based evaluation

To do more evaluation, task-based evaluation is designed. Two tasks are designed,

include Task 1: Outside to Inside Task, and Task 2: Inside Multi-floor Task. Evaluation

principles in Table 4 which is marked with ‘Task based’ will be regard as reference

principles for giving feedback. Two task instructions are listed below.

Task-1: Outside to Inside Task

Instructions:

- Go outside as directed

- Switch APP on

- Have Outside AR mode selected

- Find MSI building in graphic display

- Rotate Left/Right to align MSI building graphic with the actual MSI building

- Choose target destination

- Walk into MSI along the blue line and enter MSI building to Synchronize

Point 1

Task-2: Inside Multi-Floors Task

Instructions:

- Follows on from Task 1 at Synchronize Point 1 with App running

- Select Inside AR mode

- Push point button with the same number point on the paper

- Look around use Phone to find the synchronize point in phone

- Use RotateLeft and RotateRight button to make synchronize point in the

phone same location with the synchronize paper in the real-world.

- Select a destination

- Walk to the destination along the blue line up the stairs

- Find next synchronize point

- Push point button with the same point number on it.




- Walk to the destination along the blue line to next synchronize point

- Push point button with the same point number on it.



- Walk to the destination along the blue line to next synchronize point

To do this task evaluation, two experts from CECS in ANU were invited, tasks were

done outside and inside level 1 and level 2 in Building 145 on ANU Campus.

Figure 16: Task device and materials

Figure 17: Check point when during task


After doing the tasks and analyses the results, lots of problems are found. According to

the feedback, serious and significative problems are listed in Table 6.

Problem Description

Menu Button is hard to click, and there is no feedback to tell user’s

finger press has been detected.

Text hard to read It is difficult to read white text without background.

Path not continuous Path flashes when user walk through.

UI text errors UI text has errors in synchronize point and Rotate buttons.

Check point issues Cannot identify checkpoints, distance between device and

checkpoints is confusing for synchronizing.

Table 6: Problem list from task-based evaluation

After analyzing the problems, the menu buttons need to be larger and need feedback

when finger is pressing it. The feedback text should have a background and use a clear

color to make it easy to read. Path flashes because the GPS has low accuracy, so there

should be a function to fix it. Any text errors should be fixed. There is no way to find

and identify check points and there is no reference shows how far users should stand in

front of the check point.

The feedbacks on task-based principles from two experts are also analyzed. The task-

based principles and feedbacks after analyzed are shown in Table 7.


Principle Name Principle Definition COMMENTS

Defaults The application should be easy

for a user to use and provide an

interaction interface that includes

input buttons and that shows


Generally satisfied. But there

is unused button, and output

text is hard to read.

Visibility Properly design graphic features

so that they can be easily

understood and recognized

Clear visibility.

Learnability Application functions and use

should be easy and effective for

the user to learn

Application is difficult to

learn without help for a

person who uses this

application for the first time.

Error

management

Errors occur while using

application should be prevented

where they can be and have ready

solutions to fix when they cannot

No fix error function.



Well user control but it still

needs instructions.

Feedback System state and process

sequence should be shown to the

user visibly, immediately and

consistently.

Application only has

feedback on Mode selection

and Point section. No

feedback on system state.

Responsiveness React quickly to users’ action. Fairly quick.

Table 7: Feedbacks on principles

By analyzing the feedbacks on principles, error management needs more improvement

and feedbacks from system need to be more. Some instructions and guideline should

appear to help user learn how to use the application.


5.3 Improvement based on evaluation

The prototype is improved based on the results from heuristic evaluation. The user

interface is redesigned, references to synchronize check point and more feedbacks are

added. The new prototype is shown in Figure 18 and Figure 19. The buttons are much

larger for easily clicking. When user press button, the phone will vibrate. To make the

text easily read, all texts are in red. New references for check point are shown in Figure

19, the red circular plane is for user to synchronize check point. The path line is changed

to use a semi-transparent color for avoid covering road.

Figure 18: Screenshot of the new prototype


Figure 19: Screenshot of new prototype with reference check point

6 Limitations and Future Work

6.1 Limitations

This report describes the design and evaluation of an AR wayfinding prototype. The

major novel part of this prototype is the implementation of an indoor wayfinding system.

Its major limitations are the GPS accuracy inside of a building, the difficulty of

manually synchronizing device location, and the rendering of the indoor map. With the

present accuracy of GPS position systems, it is difficult to locate the user device in

particular places of a building in real time. In any case, the GPS coordinates do not

account for elevation and cannot distinguish different floors of the one building. This

leads to wrong locations of tags and path in AR system and make it hard to achieve

wayfinding. To cope with these deficiencies of GPS tracking, users have to manually

synchronize at each check point to adjust device locations. Manual synchronizing takes


time to finish and provides a potentially bad experience when using the prototype.

Though many excellent outdoor map databases exist already, indoor maps still need to

be rendered according to a building structure by designers. It is hard to popularize an

indoor map because it takes time and resources to render an accurate indoor map, and

proper rendering needs to use building blueprints. Design of the prototype does not

cover all the principles due to limited time. Task -based evaluation was done by people

who was familiar with the environment.

6.2 Future Work

Future work on prototypes such as the one presented in this report should look towards

implementing a better positioning method for indoor (and outdoor) maps and designing

a good model to render building according to building blueprints.

To achieve a better positioning method, computer vision technology can be used for

recognizing building tags for relocation. This not only reduces manual synchronization

time, but also has a more accurate location than a manually-adjusted location. A better

positioning method is to combine recognition and GPS, when GPS is not accurate. Real

scenes should be input to the program for a camera to recognize and relocate.

To achieve a good model to render an indoor map, there should be an algorithm to

recognize a blueprint image of a building. After recognition, an indoor map will be

rendered automatically according to the results of its input. Important rooms and space

should be marked and set as destination points.

More principles should be covered to design the prototype and have more evaluation

based on these principles. Task-based evaluation should invite more users to do tasks

to give more feedbacks.


7 Conclusion

This project has designed and evaluated a mobile wayfinding application that has been

implemented using AR technology. The application has implemented a simple

navigation system on the campus of ANU. GPS position and tag-based position were

used. The Mapbox and NavMesh packages in Unity3D were used to implement

navigation system. Heuristic Evaluation was designed and applied to the prototype.

Static audit and task-based evaluation was performed by two experts. Following the

evaluation results, problems found in the prototype were corrected to improve the

prototype. Future work in this area will need to design and implement better positioning,

particularly, inside of buildings and it will also need to further improve the user

interface.


8 Bibliography

Amadeo, R. (2017) ‘Google’s ARCore brings augmented reality to millions of Android

devices’, Ars Technica. Available at: https://arstechnica.com/gadgets/2017/08/googles-

arcore-brings-augmented-reality-to-millions-of-android-devices/ (Accessed: 14

October 2018).

Carmigniani, J. et al. (2011) ‘Augmented reality technologies, systems and

applications’, Multimedia Tools and Applications, 51(1), pp. 341–377. doi:

10.1007/s11042-010-0660-6.

Contributors, W. (2019) Unity (game engine), Wikipedia, The Free Encyclopedia.

Available at:

https://en.wikipedia.org/w/index.php?title=Unity_(game_engine)&oldid=892297050

(Accessed: 4 April 2019).

Dünser, A. et al. (2007) ‘Applying HCI Principles to AR Systems Design’, Proceedings

of 2nd International Workshop on Mixed Reality User Interfaces: Specification,

Authoring, Adaptation (MRUI ’07), (March 11), pp. 37–42.

Hegarty, C. J. (2017) ‘The Global Positioning System (GPS)’, Springer Handbook of

Global Navigation Satellite Systems, 50(3), pp. 197–218. doi: 10.1007/978-3-319-

42928-1_7.

Kim, J. and Jun, H. (2008) ‘Vision-Based Location Positioning using Augmented

Reality for Indoor Navigation’, 54(3), pp. 954–962.

Ko, S. M. et al. (2013) ‘Usability Principles for Augmented Reality Applications in a

Smartphone Environment Usability Principles for Augmented Reality Applications in

a Smartphone Environment’, 7318. doi: 10.1080/10447318.2012.722466.


Mapbox (no date) Unity | Mapbox, 2019. Available at: https://www.mapbox.com/unity/

(Accessed: 27 April 2019).

Milgram, P. et al. (1995) ‘<title>Augmented reality: a class of displays on the

reality-virtuality continuum</title&gt’;, (December 1995), pp. 282–292. doi:

10.1117/12.197321.

Narzt, W. et al. (2006) ‘Augmented reality navigation systems’, Universal Access in

the Information Society, 4(3), pp. 177–187. doi: 10.1007/s10209-005-0017-5.

POI placement | Maps SDK for Unity | Mapbox (2019). Available at:

https://docs.mapbox.com/unity/maps/examples/poi-placement/ (Accessed: 17 April

2019).

Report, T. et al. (2016) ‘Augmented Reality : Technologies , Applications , and

Limitations’, (April 2007). doi: 10.13140/RG.2.1.1874.7929.

Schueffel, P. (2017) The Concise Fintech Compendium. Fribourg: School of

Management Fribourg/Switzerland. Available at: http://www.heg-fr.ch/EN/School-of-

Management/Communication-and-

Events/events/Pages/EventViewer.aspx?Event=patrick-schuffel.aspx (Accessed: 14

October 2018).

The New Cartographers (2013). The Washington Post. Available at:

https://www.washingtonpost.com/sf/brand-connect/wp/2013/07/22/the-new-

cartographers (Accessed: 14 October 2018).

Unity - Manual: NavMesh building components (2019). Available at:

https://docs.unity3d.com/2019.1/Documentation/Manual/NavMesh-

BuildingComponents.html (Accessed: 17 April 2019).


Unity - Scripting API: LocationService (2019). Available at:

https://docs.unity3d.com/ScriptReference/LocationService.html (Accessed: 28 May

2019).

9 Appendix 1 Product Screenshot

Figure 20: Workspace in Unity3D


Figure 21: Original prototype screenshot 1


Figure 26: New prototype screenshot 1

10 Appendix 2 Contract

INDEPENDENT STUDY CONTRACT

PROJECTS

Note: Enrolment is subject to approval by the course convenor

SECTION A (Students and Supervisors)

UniID: ___u6015337 _________

SURNAME: ___Zhang_____________ FIRST NAMES: ___Zhibo__________________

PROJECT SUPERVISOR (may be external): ________Henry Gardner___________________________

FORMAL SUPERVISOR (if different, must be an RSSCS academic): ___________________________________

COURSE CODE, TITLE AND UNITS: ________COMP 4560___12 unit_________________________

COMMENCING SEMESTER S1 S2 YEAR: _____ _ Two-semester project (12u courses only): X

PROJECT TITLE:

Augmented Reality with Phones for Way Finding and Help


LEARNING OBJECTIVES:

Students will build an augmented reality application that will run on a modern smartphone. This application will

provide assistance with way finding in a local area of campus or its surrounds. Students will learn how to evaluate

their application and other applications using appropriate data and heuristics. Students will learn how to write a

formal report.

PROJECT DESCRIPTION:

Augmented reality typically involves overlaying graphical objects on top of a live video stream. When effective

geolocation is possible, the graphical objects can be used to provide information about the scene or with reference

to the scene. A popular application is the idea of “way finding” where a user will point a device such as a smart

phone at a scene and the scene will be overlayed with an arrow pointing where they might want to go next. This

project will implement a useful way-finding application for use in CECS, the ANU or the near vicinity.

There are various issues to do with the usability of augmented reality applications that need to be kept in mind

when building applications such as way finding. The project will employ a form of “heuristic evaluation” to

evaluate its effectiveness and suggest improvements. If time allows, the project will be revised following this

evaluation.


ASSESSMENT (as per the project course’s rules web page, with any differences noted below).

Assessed project components: % of mark Due date Evaluated by:

Report: style: ______________45_________________

(e.g. research report, software description...,)

(min 45, def 60)

Ben Swift

Artefact: kind: _____________45________

(e.g. software, user interface, robot...,)

(max 45, def 30) (supervisor)

Presentation :

(10) (course convenor)

MEETING DATES (IF KNOWN):

STUDENT DECLARATION: I agree to fulfil the above defined contract:

Signature Date


SECTION B (Supervisor):

I am willing to supervise and support this project. I have checked the student's academic record

and believe this student can complete the project. I nominate the following examiner, and have obtained

their consent to review the report (via signature below or attached email)

Signature Date: 27/07/2018

Examiner:

Name: Ben Swift Signature

(Nominated examiners may be subject to change on request by the supervisor or course convenor)

REQUIRED DEPARTMENT RESOURCES:

SECTION C (Course convenor approval)

………………………………………………….. ………………………..

Signature Date


11 Appendix 3 Evaluation


TASK BASED HEURISTIC EVALUATION

Completed by Henry Gardner on Thursday 2 May, 2019

TASK 1 NOTES

Issues

- I did not see the graphical buildings at first.

- I did not know that I needed to rotate to find the MSI building

- What was the “log” button there for?

- I could not read some menu items that had white writing and no background. Perhaps these could be

black writing or black writing with white shading?

- I found the menu items to be hard to click

- Menu items did not change shape or color to tell me that my thumb/finger press had been detected

- I needed to rotate left and right with repeated clicks. Perhaps have a click and hold?

- The location seems to drift. I have it sometimes and then it moves.

- The orange blocks seem to grow and compress. Perhaps this is because the display is actually

drifting.

- Note: it is hard to make notes while I am performing this test. It would be better to record my

thoughts and then play them back later.

TASK 2 NOTES

Issues:

- White text menu items as before.

- White text feedback printed to screen; perhaps the screen has too much information on it?

- Cannot click buttons easily. Particularly when buttons are between other buttons.

- Path is not continuous. It flashes into view as you move.

- Path did not go up the stairs

- Synchronise point 2 was swapped with the graphic for synchronise point 3

- Rotate left and right moved the images the wrong way! When I clicked RotateLeft the image moved

right.

- I found myself wanting to zoom the display repeatedly but this facility does not seem to exist.

- The path can completely disappear. There should be some graphic that references it so that you know

that you need to rotate the images


- I touched the mode menu by mistake with my palm and the app swapped to Outside mode.

- What is Map mode? I did not use it at all. Why is it there?






Generally satisfied. But there are

some issues. Input buttons are

unclear and difficult to press. No

indication that a button has detected

a press. Corresponding output (the

path) can be behind the observer.

There are unused buttons (LOG and

2D Map?)

The inside AR control set seemed to

be cluttered.

The use of white text with no

background was poor when outside.

Make it black or mixed colours (with

shading)?



recognized

Generally OK. I liked the graphic for

the path. It was larger near the screen

and it was clear what direction you

needed to go. The graphics for the

external buildings were confusing;

they did not really match the building

shapes that I could see.

The path can be completely invisible

if it is behind you.



I really needed help to use this

application for the first time. But I

think that it would be easy to

remember for the next time.

Error

management

Errors occur while using application should

be prevented where they can be and have

ready solutions to fix when they cannot

There were several errors. It was not

clear how to fix them.




This was ok and it could be a lot better

with more feedback that buttons had

been pressed. Also help is needed to

locate the path if it is behind you.



and consistently.

What are the system states? I think

that there are several. They are not all

communicated

- Mode – Inside/Outside (OK)

- Point selected (hard to read)

- Path out of range (not shown)

- Destination found (not

shown)

Responsiveness React quickly to users’ action. OK. But we need to have more

feedback.

Any other

observations:


TASK BASED HEURISTIC EVALUATION

Completed by Wanqi Zhao on 3rd May , 2019

TASK 2 NOTES

Issues:

1) Cannot identify the ‘checkpoint’ in the AR scene

2) Directions for synchronising the AR scene with the real-life scene (RotateLeft and RotateRight)

seem to be opposite, which is confusing

3) Not sure how far I should stand away from the synchronize paper to match the distance in the

AR scene

4) Need some notifications when I need to climb upstairs






Buttons are provided for users to

interact with application. The

output is almost graphical, some

confusion may arise if the mismatch

happens.



recognized

Buttons are grouped as function.

The text labels have clear meaning.



Button is very straight ward, and

the graphics (e.g. arrows and lines)

are easy to understand.

Error

management

Errors occur while using application should

be prevented where they can be and have

ready solutions to fix when they cannot

N/A



Kind of. I don’t feel quite

confidence to use without any

instructions.




and consistently.

The system rarely provides

feedback. Some scenarios do need

some feedback such as when users

manage to synchronise the point or

when the user arrive one check

point

Responsiveness React quickly to users’ action. Fairly quick

Any other

observations:

1) Good to have checkpoints as I don’t have to hold the device all the time, but needs to consider that I

am an expert user instead of novice user in MSI navigation

2) AR has a huge advantage in navigation as the nature of the technology enables users know the reality

as they are navigated. So they can avoid obstacles.