Reasoning about natural language phrases for semantic goal drivenexploration

Ben Talbot, Ruth Schulz, Ben Upcroft, and Gordon WyethSchool of Electrical Engineering and Computer Science, Queensland University of Technology, Brisbane, Australia

{b.talbot, ruth.schulz, ben.upcroft, gordon.wyeth}@qut.edu.au ∗

Abstract

This paper presents a symbolic navigation sys-tem that uses spatial language descriptions toinform goal-directed exploration in unfamil-iar office environments. An abstract map iscreated from a collection of natural languagephrases describing the spatial layout of the en-vironment. The spatial representation in theabstract map is controlled by a constraint basedinterpretation of each natural language phrase.In goal-directed exploration of an unseen officeenvironment, the robot links the information inthe abstract map to observed symbolic infor-mation and its grounded world representation.This paper demonstrates the ability of the sys-tem, in both simulated and real-world trials,to efficiently find target rooms in environmentsthat it has never been to previously. In threeunexplored environments, it is shown that onaverage the system travels only 8.42% furtherthan the optimal path when using only naturallanguage phrases to complete navigation tasks.

1 Introduction

The ability to navigate through built environments suchas shopping centres, airports, and university campuses isvital for both humans and robots to be able to functionin these environments. Furthermore, navigation mustbe intelligent even in places that it has never been. Toachieve this, spatial navigation depends on spatial cuesfrom the environment and self motion, computationalmechanisms, and spatial representations [Wolbers andHegarty, 2010].

Humans navigate through built environments withease in most situations, particularly where these environ-ments have been designed to provide navigational cues

∗This research was supported under Australian ResearchCouncil’s Discovery Projects funding scheme (project numberDP140103216)

Figure 1: The abstract map acts as an internal sketchpad for operating with the symbolic spatial information

in natural language phrases. The non-metric spatialrepresentation allows the system to purposely navigate

in places it has never been before.

for people who have never been there before. Muchof the navigation performed in built environments isaided wayfinding, where information is provided throughlabels, signs, maps, route instructions, and planners[Wiener et al., 2009]. When humans have access tosuch information, they typically construct abstract men-tal models of the environment, and do so equally wellwhen the information has been provided in the form ofa map, survey description, or route description of theenvironment [Taylor and Tversky, 1992].

In contrast to human navigation, robots typically useinformation from their sensors (range sensors, cameras,odometers) to construct and use a priori maps of their

environment. In many cases, laborious manual teleoper-ation of the robot is required to build a useful geometricmap before the robot can operate autonomously. Theinformation gathered by these sensors is geometric innature, which is inherently meaningful in a typical robotnavigation system. To efficiently navigate built environ-ments designed with humans in mind, robots should ben-efit from the use of human navigation cues and strategies.A number of robotic systems have been developed thatuse a variety of symbolic spatial information to guidenavigation, including gestures [Bauer et al., 2009], maps[Schulz et al., 2015a], and natural language route instruc-tions [Hemachandra et al., 2015].

However, unlike conventional geometric maps com-monly used by robots, symbolic spatial information istypically devoid of metric meaning. This results in hav-ing difficulties interpreting natural language directionsand signs due to problems understanding natural lan-guage, ambiguity in descriptions, changing frames of ref-erence, and identifying other cues in built environments.Building layouts can also be complex, with branchingcorridors and connecting rooms, often without clear di-rectional signage at choice points.

This paper extends preliminary work on construct-ing abstract maps from spatial descriptions that can beused to guide goal-directed exploration [Schulz et al.,2015b]. Natural language phrases of spatial descriptions(e.g. “down the hall”) are ambiguous, probabilistic, andnon-metric. To convert this to a form that is usableby robots some metricity needs to be introduced. Thesystem described in this paper has abstracted this infor-mation into a point-mass system (see Figure 1) that canbe resolved to determine the likely relationships betweenlocations. Natural language phrases are interpreted asmetric constraints, with places and points of referencegiven metric properties with uncertainties modelled as adynamic spring-damper system.

The remainder of this paper is organised as follows.After reviewing the related literature, section 3 explainsthe approach for a symbolic navigation system that rep-resents, interprets, and reasons about natural languagephrases. In section 4 the experiment configuration ispresented, with the results of simulated and real worldexperiments following in section 5. The paper concludeswith a discussion of demonstrated outcomes and pro-posed future work.

2 Related Work

This section reviews related work on extracting symbolicspatial information from natural language and existingrobot navigation systems that use spatial directions.

2.1 Symbolic spatial information innatural language

There are many forms in which symbolic spatial infor-mation is conveyed, including natural language, routedirections, gestures, signs, and pictorial representations.These forms each convey meaning through a medium ofcommunication and method of encoding spatial informa-tion. A form that allows people to form fairly completemental maps of an environment is a survey or route de-scription in natural langauge [Taylor and Tversky, 1992].A survey or route description is comprised of naturallanguage phrases of spatial information. Each phrasecontains specific elements that combine to provide infor-mation about relationships between places and objects.There are usually two toponyms in a phrase, the fig-ure, which is the space being described, and the refer-ence object, which provides context for understandingthe phrase. Sometimes the phrase implicitly refers to athird toponym providing additional context. The spatialrelationship between the toponyms is described with aspatial preposition [Landau and Jackendoff, 1993]. Theframe of reference [Levinson, 1996] then determines howto interpret the spatial relationship.

2.2 Robot navigation from spatialdirections

Robotic navigation systems that use different formsof symbolic spatial information have been previouslydemonstrated in the literature. Pointing gestures wereused in the autonomous city explorer project to guide amobile robot to a goal in a city [Bauer et al., 2009]. Inprevious work [Schulz et al., 2015a], a robot navigationsystem was presented that used floor plans, a pictorialrepresentation of metric symbolic spatial information, tosuccessfully perform symbolic goal-directed explorationwithin a single floor of an office environment. Otherrobot navigation systems have used natural language inthe form of route directions to navigate to the speci-fied goal [Fasola and Mataric, 2013; Kollar et al., 2010;Hemachandra et al., 2015], typically achieving successby restricting or adding to the existing environment.

Route directions, such as those used in previous robotnavigation systems, describe a single path through anenvironment to a destination. Robots potentially haveaccess to many such directions, or even to a completedescription of the environment from which directions todifferent destinations can be determined. Such descrip-tions would allow a robot to form a more complete men-tal map of the environment.

3 Approach

The symbolic navigation system described in this pa-per aims to purposefully find a goal in an unfamiliarstructured space (typically a large branching corridor)

using only natural language descriptions and informa-tion that is available in the environment. In this sys-tem, the provided semantic knowledge is a structurednatural language description of the environment’s spa-tial layout. Each individual natural language phrase isinterpreted as a set of constraints on the position of thetoponyms used in the phrase (like the constraint shownby the shaded area in Figure 2). The ambiguities, andvarying degrees of metricity in the constraints, are rep-resented by springs acting on masses (the toponyms) ina point-mass system. This is the abstract map’s internalrepresentation of an environment’s spatial layout, withthe minimum energy state representing the navigationsystem’s estimated position for ungrounded symbolic lo-cations. While navigating to a symbolic goal’s predictedlocation, the navigation system receives grounded infor-mation about the state of the world in the form of doorlabel observations. These observations are treated asanchored constraints in the abstract map’s spatial lay-out, consequently changing the minimum energy state tomatch the robot’s grounded perceptions.

3.1 Representing natural language phrases

A defined spatial information structure is used to en-capsulate the components of the symbolic informationcommunicated by natural language phrases. Figure 2shows the four components of the structure as discussedin section 2.1. Place pb represents the reference toponymused in describing the location of the figure pa. The tri-angle at pc provides context for the spatial relationshipdisplayed by the shaded region. This relationship is al-ways a spatial preposition. In this work, the figure, refer-ence object, and context are all assumed to be toponyms.Each toponym either refers to the location of the doorfor a named space, or an arbitrary point of referencewithin the space. For example, pc could be the robot’sarbitrary frame of reference, while pb could be the lifts,and pa the male bathrooms. The key difference is thata point of reference has an implicit direction, whereas aspace only refers to a location in the world.

In this information structure, context is only requiredas part of the phrase for specific spatial relationships.This is dependent on the spatial preposition used (e.g.describing something as left of is impossible to interpretwithout a context location). If no explicit context isprovided by the natural language phrase, it is implicitlyassumed to be the robot’s current position.

3.2 Interpreting natural language phrases

Spatial prepositions are the sole component used by nat-ural language phrases to describe the spatial relationshipbetween places. They typically encapsulate a wide vari-ety of possible metric relationships in a single word. Forexample, the phrases “down the hall”, “down the street”,

pc

sr

sθ

pb pa

Figure 2: Place pa is described as “right of” place pbfrom the perspective of place pc. Radial (sr) and

angular (sθ) springs approximate this impliedconstraint.

and “down the staircase” are each describing vastly dif-ferent metric meanings but are represented with exactlythe same symbolic descriptor. The ensuing interpreta-tion method accepts this metric ambiguity in naturallanguage, while attempting to only explicitly define whatis actually inferred by the language.

The interpretation of a spatial preposition is the pro-cess of trying to parametrically define the space beingreferred to by the preposition. This space is difficult toexplicitly define because it lacks rigid metric definition.For example, drawing a cross somewhere right of a pointis easy, but drawing the boundary for the region wherethese crosses could be drawn is much more challenging.Figure 2 shows an approximate region in grey for thespatial preposition “right of” being used to describe thelocation of pa relative to pb, from the perspective of pc.The spatial relationship “right of” could be describing ametric location anywhere inside this indefinitely expand-ing grey area.

The area being described by a preposition is approxi-mated as a value range along each dimension of the po-lar coordinate system defined by the three toponyms.This range represents the elasticity implied by the spatialpreposition. In Figure 2 for example, the restriction inthe radial direction is completely elastic but rigid aroundπ/2 in the angular direction. This restriction on valueand range is analogous to the restriction imposed by nat-ural length (X) and stiffness coefficient (K) parameters

relation pa pb pc Xr Kr Xθ Kθ

after,beyond,past

f r c 1 0.1 π 1

r f c 1 0.1 0 1

before,toward,towards

f r c 1 0.1 0 1

r f c 1 0.1 π 1

beside,by, near,with

f r c 0.25 1 0 0

r f c 0.25 1 0 0

between

f r#1 r#2 0.25 0.5 0 0.5

f r#2 r#1 0.25 0.5 0 0.5

r#1 f r#2 0.25 0.5 π 0.5

r#2 f r#1 0.25 0.5 π 0.5

down f c cθ 1 0.1 0 1

left of f r c 1 0.1 −π2 1

right of f r c 1 0.1 π2 1

up f c cθ 1 0.1 π 1

Table 1: Parameters for conversion between spatialpreposition and parameters Xr, Kr, Xθ, and Kθ. Theparameters define spring-based constraints along thelocal polar dimensions r and θ. Each local coordinateframe is defined as a pole at pb, with the polar axis

heading towards pc.

in a mechanical spring. Consequently, a parametrisedspring is used along each of the polar dimensions (srand sθ for r and θ respectively) to represent the spacereferred to by a given preposition.

A single constraint is therefore defined by four pa-rameters Xr, Kr, Xθ, and Kθ. The interpretation ofeach spatial preposition is defined as a series of theseparametrised constraints on any one of the three to-ponyms that make up the natural language phrase. Ta-ble 1 shows the interpretation data used for natural lan-guage phrases, based on their preposition. For example,the prepositions “after”, “beyond” and “towards” are

each interpreted as two separate constraints. The firstconstrains the angle of the figure toponym (f), in relationto the reference toponym (r), from the perspective of thecontext (c), to π◦. The second, constrains the angle ofthe reference (r) in relation to the figure (f) as 0◦.

3.3 Constraints as mechanical systems

Each spring-based constraint is modelled as a mechanicalsystem comprising of a spring-damper subsystem actingalong each dimension of the local polar coordinate frame.The three toponyms in the constraint are each treatedas a point-mass. For each constraint, it is assumed thatthere is only one free mass, with the position of the othertwo fixed. The mechanical system representing a con-straint on mass ma, relative to the position of massesmb and mc is shown in Figure 3. Finally, as there are nopreferences to specific spaces or locations in this frame-work, every mass in the system is assumed to be 1kg.

The mechanical system for a constraint is a combina-tion of two spring damper systems; one controlling thedistance (r) from mb to ma, and the other controlling theanti-clockwise angle (θ) of ma relative to the vector frommb to mc. The radial spring-damper is a spring sr withnatural length Xr and spring constant Kr, and a damperdr with damping coefficient Dr. Likewise, the rotationalspring damper (sθ and dθ) is defined by the analogousparameters Xθ, Kθ, and Dθ. By resolving the forces onma, and applying the unit mass assumption, the radial

r

θ

dr

dθ

sr

sθ

ma

mb

mc

Figure 3: Mechanical system used to represent aconstraint on ma, relative to mb, from the perspectiveof mc. A spring-damper is applied to polar dimensions

r and θ (sr dr and sθ dθ respectively.)

and angular accelerations (r and θ respectively) actingon ma are defined as:

r = −θ2r︸ ︷︷ ︸centripetalcomponent

−Kr(r −Xr)−Dr r︸ ︷︷ ︸radial component

(1)

θ =−Kθ(θ −Xθ)−Dθ θ

r2︸ ︷︷ ︸angular component

(2)

To control the frequency response characteristics ofthe system, damping coefficients Dr and Dθ are calcu-lated by approximating the mechanical system as twoindependent second-order systems. This assumption ne-glects the effect of the centripetal component in Equation1. Under the assumption of radial and angular indepen-dence, a slightly underdamped damping ratio ζ (0.75) isused for all dampers based on the corresponding springconstant:

D = 2× 0.75√K (3)

3.4 Reasoning about natural language (thepoint-mass system)

The spatial layout of the point-mass system is the ab-stract map’s internal representation of what the symbolicnavigation system believes the environment looks like.It depicts what the robot can expect to see in its imme-diate surroundings, specifically entries to other spaces.Relaxation of the point-mass system is performed by it-eratively adjusting the position of the masses based onthe energy in each of the mechanical components. Eachiteration comprises of four distinct steps to advance thestate of the system by timestep ∆t:

1. The system starts at time t, with the state of eachmass mi represented by position (x, y) and veloc-ity (x, y) vectors. Each constraint cj is represented

by position (r, θ) and velocity (r, θ) states in alocal polar coordinate frame. The state of each con-straint, describing the location of mass ma in rela-tion to masses mb and mc, is calculated from thecorresponding mass states.

2. Equations 1 and 2 are used to find the radial andangular accelerations (r and θ respectively) in eachconstraint cj .

3. The radial and angular accelerations (r and θ re-spectively) of each constraint cj are cumulativelyapplied to find the acceleration states x and y ofeach individual mass.

4. The state of each mass mi (x, y, x, y) is updatedto apply the change in time ∆t. The point-masssystem is now at time t+ ∆t.

3.5 Incorporating grounded symbolicspatial information

The spatial layout within the abstract map functionsas the culmination of symbolic spatial information fromboth natural language and grounded robot observations.Sections 3.2 and 3.3 showed how the abstract map cre-ates a conceptual layout from ungrounded symbolic spa-tial information. This section shows how the abstractmap uses grounded symbolic spatial information to in-crease the coherence of it’s spatial layout with that ofthe robot’s environment.

Grounded information comes in two different forms -door label observations at a specific location, and thestatus of the robot’s metric navigation goal. Each ofwhich are incorporated into the point-mass system byapplying scaling factors (Osc and Esc respectively) to theradial spring’s natural length (Xr) in each constraint. Asa result of this, Equation 1 is redefined as:

r = −θ2r −Kr(r −Xr ×Osc × Esc)−Dr r (4)

Once a door label is observed by the robot, the po-sition of that toponym is locked in the point-mass sys-tem to the location at which the label was observed.When both toponyms a and b are observed for a con-straint, an observed length can be inferred from the rel-ative positions of ma and mb. Consequently, each ofthe n observed constraint lengths provide a ratio of ob-served length Or to natural length Xr. We define the ob-servation scaling factor Osc as the weighted arithmeticmean of these ratios for all n constraints with an ob-served length:

Osc =

n∑i=1

KrOr

Xr

n∑i=1

Kr

(5)

The second scaling factor is used to handle situa-tions where the robot’s navigation state dictates that thepoint-mass system needs to change its estimate. This isthe case when the robot either reaches the current pre-dicted location for the goal without finding the goal, orhas a predicted location that it is incapable of reach-ing. The function of the exploration scaling factor Escis to expand the search range until more information isobtained. In this way, its function is analogous to a hu-man’s typical first navigation step when their goal is notwhere they expect it to be - they expand their searchscope. The scaling factor Esc is incremented by an ex-ploration step ∆E whenever the robot reaches or abortsa predicted location:

Esc = Esc + ∆E (6)

Algorithm 1 Robot navigation controller

1: while !isAt(Glabel) do2: performRelaxationStep();3: Gnew ← getPredictedPosition(Glabel);4: if euclidDist(Gcurr, Gnew) > D then5: sendGoal(Gnew);6: Gcurr ← Gnew7: else if isDone(Gcurr) or isAborted(Gcurr) then8: Esc ← Esc + ∆E

9: end if10: end while

3.6 Robot navigation controller

The navigation controller links the information in theabstract map to a typical robot navigation system. Thecommunication between these two systems is bidirec-tional. In one direction, the abstract map’s estimatefor the current goal provides a metric goal for the robotnavigation system. Conversely, the information from therobot navigation system provides grounded informationto manipulate the configuration of the point-mass sys-tem in the abstract map.

The robot navigation controller is implemented as de-scribed in Algorithm 1. The robot is given a symbolicgoal Glabel, and iteratively repeats two distinct steps.The first is the refinement of the goal’s estimated posi-tion Gnew. This is done by performing graph relaxationon the point-mass system for a single timestep. Mean-while, incoming door label observations are processed inan asynchronous process. The position the label is ob-served is anchored in the point-mass system, and Osc isrecalculated. These changes are reflected in the relax-ation step. A door label observation also decreases Escby ∆E if it is above 1. This is to control large initialexplore scales that were needed to expand the systeminitially.

Secondly, the metric robot navigation goal is moni-tored and updated as necessary. When the Euclideandistance between the estimated position and current po-sition of the goal Gcurr becomes greater than a goal ad-justing threshold D, Gcurr is updated as Gnew. If at anypoint the robot’s current goal is completed or aborted,the exploration scaling factor is increased as discussedabove. This guarantees that the robot will either findits navigation goal or explore the entire space withoutlocating the goal.

4 Experimental Design

4.1 Environments and testing platforms

Experiments were performed both in simulation and inthe real world. In total, the system was evaluated on asignificant branching corridor in three different environ-

ments (Figure 4). The ‘key floor’ environment was syn-thetically created, with the other two environments gen-erated from floor plans of S Block at QUT Gardens PointCampus.The simulated experiments were performed in asimulator with no noise in mapping or localisation, anda simulated door label detector providing symbol obser-vations.

The real-world evaluation of the system was performedon a GuiaBot from MobileRobots, shown in Figure 1.The robot has four on-board computers all runningRobotic Operating System, ROS. The underlying robotnavigation system uses sensor observations from a laserrangefinder and a mounted Microsoft Kinect RGB depthcamera. Door label observations were obtained througha simulated door label detector that used a ground truthmodel of the environment.

4.2 Experiment

The system was tested in five trials in each of the sim-ulated environments, and the results were confirmed bythree trials in the real-world environment. Note that therobot had no prior map of the simulated or real environ-ments. It built a map as it traversed the area. Prior toeach trial, the robot was provided with a collection ofnatural language descriptions of the environment’s spa-tial layout from the perspective of the robot’s startingposition. The same collection of phrases were providedfor each trial of the same environment, with the numberof phrases ranging from 19 to 64.

From an initial starting position, the robot was seta target room to find using only the natural languagephrases provided and any door label observations it re-ceived. On completion of the symbolic navigation task,the total distance travelled by the robot was recorded.

5 Results

The results demonstrated the effectiveness of the ab-stract map’s spatial layout in effectively and efficientlycompleting symbolic goal-directed exploration tasks. Onaverage, the system travelled only 10.16% further thanthe minimum distance path from a system with an a pri-ori map. In each of the five trials for the three simulatedworlds, the system successfully reached the goal locationby taking the most direct route possible in most cases. Ineach of the three trials in the real world environment, therobot successfully reached the goal with a direct routecontaining minor detours. The underlying metric pathplanning system was responsible for these detours, con-structing some poor plans for local paths as a result ofnoisy sensor readings. In other words, these results weredisturbed by influences external to the symbolic navi-gation system presented in this paper. The navigationsystem, when removing these results, could find a loca-tion in a place it had never been before by travelling only

(a) Key floor (simulated)

(b) S Block, Level 11 (simulated)

(c) S Block, Level 10 (simulated and real)

Figure 4: Environments used in the experiments. Thebranching corridor used in the experiment is shaded,with the robot’s starting position denoted by the redtriangle

EnvironmentGoalLabel

Path Distance (m)

Min. Taken ±%

Key Floor(simulated)

C 20.0 20.6 2.94

F 25.6 26.1 1.91

G 34.7 37.0 6.61

K 27.4 34.8 27.0

I 44.9 44.9 0.00

S Block,Level 11(simulated)

S1124 30.6 30.9 0.97

S1107 24.2 25.6 5.91

S1105 26.8 28.8 7.63

S1128 18.9 25.4 34.3

S1115 24.4 26.3 7.66

S Block,Level 10(simulated)

S1013 23.6 24.0 1.67

S1002 22.3 23.5 5.69

S1036 31.3 33.7 7.70

S1054 27.0 30.5 13.0

S1034 41.1 42.5 3.29

S Block,Level 10(real)

S1016 23.6 25.5 8.15

S1002 22.3 29.1 30.7

S1036 31.3 36.9 17.8

Table 2: Summary of results for the symbolicgoal-directed exploration tasks.

@RP#0

A

B DF

C E

“You are in the Hall”

“You are near Room A”

“Room B is down the Hall”

“Room D is down the Hall”

“Room F is down the Hall”

“Room C is right of Room B”

“Room E is right of Room D”

(a)

@RP#0A

B D F

C E

“Room D is before

Room F”“Room D is after

Room B”“Room F is beyond

Room D”

(b)

@RP#0

AB D F

C E

G

H

K

J

I

“Room G is beyond

Room F”

“Room G is towardsRoom H”

“Room G is left

of Room F”

“Room K is right

of Room F”

“Room J is right

of Room F”

“Room J is betweenRooms K,I”

(c)

@RP#0

A

B D F

C E

GH

K JI

“Room H is besideRoom I”

“Room I is after

Room J”

“Room K isnear Room F”

(d)

@RP#0

ABDFCEGHKJI

(e)

@RP#0A

B D F

C E

GH

K

JI

(f)

@RP#0A B D

F

C E

G H

KJ

I

(g)

@RP#0A B D

F

C E

GH

K

J

I

(h)

Figure 5: Full symbolic goal-directed exploration run. The system creates constraints (blue edges) betweentoponyms (blue nodes) from received spatial language phrases (a - d). The robot starts at a location in the world(red) and tries to reach the system’s estimate of the goal location (red cross). Along the way it produces a metric

map as normal (grey dots), and gathers grounded symbolic spatial information in the form of door labelobservations (green lines). When a toponym is seen, its position is anchored to where it was seen (green squares).

8.42% further than the absolute minimum path. This isa considerable result given that all is required is naturallanguage describing the environment.

5.1 Example Trial - Key Floor (Room ‘I’)

This section provides a detailed description of the ex-perimental result for navigating to room ‘I’ in the simu-lated ‘Key Floor’ environment. The abstract map beginswith no symbolic information describing the world, andno grounded information from the robot’s sensors. Therobot is initially located at a starting location to the farleft of the hallway, facing towards the right.

The system proceeds through the eight stages show inFigure 5:

(a) The system receives an initial set of natural lan-guage phrases and turns the arbitrary point de-scribed as “in the hall” into a reference point de-noted by “@RP#0”. This arbitary point is anchoredto the robot’s current position. The other toponymssettle in clumps due to the lack of interdependenceamongst the phrases.

(b) The system receives more natural language phrasesthat introduce interdependencies within the layout.Visually, this can be seen by the increased connec-tivity of the nodes within the spatial layout.

(c) Symbolic spatial information is added describing theleft and right branches of the key loop. With noconstraints tying nodes ‘H’ and ‘I’ together, the twobranches diverge. Also with no constraint specifyingthe order of ‘I’ and ‘K’ in the right branch, the nodessettle in the wrong order.

(d) Final phrases are added that pull nodes ‘H’ and ‘I’together, while also specifying the order of the rightbranch. These additional constraints result in thespatial layout settling in a configuration with dis-tinct similarities to the ground truth.

(e) The robot begins its goal directed exploration. Withno scale information, the exploration scale factor Escis increased, forcing the system to expand its searchscope.

(f) The robot observes door ‘A’, and as a result observesa constraint length. The resulting observation scal-ing factor Osc expands the spatial layout. The navi-gation controller directs the robot down the hallwayto where it believes ‘I’ is.

(g) The robot observes more door labels as it travelsalong the main hallway, updating Osc along theway. By the time it reaches the loop, it has enoughgrounded information for the spatial layout to cor-rectly direct it to the the right branch.

(h) The navigation system observes the goal locationand successfully completes the goal-directed explo-ration task while taking the optimal path.

6 Conclusions and Future Work

The symbolic navigation system presented in this pa-per provides an architecture to perform efficient sym-bolic goal-directed exploration when provided with nat-ural language phrases. The primary tool of the systemis an elastic spatial layout maintained by the abstractmap. This layout uses a novel spring-based mechanicalmodel to interpret and reason about the spatial informa-tion encoded in natural language phrases. The studiespresented illustrate the effectiveness of the abstract mapin directing a metric navigation system for the comple-tion of goal-directed tasks. The system can use naturallanguage phrases and grounded symbolic information toreach symbolic goals in places that it has never beenbefore by only travelling 8.42% further than the mostefficient path available.

The instantiation of the abstract map presented inthis paper is only capable of handling natural lan-guage phrases describing the spatial layout of a singlespace. This spatial parameterisation for natural lan-guage phrases will become more useful when incorpo-rated in a symbolic navigation system which makes useof the other encoded spatial meanings in symbolic spa-tial information. These include the connectivity and hi-erarchical organisation of spaces. In future work, it isintended to make use of this representation for spatiallayout in a symbolic navigation system that utilises thefull breadth of spatial information available in complet-ing complex symbolic navigation tasks across multiplefloors and buildings.

References

[Bauer et al., 2009] Andrea Bauer, Klaas Klasing, Geor-gios Lidoris, Quirin Mhlbauer, Florian Rohrmller, Ste-fan Sosnowski, Tingting Xu, Kolja Khnlenz, DirkWollherr, and Martin Buss. The autonomous city ex-plorer: Towards natural human-robot interaction inurban environments. International Journal of SocialRobotics, 1(2):127–140, 2009.

[Fasola and Mataric, 2013] Juan Fasola and Maja JMataric. Using semantic fields to model dynamic spa-tial relations in a robot architecture for natural lan-guage instruction of service robots. In InternationalConference on Intelligent Robots and Systems (IROS),pages 143–150. IEEE, 2013.

[Hemachandra et al., 2015] Sachithra Hemachandra,Felix Duvallet, Thomas M Howard, Nicholas Roy,Anthony Stentz, and Matthew R Walter. Learningmodels for following natural language directions

in unknown environments. In IEEE InternationalConference on Robotics and Automation 2015, Wash-ington State Convention Center, Seattle, WA, May2015. IEEE.

[Kollar et al., 2010] Thomas Kollar, Stefanie Tellex,Deb Roy, and Nicholas Roy. Toward understandingnatural language directions. In 5th ACM/IEEE In-ternational Conference on Human-Robot Interaction(HRI), pages 259–266. IEEE, 2010.

[Landau and Jackendoff, 1993] Barbara Landau andRay Jackendoff. “What” and “where” in spatiallanguage and spatial cognition. Behavioral and BrainSciences, 16:217–238, 6 1993.

[Levinson, 1996] Stephen C Levinson. Frames of ref-erence and molyneuxs question: Crosslinguistic evi-dence. Language and space, pages 109–169, 1996.

[Schulz et al., 2015a] Ruth Schulz, Ben Talbot, Oba-diah Lam, Feras Dayoub, Peter Corke, Ben Upcroft,and Gordon Wyeth. Robot navigation using humancues: A robot navigation system for symbolic goal-directed exploration. In IEEE International Confer-ence on Robotics and Automation 2015, WashingtonState Convention Center, Seattle, WA, May 2015.IEEE.

[Schulz et al., 2015b] Ruth Schulz, Ben Talbot, Ben Up-croft, and Gordon Wyeth. Constructing abstractmaps from spatial descriptions for goal-directed explo-ration. In Workshop on Model Learning for Human-Robot Communication at Robotics: Science and Sys-tems 2015, 2015.

[Taylor and Tversky, 1992] Holly A Taylor and BarbaraTversky. Spatial mental models derived from surveyand route descriptions. Journal of Memory and lan-guage, 31(2):261–292, 1992.

[Wiener et al., 2009] Jan M Wiener, Simon J Buchner,and Christoph Holscher. Taxonomy of humanwayfinding tasks: A knowledge-based approach. Spa-tial Cognition & Computation, 9(2):152–165, 2009.

[Wolbers and Hegarty, 2010] Thomas Wolbers andMary Hegarty. What determines our navigationalabilities? Trends in cognitive sciences, 14(3):138–146,2010.