Designing Conversational Navigationfor Blind Pedestrians

Jan BalataZdenek MikovecFaculty of Electrical EngineeringCzech Technical University inPrague, Czech Republicbalatjan@fel.cvut.czxmikovec@fel.cvut.cz

Copyright is held by the author/owner(s).CHI’17, May 6–11, 2017, Denver, CO, USA. Workshop on Conversational UX Design.

AbstractIn this workshop, we discuss our long-term research lead-ing towards a design of conversational navigation appli-cation for blind pedestrians. We describe steps we took inorder to capture the structure of dialogs on human-humanconversation example and approach we applied in orderto generate the appropriate knowledge base for the ap-plication. Finally, we discuss technologies used and thequestions their usage raise in the context of evaluation andempirical studies.

Author Keywordsconversational UI; navigation; accessibility; blindness

ACM Classification KeywordsH.5.2 [User Interfaces]: Natural language.

IntroductionThe ability to travel independently to desired destinationsis required for a satisfactory level of quality of life and self-confidence, however, people with a severe visual impair-ment are restricted of this ability and their mobility is andtravel-related activities are significantly reduced [6]. Al-though, visually impaired people undergo special training tolearn specific navigation and orientation skills, techniquesand strategies, 30 % of them never leave their home alonewithout sighted guide.

There is a large number of navigation aids for visually im-paired pedestrians based on various sensors [4], auditorydisplays [7], tactile compass [8], or rely on some kind of po-sitioning system (e.g. GPS) in combination with geograph-ical information system (GIS), like BlindSquare, GoogleMaps, etc.. Unfortunately, navigation aids based on conven-tional GIS systems suffer from an insufficient description ofthe environment for blind pedestrians (missing sidewalks,crossings, slopes, sounds, materials, etc.), and inappropri-ate routing algorithms that do not address specific needsof blind pedestrians (inability to cross large open spaces,dangerous crossings, etc.).

One already existing solution is based on tele-assistancecenter with professional operators, who identify the positionof the visually impaired pedestrians (either by camera [3] orpurely by verbal description [9]) and provide suitable guid-ance verbally. However, the main problem of this solutionsis scalability, in a sense of operation area, as the gatheringof a suitable set of landmarks for particular area often re-quires physical presence of the operator on the spot; andin a sense of operation hours as a key feature of such tele-assistance service is its non-stop availability.

Sounds and interaction using natural language form an es-sential modality enabling visually impaired and blind peopleto interact with the world around them. With the advancesin conversational technologies, we see an opportunity to ex-plore their application in accessibility, particularly navigationof blind pedestrians.

In this position paper, we discuss the process of creating aconversational navigation application for blind travelers.

Figure 1: GIS used to generatecontent provided by theconversational navigationapplication. Dashed lines representstreet network, green pavementnetwork, red pedestrian crossingsand yellow stairs.

Design and Creation ProcessTo understand the structure of the dialog about navigationof blind pedestrians and to know the queries they may have

about the route, we created a conversation model based onexamples of human-human conversation [5]. We explored asituation when the blind navigator (navigator) forms a natu-ral source of suitable landmarks with their descriptions andwith routing strategies optimized for blind travelers (traveler)[2]. During the experiment, in 19 sessions, we observed thenavigator navigating the traveler via phone call in an urbanenvironment, we gathered a set of problems that occurredduring the navigation, and we classified the problems intoactivities performed by the navigator and by the traveler.In the end, we identified the activities of the navigator andthe traveler, their turn-taking, and grounding strategies. Fi-nally, we transcribed and categorized all dialogs, identifiedqueries made by the actors. The dialog between blind navi-gator and the blind traveler looked like this:

Navigator: Well, you’ll go approximately 50 meters. There aresome shop door open along the sidewalk, and afterthat 50 meters, there is different paving - cobblestones.

Traveler: Will I go downhill again?Navigator: Yes, you’ll go downhill and after that 50 meters

there will be also a large door at that place withcobble stones.

...Traveler: Ok, there are some dustbins and big door. Great.Navigator: Well, turn so that you have the door at your back,

and then cross the street.Traveler: Turn my back to the door and cross the street, yes?Navigator: Yes, and keep slightly to the left, there are

sometimes cars parked.

In general, we identified following classes of activities: nav-igator describing the environment, navigator giving naviga-tion instructions, navigator determining traveler’s position,traveler executing navigation instruction, and traveler identi-fying landmarks. These activities were later used as intentsin the dialog system.

Next, we needed to be able to create the content, whichcan be provided by the conversational navigation applica-tion – i.e. create a description of any possible route as simi-lar as possible to those prepared by orientation and mobilityspecialist. By using sophisticated data structures (in ourcase GIS, see Figure 1) and algorithms we addressed theissue of time requirements (the route itinerary is generatedimmediately) and rigidity (user can select whichever originand destination s/he wants) and evaluated the efficiency ofsuch route description in a series of experiments [1]. Thegenerated content may look like this:

You are at the beveled corner of Karlovo square and Odboru streets.Turn to the left and go approximately 100 meters slightly downhillto the corner with Myslikova street. The street bends to the left.Keep the buildings on your right-hand side. The sidewalk is madefrom small paving block.

Figure 2: A screenshot of theaccessible, conversationalnavigation application (read thetext from the bottom).

Figure 3: Artist’s portrayal ofNaviterier entity (credit: Delarock).

The result could lead to long description even for a 100 me-ters long segment of a route, where some information maybe less important in different environment context. Thus wedecided to use conversation system to enable the user tocontrol the level of detail of the route description through adialog with the navigation application.

For that, we needed appropriate technology. We choseIBM Watson Conversation, which is used for identifying in-tents, entities and dialog management. As the content isdifferent for each segment of a route, we separate it fromthe dialog system. Dialog management thus works on anabstract representation of the content properties see Figure4. The dialog may look like this:

Naviterier: Turn to the left and go to the corner.Traveler: How far is it?Naviterier: It is approximately 150 meters.Traveler: What is the shape of that corner?Naviterier: There will be a round corner.

Traveler: Ok, I am here.Naviterier: Ok, I’ll tell you how to continue. Continue straight

and cross the street.Traveler: Tell me about the traffic.Naviterier: It is a crossing with tactile and sound signalization,

with one-way traffic from the right.Traveler: I’m on the other side.Naviterier: Ok, I’ll tell you how to continue. Turn to the left

and go to address Odboru 5....Naviterier: Do you have the buildings on you right-hand side?Traveler: I don’t know.Naviterier: Please stop walking and try to explore your surround-

ings. Do you have the buildings on you right-hand side?Traveler: Yes.Naviterier: Great. You can continue.

From the previous dialog example (and Figure 2), we canidentify the following UX design approaches. The entitycalled Naviterier (see Figure 3) uses informal addressing(e.g. German ”Du”, French ”Tu”) to communicate with theuser. It should induce a feeling of briskness of the navi-gation instructions and makes the whole application morepersonal. We also trained the dialog system on a smalltalk. However, when the user starts small talk, Naviterierresponds sharply to get back to the navigation task, whichshould emphasize the serious matter of navigation. Time totime, Naviterier proactively asks users about their progress,which intends to both get their confirmation of location andto make them feel that Naviterier cares about them andtheir safety. The user can ask for repetition of any Navi-terier’s utterance based on last recognized intent stored asa dialog system context variable. Users can either dictate ortype their queries/responses depending on their preference.

Evaluation DesignIn the experiment design, we discussed the influence ofa rather peculiar user interface on measured variables –whether we evaluate conversation nature of the interface or

the systems behind the interface, e.g. speech to text, nat-ural language understanding, dialog management. Giventhe user group of blind people, who usually struggle usinga touchscreen for text input (which is needed in situationswhen dictation does not work), we decided to use a Wizard-of-Oz method to shield them from the troubles, that conver-sational UI in a touchscreen device can cause. One may,however, argue we could use trigger words to start a voice-only conversation.

As conversational user interfaces have the possibility to userather small devices, we plan to use a Bluetooth speakeras an input/output device in the Wizard-of-Oz experiment.The main interests in our case are how the participants willcustomize the information they get for particular segmentsof the route.

Figure 4: Abstract representationof the dialog content in the dialogmanagement.

Conclusion and Lessons LearnedIn this position paper, we showed the approach we took inorder to design and implement conversational navigationapplication and opened some questions regarding eval-uation of this types of user interfaces. We observed theconversation on a human-human example, and we find thisapproach very useful (yet demanding). Using a dialog man-agement on an abstract level seems to be useful for ouruse-case, where the system has to respond differently tothe same queries based on highly structured environmentcontext. Finally, we discuss the user experience design ap-proaches we followed in creation of the application.

We apply similar approach also with students in our courses,where one of the biggest questions is prototyping of conver-sational UI. Currently, we use flow-diagram where nodeshave prerecorded utterances of the prototyped conversa-tion, we have fairly positive feedback and results from thisapproach.

AcknowledgementsThis research has been supported by the project Navigationof handicapped people funded by grant no. SGS16/236/OHK3/3T/13 (FIS 161 – 1611663C000).

References[1] J Balata, Z Mikovec, P Bures, and E Mulickova. 2016.

Automatically generated landmark-enhanced navi-gation instructions for blind pedestrians. In FedCSIS.IEEE, 1605–1612.

[2] J Balata, Z Mikovec, and I Maly. 2015. Navigationproblems in blind-to-blind pedestrians tele-assistancenavigation. In INTERACT. Springer, 89–109.

[3] M Bujacz, P Baranski, M Moranski, P Strumillo, andA Materka. 2008. Remote guidance for the blind – Aproposed teleassistance system and navigation trials.In HSI. IEEE, 888–892.

[4] J Faria, S Lopes, H Fernandes, P Martins, and J Bar-roso. 2010. Electronic white cane for blind peoplenavigation assistance. In WAC. IEEE, 1–7.

[5] D Frohlich and P Luff. 1990. Applying the technologyof conversation to the technology for conversation.Computers and conversation (1990), 187–220.

[6] RG Golledge, RL Klatzky, and JM Loomis. 1996. Cog-nitive mapping and wayfinding by adults without vision.In The construction of cognitive maps. 215–246.

[7] JM Loomis, RG Golledge, and RL Klatzky. 1998. Nav-igation system for the blind: Auditory display modesand guidance. Presence: Teleoperators and VirtualEnvironments 7, 2 (1998), 193–203.

[8] M Pielot, B Poppinga, W Heuten, and S Boll. 2011. Atactile compass for eyes-free pedestrian navigation. InINTERACT. Springer, 640–656.

[9] J Vystrcil, I Maly, J Balata, and Z Mikovec. 2014. Nav-igation dialog of blind people: recovery from gettinglost. EACL (2014), 58.