INTERACTING WITH AIMODULE 3Session 3, November 10, 2020

Amela Karahasanović, SINTEF and UiO

Module 3Living and working with AI

Objectives

Understanding of challenges related to use of AI infused systems in everyday life and at work

How to evaluate them?

When and how to use them?

What do we know about living and working with them?

2

Module 3Overview

Evaluation of interaction with AI [27th of October]

Human - AI partnership [3rd of November]

Lessons learned from studies of human – AI interaction [10th of November]

3

EXAMPLES

Human – human interaction -> Human - AI interactionHow to create patterns (building blocks)

for human interactions that can be used for human-robot interactions?

Interaction design patterns Interaction blocks - a visual prototyping

environment Evaluating usability of the environment

Sauppe, A., Mutlu, B., Design Patterns for Exploring and Prototyping Human-Robot Interactions, In Proc. of CHI 2014, 1439-1448, ACM

6

Human – human interaction

A lot of studies in linguistic and psychology on behaviour, but design patterns that can be translated in human-robot interactions were missing

16 participants from UoWM (19-62) assigned to 5 different interaction scenarios, randomly assigned into dyads and roles, sessions video recorded, about one-hour long sessions Conversation – discuss their educational experiences and

goals Collaboration – sorting the grocery bags Instructions – assembling some pipeline constructions Interview – a job interview, 14 questions Story telling – retelling a video, 7 minutes video

Human – human interaction

Analysis – coding (one researcher, 10% also by the other researcher to confirm reliability)

Codes for states – important events in interaction (question, answer) and the transition between the states to discover patterns

Identified 7 patterns

Design patterns for exploring and prototyping human-robot interaction

9

• Introductory monologue "Hi Welcome…"

• Questions and answer pair Storyteller: "Do you know who Marvin the Martian is?", Listener: "Oh yeah,…"

• Generic comment and personal comment "Interesting" "I had a similar experience once…"

• Monologue and generic commentStoryteller: "…and then, all of the sudden, thousands of these aliens appear at the Earth." Listener:"That's a lot"

Design patterns for exploring and prototyping human-robot interaction

10

• Instructions and actionInstructor: "Now connect the long pipe with the one shaped like an S" Student: <locates the pipe and connect it>

• Finished commentParticipant: "I think that's it, so we should be done"

• Wait (while the other participant is talking)

11

Implemented and evaluated an authoring environment based on these patterns with interaction designers and developers

13

Your turn Where studies of human-human interaction systems can

be useful? For which kind of systems? Which methods can be used for data collection?

1-2 minutes note for yourself

14

EXAMPLES

Evaluating chatbots

Evaluating and Informing the Design of Chatbots, Jain M. et al. 2018, DIS 2018

Evaluation of 8 chatbots with 16 first-time chatbot users over multiple sessions on the Facebook Messenger platform

Three days interactions with chatbots

Face-to-face semi-structured interview with the participants to elicit their understanding of the chatbots, perceived benefits/limitations, interesting conversations/experiences and areas of improvements

Quantitative data analysis: total interaction time, message count, interactive elements (composition of the chatbots' and participants' messages)

16

Chatbot evaluation

17

Chatbots: a trave agent, entertainment, a shopper, news, weather forecast, chit-chat, social chat, a quiz game

Quantitative data• similar amount of time with each chat-bot, the total

number of messages exchanged significantly different (chit-chat > shopper)

• game and chit-chat bots had more chatbot messages

Qualitative data

Three researchers developed together iteratively a coding schema

4 high-level themes:functionalityconversational intelligence

personality interface

18

Design implications

Chatbot designers

Clarify capabilities at the start and on-demand

Evaluate application-interface match

adding links to external webpages is not recommended; clear added value compare to search engine

Enable dialog efficiency through context resolution

Remain context from earlier conversations, intelligent questions

Consistent personality with small-talk and humor

Design for dialog failure – admitting failure

19

Design implications

Chatbot platform UX designer

Combine text-based interface with buttons and media

participants did not like when the chatbot, for example, opened a news article in a new browser window

Enable efficient input from users

Auto-suggestion buttons

Provide persistent view on chatbot capabilities and context

In the beginning, but also later, keeping the chatbot and the user in the same state of mind

20

Your turn What have you found in you chatbot evaluations? What

would you recommend to chatbot designers?

1-2 minutes note for yourself

21

EXAMPLES

ATM exampleEvaluation of DAC with ATCOs

Karahasanovic et al., 2019, Supporting Air Traffic Controllers During Sector Configuration Changes in Dynamic Airspace Configuration, SID 2019

23

Data collection

Observations

Log files (simulator, UI)

Screen captures

Video recording of screens

Interviews (audio records)

Audio records of the communication between the ATCOs and pilots

Questionnaires

24

Results overview

The overall workload - acceptable and the same or slightly reduced for DAC scenario

Few errors - mostly due to the simplifications of the protype's UI

Overall feedback positive

"Good", "almost perfect", "it enabled us to easily maintain the picture of the traffic"

Solutions for notifying and visualizing both horizontal and vertical changes were useful

Some improvements proposed

25

Results overview cont.

Situation awareness was very good DAC concept potentially useful for supervisors There is a need for a better alignment of the

change with the ATCOs workload

26

MED MIN MAX STDEV

All sessions 8 5 10 1,43

Ref. scenario 8 5 10 1,31

Sol. Scenario 8 5 9 1,56

EXAMPLES

RoboCare

• A system that assists elderly at home• Evaluation

• Video based trials, 8 short movies with different domestic scenarios

• User initiative and system initiative• 100 participants (59 – 90 years)• Questionnaire (background, evaluation)

Cesta et al. Monitoring elderly people with the robocare domestic environment: interaction synthesis and user evaluations, Computational Intelligence, Vol 27, 2011

Findings

High usefulness and acceptability

Useful for personal and environmental safety, reminding to take medications and finding objects

Not useful for providing suggestions

Appreciated help with cognitive problems (reminding them to do something, finding things)

Your turn What have you read/seen (news, articles, videos) about

validations of robots in care of elderly? You can check it quickly now.

4-5 minutes note for yourself

30

EXAMPLES

Can robots manifest a personalityAIBO (robot dog) 48 participants based on the personality scores Randomly assigned to the introvert or extrovert robot(4 conditions) Verbal cues Asked to interact with AIBO (verbally and tactile commands) AIBO respond (extrovert turn faster and wider) Questionnaire on personality, intelligence and attractiveness of AIBO Participant enjoyed the interaction more with a compatible robotLee, K. M., Peng, W., Jin, S., & Yan, C. (2006). Can robots manifest personality? An empirical test of personality recognition, social responses, and social presence in human-robot interaction. Journal of Communication, 56(4), 754-772.

EXAMPLES

Trust in automation An experiment to investigate how the experience of automated

driving change trust in automation and attitudes towards automation 72 participants different ages 19 – 79 (a half under 30) Questionnaire before and after a driving experience in the simulator Gaze recording Highly automated driving on a three line highway, 120 km/h Three take over scenarios, warned 7 seconds in advance, different

traffic densities Older participants more positive Driving increased trust

Trust in Automation – Before and After the Experience of Take-over Scenarios in a Highly Automated Vehicle, Gold et al., AHFE 2015

Module 3

35

The End