Overview for Today’95-’98 My Background HCI 27 . 14 Lecture 1 - Introduction 28 Lecture 1 -...

1

Lecture 1 - Introduction 1

Empirical Research Methods [in Information Science]

IS4800 / CS6350

Prof. Timothy Bickmore


Overview for Today

n  Why we’re here n  Overview of the Course n  Introductions n  Homework

2

4

Empirical Research

n  Research is the process of increasing our knowledge

n  Research methods are the tools and techniques considered valid by a given research community

n  Empirical research involves collection and analysis of data from observation

n  Contrast with analytical research

Epistemology

n  Positivism n  The only authentic knowledge is that gained through

positive affirmation of theories following the strict scientific method.

n  Empiricism n  Observational evidence is indispensable for knowledge

of the world.

n  Rationalism n  Knowledge can only be inferred through sound logical

reasoning and “self evident truths”. 5

3

Our focus: Logical empiricism

n  Combines empiricism with rationalism n  Only scientific, mathematical, and

logical statements are literally meaningful, or have truth values.


What are the roles of research methods in…

Information Science? Health Informatics?


4

8

Roles of Empirical Research in IS/PHI: evaluation

From Nielsen, Usability Engineering


Empirical Research in IS/PHI: other roles

n  Requirements analysis n  Assessing attitudes

n  Any systematic collection and analysis of data to answer a research question

n  Essential for n  Conducting research n  Consuming research

5


Course Overview


Overview of Course Content

n  Methods to help provide objective answers to questions about system n  Usability n  Effectiveness n  Acceptability

n  and Impact on n  Individuals, Work groups, Organizations and

Society

6

12

Overview of Course Goal

n  IS: Prepare you for senior project n  PHI: Prepare you for project course n  CS: Enable HCI evaluation

n  Very hands-on n  significant amount of fieldwork

n  Lots of practice applying methods n  Review of applied statistics


Overview of Course Organization

n  first half – building a toolbox n  basics of the scientific method, building bottom-up

from a survey of objective measures to the fundamentals of hypothesis testing using relatively simple research designs.

n  second half – applying it n  alternates between team projects encompassing

the design, conduct and presentation of small empirical studies and lectures covering more advanced research designs and statistical methods.

7


Overview of Course Objectives

n  Scientific method. n  Research methods used in IS/PHI/HCI. n  Identify research questions which are answerable using

empirical methods. n  Research models, including measures, hypotheses, and

statistical tests. n  Fieldwork to collect data using a range of techniques. n  Descriptive statistics. n  Inferential statistics. n  Document and present results from empirical studies. n  Ethical issues in human subjects studies.

15

Course Materials

8

Optional


The R Project for Statistical Computing


9


Administrivia n  Course web site: www.ccs.neu.edu/course/is4800sp18 n  Instructor: Timothy Bickmore, n  [email protected] n  Course-wide distribution list (grads & ugrads)

[email protected] n  Grads only distribution list

[email protected]


Homework

n  Email to [email protected] by noon on due date.

n  Late = automatic full grade lowering

10


Quizzes

n  At start of most classes. n  Closed book, 10 minutes (unless

otherwise noted) n  Covers readings assigned for that class.


Grades (IS4800) n  Quizzes (10%). n  Class participation (10%), including in-class

presentations. n  Individual homework (20% divided equally

among assignments). n  Team projects (20%, consisting of 15%

project grade from the instructor and 5% peer evaluation).

n  Midterm exam (20%). n  Final exam (20%).

11



Rough course outline

Wk 1 Scientific Method, Literature Wk 2 Human Subjects, Ethnography Wk 3-5 Research Models, Measures Wk 6-n Hypothesis testing, Exp designs, Miscellany

12

CCIS Policies

n  No makeup or alternate exams n  Midterm (3/2) and Final (4/20-27)

n  No cheating n  If there is ANY suspicion you WILL be

referred to NU and CCIS ethics committees



Introductions

13


’95-’98

My Background HCI

27

14



Affective Computing

15


My Current Research

DOM Movie or FT video


16


Dialogue Systems for Longitudinal Health Counseling

Timothy Bickmore Assistant Professor

College of Computer and Information Science Northeastern University, Boston

17

relationalagents.com


Empirical Methods in Conversational Agent Research

18


Posture Shifts

Monologues (0.06/s)

Dialogues (0.07/s)

ps/s

ps/int

energy

ps/s

ps/int

energy

Inter-dseg

0.340

0.837

0.832

0.332

0.533

0.844

intra-dseg

0.039

0.701

0.053

0.723

Posture shifts with respect to discourse segment


Handheld ECAs

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Gesture Gaze Nods Postures Brows

HumanHECA

19


Intervention Efficacy Study: MIT FitTrack


MIT FitTrack Evaluation Study Objective

n  Determine if

n  Agent can build a working alliance. n  This translates into gains in behavior change.

n  Behavior change objective n  “30 minutes or more of moderate or better activity on most,

if not all, days of the week” (recommend walking) n  Secondary goal: 10,000 steps a day

n  Between Subjects Design: n  RELATIONAL – relational agent n  NON-RELATIONAL – relational behaviors ablated n  CONTROL – no agent

20


Study Design

Baseline Intervention Two Week

Break Followup Deb

rief

CON- TROL

NON- REL

REL

Intake 1st Login 2 7 27 29 30

WAI

WAI

WAI

WAI

Single items Farewell

Single items Farewell

Single items

Single items

Relational Measures: Hypotheses: NON-REL < REL


Subjects

986 v isitwebsite

187 scheduledintake meet ings

101 start intervention

91 complete atleast 1st week

82 complete follow-up

46 complete atleast 1st week

41 complete follow-up

Screened out by web site:128 maintainers30 poor health641 decided not to participate

76 no shows or screened outduring intake

10 disqualified:7 did not complete

1st week2 discovered

manipulat ion1 had serious

technical problems

“All” analysisgroup (section 9.3 )

“Sedentary” analysis group (section 9.4 )2 drop out during

intervention7 did not dofollow up

1 drops out duringintervention

4 did not do fo llow up

Under 30 mins/day of moderate activity in baseline (recall) week or in 1st week

986 v isitwebsite986 v isitwebsite

187 scheduledintake meet ings187 scheduledintake meet ings

101 start intervention101 start intervention

91 complete atleast 1st week91 complete atleast 1st week

82 complete follow-up82 complete follow-up

46 complete atleast 1st week46 complete atleast 1st week

41 complete follow-up41 complete follow-up

Screened out by web site:128 maintainers30 poor health641 decided not to participate

76 no shows or screened outduring intake

10 disqualified:7 did not complete

1st week2 discovered

manipulat ion1 had serious

technical problems

“All” analysisgroup (section 9.3 )

“Sedentary” analysis group (section 9.4 )2 drop out during

intervention7 did not dofollow up

1 drops out duringintervention

4 did not do fo llow up

Under 30 mins/day of moderate activity in baseline (recall) week or in 1st week

21


Relational Results

Differences in BOND subscales significant: WK1 p<.05 WK4 p=.007

Working Alliance Inventory

1

2

3

4

5

6

7

COMP WK1

BOND WK1

TASK WK1

GOAL WK1

COMP WK4

BOND WK4

TASK WK4

GOAL WK4

NON-RELRELATIONAL

WEEK 1 WEEK 4


Behavioral Results

Only difference between groups is WK4, CONTROL < AGENT p=.06 Significant increase WK0-WK4 p<.001 Significant decrease WK4-WK6 p<.001 2

2.5

3

3.5

4

4.5

5

5.5

6

6.5

7

MIN30_0 MIN30_1 MIN30_2 MIN30_3 MIN30_4 MIN30_6

CONTNONRELREL

Days per week over 30 minute goal

22


Participation Results All Subjects

Significant difference in educational pages viewed: CONTROL < AGENT p<.05

PAGES/SESSION

1

1.05

1.1

1.15

1.2

1.25

1.3

CONTROL NON-RELATIONAL RELATIONAL

Pages viewed per session


Qualitative Analyses

n  28 interviews, 78 feedback messages

23


Qualitative Analyses Repetitiveness

n  Most frequent complaint

The first couple of days I was impressed by it. But, there didn't seem to be a lot of variety going on after that, so it kind of lost my interest, it lost the engagement factor. Maybe, six or seven days into the study I could almost predict what she was going to say, and once the engagement was lost you sort of lose the power of the animated instructor. ... (NON-RELATIONAL) Like 15 days into the study when I could almost predict what she was going to say, it became easier to do things like check my mail in between her responses. … Even with just little bits of variety your mind doesn't shut off. (NON-RELATIONAL) In the beginning I was extremely motivated to do whatever Laura asked of me, because I thought that every response was a new response. Whereas, towards the end I could tell what she was going to say to a couple of my responses. (RELATIONAL)

Design Study

n  How human should our agents be in “serious” application domains?

n  Should they relate human backstories?

24

Experiment n  Between-subjects, longitudinal design n  1ST-PERSON vs. 3RD-PERSON n  H1 (engagement): Participants in the 1st-person condition will use

the system significantly more than those in the 3rd-person condition. n  H2 (engagement): Participants in the 1st-person condition will

report greater enjoyment of the stories and greater engagement with the agent than those in the 3rd-person condition.

n  H3 (deceit): Participants in the 1st-person condition will report greater perceived dishonesty by the agent than those in the 3rd-person condition.

NSF Virtual Laboratory

49

SERVER CLIENT

• Virtual Agent• Dynamic menu input• Web browser

ExperimentDatabase

AgentDatabase

MeasuresDatabase

ExperimentPlanner

ExperimentEvaluator

SessionExecutive

WebServer

DialogEngineExperiment

Specifications

ExperimentResults

25

Manipulation I’d like to tell you some stories

about myself.

I’m not quite sure if I told you about this before.

When my family was living in Falmouth, my parents always had us doing outdoor stuff.

So especially when it was nice out I would go biking or hiking or we would just go for a walk and have a picnic, things like

that.

I’d like to tell you some stories about a friend of mine. She’s an

exercise counselor too.

I’m not quite sure if I told you about this before.

When her family was living in

Falmouth, her parents always had them doing outdoor stuff.

So especially when it was nice out she would go biking or hiking or

they would just go for a walk and have a picnic, things like that.

1ST-PERSON 3RD-PERSON P

roba

bilit

y of

Com

plet

e Se

sssi

on

1st person

3rd person

Results: Engagement

Effects: Condition: p<.05

Day: p<.001

Study Day

26


Introductions

n  Name n  Your background

n  Describe a research study you might have done (or did do) in a past job.


Homework, etc.

n  Read B&A Ch 1 & 3 (67-100) n  Read sample research plan n  Do Homework 1 (not graded)

n  Read through course website n  Find & do homework 1

Overview for Today’95-’98 My Background HCI 27 . 14 Lecture 1 - Introduction 28 Lecture 1 -...

Documents

Transcript of Overview for Today’95-’98 My Background HCI 27 . 14 Lecture 1 - Introduction 28 Lecture 1 -...