Seeking Synchronicity: Evaluating Virtual

Reference Transcripts

Presented by Lynn Silipigni Connaway

OCLC Members Council February 14, 2006

Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives

• $1,103,572 project funded by:

– Institute of Museum and Library Services $684,996 grant

– Rutgers, The State University of New Jersey and OCLC Online Computer Library Center $405,076 in kind contributions

Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives

Project duration10/1/2005-9/30/2007

Four phases:I. Focus group interviews*II. Analysis of 1,000 QuestionPoint

transcriptsIII. 600 online surveys*IV. 300 telephone interviews*

*Interviews & surveys with VRS users, non-users, & librarians

Phase II:24/7 Transcript Analysis

• Generated random sample– July 7, 2004 through June 27, 2005– 263,673 sessions– 25 transcripts/month = 300 total

• 256 usable transcripts – Excluding system tests and technical problems

5 Analyses

• Geographical Distribution• Type of Library• Type of Questions

• Katz/Kaske Classification • Subject of Questions• Interpersonal Communication

• Radford Classification• Manual and NVivo coding

110, 41%

12, 4%

105, 38%

40, 14%

8, 3%1, 0%

East

Midwest

West

Australia

Canada

England

Geographical Distribution of Question Origination

Type of Library

88

76

51

38

24

12 10

10

10

20

30

40

50

60

70

80

90

100

Consortium Public University National Law Unclear State K-12

6%0%

25%

2%

30%

37%

Holdings

Inappropriate

Procedural

Research

ReadyReference

Specific Search

Type of Questions

Combined 1st and 2nd Questionsn=273 questions

Subjects of Questions64

35

3027

20

1511 11 11

9 9 8

0

10

20

30

40

50

60

70

1

Procedural

Law

General/Multidiscilinary

Science

History

Business

Education

Literature

Biography

Social Science

Recreation

Geography

Combined 1st and 2nd Questionsn=273 questions

Classification Methodology

• Qualitative Analysis

• Development/refinement of category scheme

• Careful reading/analysis

• Identification of patterns

Time intensive, but reveals complexity

Research QuestionsInterpersonal Communication Analysis

• What relational dimensions are present?

• Are there differences in relational dimensions/patterns of chat users & librarians?

• If so, what are they?• How do users & librarians compensate for lack of nonverbal cues?

• What is the relationship between content & relational dimensions in determining quality?

2 Major Themes

• Relational FacilitatorsInterpersonal aspects of the chat conversation that have a positive impact on the librarian-client interaction and that enhance communication.

• Relational BarriersInterpersonal aspects of the chat conversation that have a negative impact on the librarian-client interaction and that impede communication.

Results Interpersonal Communication Analysis

Manual Coding Results

• 200 Transcripts– 177 Usable Transcripts

Librarian Relational Factors (1-200)

Librarian Relational Facilitators:Manual Analysis

132

8674

64

41

0

20

40

60

80

100

120

140

160

180

200

LF 1 RapportBuilding

LF 2 Deference LF 5 ClosingRitual

LF 4 GreetingRitual

LF 3 Rep. OfNon-Verbal

Cues

n=177 transcripts

121102

62

4027

0

20

40

60

80

100

120

140

160

180

200

CF 1 Deference CF 2 RapportBuilding

CF 3 ClosingRitual

CF 4 Rep. OfNon-Verbal

Cues

CF 5 GreetingRitual

Client Relational Facilitators:Manual Analysis

n=177 transcripts

Comparison Relational Facilitators: Manual Analysis

132

86

41

6474

102121

4027

62

0

20

40

60

80

100

120

140

160

180

200

RapportBuilding

Deference Rep. Of NVCues

Greeting Ritual Closing Ritual

Librarians Clients

n=177 transcripts

3425

0

20

40

60

80

100

120

140

160

180

200

LB 1 Relational Disconnect LB 2 Negative Closure

Librarian Relational Barriers: Manual Analysis

n=177 transcripts

78

33

0

20

40

60

80

100

120

140

160

180

200

CB 1 Closing Problems CB 2 Relational Disconnect

Client Relational Barriers: Manual Analysis

n=177 transcripts

34 3325

78

0

20

40

60

80

100

120

140

160

180

200

Librarians Clients

Relational Disconnect Negative Closure

Comparison of Relational Barriers: Manual Analysis

n=177 transcripts

NVivo Coding

• 100 Transcripts– 79 Usable Transcripts

Librarian Relational Facilitators:NVivo Analysis

n=79 transcripts

3832

63

37

26

0

10

20

30

40

50

60

70

LF1 RapportBuilding

LF2 Deference LF3 NonverbalCues

LF4 GreetingRitual

LF5 ClosingRitual

Relational Facilitators

Nu

mb

er

of

Tra

ns

cri

pts

Client Relational Facilitators:NVivo Analysis

n=79 transcripts

51

11 9

56

25

0

10

20

30

40

50

60

70

CF1 Deference CF2 RapportBuilding

CF3 ClosingRitual

CF5 GreetingRitual

CF4 NonverbalCues

Relational Facilitators

Nu

mb

er o

f T

ran

scri

pts

Comparison Relational Facilitators: NVivo Analysis

n=79 transcripts

63

38 3732

26

5156

9 11

25

0

10

20

30

40

50

60

70

RapportBuilding

Deference Nonverbal Cues Greeting Ritual Closing Ritual

Relational Facilitators

Nu

mb

er o

f T

ran

scri

pts

Librarian Client

Librarian Relational Barriers: NVivo Analysis

n=79 transcripts

2122

0

5

10

15

20

25

30

35

40

45

LB2 Negative Closure LB1 Relational Disconnect

Relational Barriers

Nu

mb

er o

f T

ran

scri

pts

Client Relational Barriers: NVivo Analysis

n=79 transcripts

40

10

0

5

10

15

20

25

30

35

40

45

CB1 Closing Problems CB2 Relational Disconnect

Relational Barriers

Nu

mb

er o

f T

ran

scri

pts

Comparison Relational Barriers: NVivo Analysis

n=79 transcripts

22 21

40

10

0

5

10

15

20

25

30

35

40

45

Negative Closure/Closing Problems Relational DisconnectRelational Barriers

Nu

mb

er o

f T

ran

scri

pts

Librarian Client

NVivo Search TotalsWord Total Percent

sorry 20 25.32

thank you 51 64.56

thanks 34 43.04

thank u 3 3.80

thanx 0 0.00

TY 0 0.00

ty 0 0.00

n=79 transcriptsIncludes words in scripts

NVivo Search TotalsPerson Word Total Percent

Librarian sorry 14 17.72

Client sorry 7 8.86

Librarian thank you 36 45.57

Librarian thanks 18 22.78

Client thank you 28 35.44

Client thanks 24 30.38

Client thank u 3 3.80

n=79 transcriptsIncludes words in scripts

Transcript Reading

• Positive VRS experience– Duration = 1 hour 11 minutes– Academic User– Question – Boston drug company - diabetes– Relational Work– Enthusiastic user– Helpful librarian

• Less than positive VRS experience– Duration = 39 minutes– Middle school or high school student– Question – physics – car acceleration– Poor reference work– Extreme negative closure

End Notes

• This is one of the outcomes from the project Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives, Marie L. Radford and Lynn Silipigni Connaway, Co-Principal Investigators.

• Funded by IMLS, Rutgers University and OCLC, Online Computer Library Center.

• Project web site: http://www.oclc.org/research/projects/synchronicity/

Questions

Marie L. Radford, Ph.D.Email: mradford@scils.rutgers.eduwww.scils.rutgers.edu/~mradford

Lynn Silipigni Connaway, Ph.D.Email: connawal@oclc.orgwww.oclc.org/research/staff/connaway.htm