E-Valuating Virtual Viewpoints: User, Non-User, and Librarians

Perspectives on Live Chat-Based Reference

Marie L. Radford, Ph.D.Associate Professor, Rutgers, The State University of New Jersey

Lynn Silipigni Connaway, Ph.D.Senior Research Scientist, OCLC

Panel: E-Valuating E-Reference: Transforming Digital Reference through Research and Evaluating

ASIST Annual MeetingOctober 24-29 2008Columbus, OH

$1,103,572 project funded by Institute of Museum and Library ServicesRutgers, The State University of New Jersey OCLC, Online Computer Library Center, Inc.

Four phases:Focus group interviewsAnalysis of 850 QuestionPoint live chat transcriptsOnline surveys

176 VRS librarians184 VRS non-users

137 VRS users Telephone interviews

283 Total

Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives

Critical Incident Technique

Flanagan, 1954 Qualitative technique Focuses on most memorable

event/experience Allows categories or themes to emerge

rather than be imposed

Online Survey CI Questions

Librarians & UsersThink about one experience in which you felt a

chat reference encounter achieved (or did not achieve) a positive result

Non-usersThink about one experience in which you felt you

achieved (did not achieve) a positive result after seeking library reference services in any format

Interpersonal Communication Analysis: Results

Relational FacilitatorsInterpersonal aspects of chat conversation that

have a positive impact on librarian-client interaction & that enhance communication.

Relational BarriersInterpersonal aspects of chat conversation that

have a negative impact on librarian-client interaction & that impede communication.

Relational Theory & Approach to Interpersonal Communication

Every message has dual dimensions – both content & relational

(Watzlawick, Beavin, & Jackson, 1967)

Librarians: Positive Result(CI N=142)

Number % Both Relational & 85 60% Content Primarily Content 54 38% Primarily Relational 3 2%

Librarians: Positive Result(CI N=142)

Relational Themes*

Number % Attitude 69

49% Relationship quality 33 23% Familiarity 3 2%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Librarians: Positive Result(CI N=142)

Content Themes* Number %

Providing information 12085%

Providing instruction 49 35% Demonstrating knowledge 14 10% Convenience/multi- 10 7% tasking/ time or money

saving*The percentages do not total to 100% because each CI can be coded into more than one theme

Librarians: Negative Result (CI N=124)

Number % Primarily Relational 53 43% Primarily Content 40 32% Both Relational & 31 25% Content

Librarians: Negative Result (CI N=124)

Relational Themes* Number %

Attitude 67 54% Relationship quality 28 23% Impact of technology 7 6% Approachability 1 1%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Librarians: Negative Result (CI N=124)

Content Themes* Number %

Lack of information 64 52% Lack of knowledge 15 12% Task unreasonable 1 1%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Users: Positive Result(CI N=129)

Number %

Primarily Content 79 61% Both Relational & 33

26% Content Primarily Relational 17

13%

Users: Positive Result(CI N=129)

Content Themes* Number %

Providing information 91 71% Convenience/multi- 36 28%

tasking/time saving/ money saving Providing instruction 14 11% Demonstrating knowledge 7 5%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Users: Positive Result(CI N=129)

Relational Themes* Number %

Attitude 36 28%

Relationship quality 21 16% Impact of technology 1 1%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Users: Negative Result(CI N=68)

Number % Primarily Content 46 68% Primarily Relational 15 22% Both Relational & 7 10% Content

Users: Negative Result(CI N=68)

Content Themes* Number %

Lack of information 48 71% Lack of knowledge 8 12%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Users: Negative Result(CI N=68)

Relational Themes* Number %

Relationship quality 17 25% Attitude 13

19% Approachability 1 1% Impact of Technology 1

1%*The percentages do not total to 100% because each CI can be coded into more than one theme

Non-users: Positive Result (CI N=154)

Number % Primarily Content 79 51% Both Relational & 48

31% Content Primarily Relational 27

18%

Non-users: Positive Result (CI N=154)

Content Themes* Number %

Providing information 75 49% Providing instruction 35 23% Demonstrating knowledge 21 14% Convenience/multi- 18 12%

tasking/time saving/ money saving

*The percentages do not total to 100% because each CI can be coded into more than one theme

Non-users: Positive Result (CI N=154)

Relational Themes* Number %

Attitude 51 33% Impact of FtF assisting 32 21%

relationship development Relationship quality 25 16% Impact of phone/Email 5 3%

assisting information seeking process Approachability 4 3% Familiarity 1 1%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Non-users: Negative Result (CI N=100)

Number %

Primarily Content 52 52% Primarily Relational 33 33% Both Relational & 15 15% Content

Non-users: Negative Result (CI N=100)

Content Themes* Number %

Information 60 60% Lack of knowledge 24 24% Instruction 9 9% Task unreasonable 4 4%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Non-users: Negative Result (CI N=100)

Relational Themes* Number %

Attitude 47 47%

Relationship quality 24 24% Approachability 3 3% Impact of technology 2 2%

*The percentages do not total to 100% because each CI can be coded into more than one theme

Implications: Librarians

Value Delivery of accurate answers/

information Polite, interested users

Find rude or impatient users disruptive to chat success

Implications: Users & Non-Users

Value Accuracy of answers/information

Delivery of specific content Knowledge of sources & systems Positive attitude Good communication skills

Younger VRS users Impatient & want info delivered quickly - no

fuss Not as concerned as librarians w/ instruction

Recommendations

Provide Specific info Variety of formats Friendly & courteous service Marketing to non-users

User education needed for more realistic expectations

Do not force instruction unless wanted

Future Directions

Online survey results informed 283 telephone interviews Collected more critical incidents Analysis in progress

Future Directions

Write, write, write!

Special Thanks

Rutgers University & OCLC Grant Project TeamProject Managers:

Jocelyn DeAngelis Williams Timothy J. Dickey

Research Assistants:Patrick A. ConferDavid DragosJannica HeinstromVickie KozoMary Anne ReillyLisa Rose-WilesSusanna Sabolsci-BorosAndrea SimzakJulie StrangeJanet Torsney

End Notes

This is an updated version of a presentation given at ALISE 2008

This is one of the outcomes from the projectSeeking Synchronicity: Evaluating Virtual Reference Services from

User, Non-User, and Librarian Perspectives

Funded by IMLS, Rutgers University, & OCLC Online Computer Library Center, Inc.

Slides available at project web site: http://www.oclc.org/research/projects/synchronicity/