Measuring Anonymity in Academic Virtual ReferenceKaren Sobel – University of Colorado Denver & Kristin

Grabarek Roper – Denver Public Library

Introduction

MethodologyThe following methodology can be easily adapted to assess patron group demographics at any library. While Sobel and Roper performed their research at an academic library, public and other libraries providing IM reference service can investigate the patron characteristics that matter most to their work.

1) The researchers selected representative time periods to study (Fall semester 2009 and Fall semester 2010). 2) The researchers created a random sample scheme for selecting transcripts to study. They chose to do this by identifying

peak times of service: 10AM-noon and 2PM-4PM. They then selected the transcript closest to the middle of each of these time periods every day. This created the group of samples from which they drew their conclusions.

3) The researchers identified the demographic characteristics most affecting individual reference transactions at their institution. Most reflected demographics; some data on academic needs was also gathered. Choices were based on their personal expertise and judgment. Characteristics selected were: origin of message (for example, through a chat box embedded on a course page), gender, age group, school (the library serves three institutions of higher education), full-time or part-time enrollment, year in school, and English as a Second Language status. Information on academic needs included the subject area of the question (when applicable), and major category of reference need (such as assistance with a database or basic library information).

4) The researchers wrote detailed criteria for evaluating each characteristic. For example, transcripts identified as coming from a speaker of English as a Second Language might include an explicit explanation “Hi, I am an international student from Japan and I need help with….” It might also include language or spellings pointing to other languages (such as “jaja” instead of “haha,” or syntax pointing to non-native speech.

5) Roper carefully reviewed all the selected transcripts and reviewed each characteristic for every transcript.6) The researchers compiled the data in a spreadsheet using Microsoft Excel. They had decided prior to performing the study

that they were most interested in raw numbers and simple frequencies. (More robust statistical software such as R or SPSS could calculate correlations or evaluate statistical validity for libraries interested in exploring these factors.)

Librarians who provide instant message-based (IM) virtual reference services often perceive that their patrons self-identify to some degree, even when transactions are anonymous. Over time, they begin to form conclusions about their patrons’ characteristics as a group.

Sobel and Roper decided to investigate the realities behind these perceptions. They analyzed two years’ worth of IM reference transcripts to see what demographic information their patrons really provided. This work resulted in two major products:

1) Detailed and concrete information about their IM reference users’ characteristics2) A method for analyzing transcripts to identify demographics of patron groups over time

The researchers believe that other libraries will benefit from using the methodology they have developed to investigate their users’ demographics through identifiers provided during individual conversations.

The following types of data were analyzed:• Origin: Users can initiate an IM reference transaction from a variety of locations on the Auraria Library website.  Transcript

analysis began by determining where the user was on the website when s/he initiated the IM reference transaction. • Gender: While IM reference service users are not required to provide any specific information about themselves before

beginning a transaction, some do choose to state their name.  Often, a name will suggest the gender of the user.  In cases where the gender was reasonably assumed, it was recorded in the analysis.  In cases where a name did not clearly indicate gender, the field was marked as unknown.

• Age:  Because of the diversity of the campus population, researchers thought it might be useful to determine if there is a difference in IM reference service use based on age.  However, age is challenging to determine even in face-to-face reference interviews; attempting to glean information from IM transcripts proved futile and this category was abandoned.

• School:  Auraria Library users may be affiliated with one of three schools: CCD, MSCD, or UCD.  Additionally, Colorado residents may use certain materials at the Auraria Library. The Auraria Library web page and specifically the IM reference service may be accessed from literally any place in the world and used by any person regardless of affiliation; therefore, the researchers proceeded with caution when recording affiliation information.  

• Status:  This category was used to indicate whether the user was a student, an instructor, or an alum.  The term “Other” was included to reflect users who specifically identified as something other than student, instructor, or alum.  Due to the large number of “Other” users who identified as “Parent,” that category was added.  Students, instructors, and alum from institutions other than CCD, MSCD, and UCD were identified as “Other.”

• Year: This category was used to identify graduate students, undergraduate students, instructors, or “not applicable.”  “Not applicable” refers to instances in which the user self-identified specifically as someone other than a student and therefore it was clear that the category of “year” was not applicable. Graduate, undergraduate, or instructor status was determined only if the user specifically identified as one of the three, either by stating so or by asking a question about a class that could easily be identified as a grad or undergrad class.  In all other cases, this field was marked unknown.

• English as a Second Language (ESL): This category was designed to capture the number of ESL IM reference service users through language nuances.  This category was approached with caution as well; it is difficult to determine whether poor grammar on IM results from lack of English language proficiency due to ESL status, to lack of proficiency in a native speaker, or even due to the user employing chat-style conversation techniques which are less grammatically correct. 

• Type of question & Subject area: Roper examined the first question each user asked in order to determine the classification into which that reference transaction fell.  These transactions were coded as one of the following: University information, Library information, Reserves, Databases, Catalog, and Reference. In addition to identifying specific areas of study, other subject areas were included, such as Textbooks, Policy, and Interlibrary Loan.   

Types of Data

These categories were selected because they closely reflected the curiosity of Auraria librarians interested in learning what patron groups use IM reference service, and because they match the categories used in previous research of this type. These categories also reflect the unique nature of the Auraria Campus, where three institutions share the same library resources.• What types of data might your institution code for in a transcript analysis of this nature? • Has your institution undertaken an IM reference service satisfaction survey? If so, how would your transcript analysis

markers compare to those used to compile your survey? • What elements would a transcript analysis capture that might be missed in satisfaction survey results?

Applications

Transcript ExamplesTranscript examples provide insight as to how Sobel and Roper coded the transcripts. Because of the complexity of the category, examples of Types of Questions are presented here.

The researchers examined the first question each user asked in order to determine the classification into which that reference transaction fell.  These transactions were coded as one of the following: University information, Library information, Reserves, Databases, Catalog, and Reference.

University information refers to questions related to general information about one of the three colleges, or about the Auraria Campus: “Can you send me the ISBN number for a book for a course?”  (August 12, 2009)

Library information refers to questions related to the location of materials within the library and library policies, including inter-library loan policies: “How many times can I renew a book I have checked out?”  (August 12, 2009)

Reserves refers to questions regarding materials instructors had placed on reserve for a particular course, or how to place items on reserve for a course: “How do I e-reserve something?” (September 3, 2009)

Databases refers to technical questions regarding database use and how to locate a specific article within a database: “How do you access a database?”  (September 3, 2009)  or “How do I find the "Expanded Academic Index ASAP?" I've been looking under Article Databases & Indexes but can't seem to find it.”  (September 4, 2009)

Catalog refers to questions regarding the availability of a specific resource: “ I am not sure if your copy of "Planet In Peril" (video) is available on your website or if you have a copy to rent out at your library?”  (October 14, 2009)

Reference refers to questions on a particular topic, where the user has not yet selected a resource: “What is Boolean logic?” (October 1, 2009)  Reference also includes reference questions that result in database use instruction: “Hi, I am doing a research paper on disney and trying to show its monopolistic ways I am having trouble researching what would be the best way to do this.”  (November 2, 2009)

(Forthcoming) Roper, Kristin Grabarek and Karen Sobel. “Anonymity versus Perceived Patron Identity in Virtual Reference Transcripts.” Public Services Quarterly8.3(Summer 2012).

Graphic Representation of Types of Questions