Measuring the Impact of Networked Electronic Resources: Developing an Assessment Infrastructure for...

Measuring the Impact of Networked Electronic Resources:Developing an Assessment Infrastructure for Libraries,

State, and Other Types of Consortia

Presented By:

Terry Plum, Simmons GSLISBrinley Franklin, University of Connecticut

Martha Kyrillidou, ARL Gary Roebuck, ARL

MaShana Davis, ARL

Library Assessment Conference 2008University of Washington

Seattle, WAAugust 4, 2008

MINES for Libraries® Association of Research Librarieswww.minesforlibraries.org Statistics and Measurement Program

MINES for Libraries® Background

1982 First indirect cost study – in-house 1986 Agreement with DHHS 1998 CARL surveying Medline and

FirstSearch 1999 ARL New Measures – Terry Plum 2003 ARL adopts MINES into StatsQUAL®

2005 ARL interactive data with OCUL 2007 ARL in discussion with EZproxy 2008 IMLS grant application


The most popular current method of measuring usage of electronic resources by libraries is not through web-based usage surveys, but through vendor supplied data of library patron usage or transaction usage.

Web-based usage surveys are increasingly relevant in the collection of usage data to make collection development and service decisions, to document evidence of usage by certain patron populations, and to collect and analyze performance outputs.

Brinley Franklin and Terry Plum, “Successful Web Survey Methodologies for Measuring the Impact of Networked Electronic Services (MINES for Libraries®)” IFLA Journal 32 (1) March, 2006

Measuring Digital Content Use


ARL New Measures Toolkit: StatsQUAL®

LibQUAL+® is a rigorously tested Web-

based survey that libraries use to solicit, track, understand, and

act upon users‘ opinions of service

quality.

LibQUAL+®

DigiQUAL®

The DigiQUAL® online survey designed for

users of digital libraries that measures reliability and trustworthiness of

Web sites. DigiQUAL® is an adaptation of

LibQUAL+® in the digital environment.

MINES for Libraries®

Measuring the Impact of Networked Electronic

Resources (MINES) is an online transaction- based survey that

collects data on the purpose of use of

electronic resources and the demographics

of users.

ARL Statistics™

ARL Statistics™ is a series of annual publications that

describe the collections, expenditures, staffing,

and service activities for Association of Research Libraries (ARL) member

libraries.

ClimateQUAL™

ClimateQUAL™: Organizational Climate

and Diversity Assessment is an online

survey that measures staff perceptions about:

(a) the library's commitment to the

principles of diversity, (b) organizational

policies and procedures, and (c) staff attitudes.


What is MINES?

Action Research Historically rooted in indirect cost studies Set of recommendations for research design Set of recommendations for web survey

presentation Set of recommendations for information

architecture in libraries Plan for continual assessment of networked

electronic resources An opportunity to benchmark across libraries


What is MINES? (cont’d)

Measuring the Impact of Networked Electronic Services (MINES)

MINES is a research methodology that measures the usage of networked electronic resources of a library or consortium by a specific category of the patron population.

MINES is a Web-based survey form consisting of 5 questions that is administered at the time of transaction.

MINES measures: User status and discipline/affiliation (who) Physical location (where) Primary purpose and reason of use (why)

MINES a part of Association of Research Libraries’ New Measures & Assessment Initiatives (since 2003).

MINES is different from other electronic resource usage measures that quantify total usage (e.g., Project COUNTER, E-Metrics) or measure how well a library makes electronic resources accessible (LibQUAL+®).

MaShana Davis

Discussion randomization here?


Recent Data Collection Activities via ARL Ontario Council of University Libraries

(OCUL) University of Iowa Libraries University of Macedonia


Questions Addressed How extensively do sponsored researchers use the new

digital information environment?

Are researchers more likely to use networked electronic resources from inside or outside the library?

Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?

What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?

Are particular network configurations more conducive to studies of digital libraries patron use?


Reliance on Vendor Statistics Vendor statistics, while more reliable than in the past, are still maturing.


Library User Survey


Library User SurveyPatron Status


Library User SurveyAffiliation


Library User SurveyLocation


Library User SurveyPurpose


Issues with Web surveys

Research design Coverage error

Unequal access to the Internet Internet users are different than non-users

Response rate Response representativeness

Random sampling and inference Non-respondents

Data security


MINES Strategy A representative sampling plan, including sample size, is

determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.

Random moment/web-based surveys are employed at each site.

Participation is usually mandatory, negating non-respondent bias, and is based on actual use in real-time. IRB waiver or approval

Libraries with database-to-web gateways or proxy re-writers offer a comprehensive networking solution for surveying all networked services users during survey periods.


MINES Strategy Placement

Point of use Not remembered, predicted or critical incident

Usage rather than user What about multiple usages Time out ? Cookie or other mechanism with auto-population or

more recently counting invisibly with a time out. Distinguish patron association with libraries.

For example, medical library v. main library. But what if the resources are purchased across

campus for all. Then how to get patron affiliation?


Web Survey Design Guidelines

Web survey design guidelines that MINES followed: Presentation

Simple text for different browsers – no graphics Different browsers render web pages differently

Few questions per screen or simply few questions Easy to navigate Short and plain No scrolling Clear and encouraging error or warning messages Every question answered in a similar way - consistent

Radio buttons, drop downs ADA compliant Introduction page or paragraph Easy to read

Must see definitions of sponsored research. Can present questions in response to answers – for example if

sponsored research was chosen, could present another survey


Quality Checks Target population is the population frame – surveyed the

patrons who were supposed to be surveyed - except in libraries with outstanding open digital collections.

Check usage against IP. In this case, big numbers may not be good. May be seeing the survey too often.

Alter order of questions and answers, particularly sponsored and instruction.

Spot check IP against self-identified location Spot check undergraduates choosing sponsored research –

measurement error Check self-identified grant information against actual grants Content validity – discussed with librarians and pre-tested. Turn-aways – number who elected not to fill out the survey Library information architecture -- Gateway v. HTML pages –

there is a substantial difference in results.


What is a session?

Session A successful request of an online service. It is one cycle of user activities that typically starts when a user connects to the service or database and ends by terminating activity that is either explicit (by leaving the service through exit or logout) or implicit (timeout due to user inactivity) (NISO)

Counter Code of Practice: Journals and Databases. Release 3. Glossary of Terms. http://www.projectcounter.org/code_practice.html


Federated searches are different

Search and session activity generated by federated search engines and other automated search agents should be categorized differently from regular searches. Any searches or sessions derived from any federated search engine (or similar automated search agent) should be included in separate “Searches_federated” and “Sessions_federated” counts….and are not to be included in the “Searches_run” and “Sessions” counts.

Counter Code of Practice: Journals and Databases. Release 3. Glossary of Terms. http://www.projectcounter.org/code_practice.html


Sample Survey Data File Generated

Other UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://newfirstsearch.oclc.org/done=referer;dbname=WorldCat;autho=100122319;FSIP12:36:5012/3/2004Off CampusUConn Faculty12.101.104.84

Family StudiesInstruction/Education/Departmental (Non-Funded) Researchhttp://www.jstor.org/cgi-bin/jstor/gensearch12:37:4312/3/2004Off CampusUConn Undergraduate Student12.101.104.84Non-UConnOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:08:4112/3/2004Off CampusNon-UConn12.18.36.40

Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html13:31:2912/3/2004Off CampusNon-UConn12.76.131.246

Non-UConnOther Activitieshttp://magic.lib.uconn.edu/index_real.html12:11:0612/3/2004Off CampusNon-UConn130.132.86.156Agriculture & Natural ResourcesInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html12:33:5712/3/2004

Off CampusNon-UConn131.128.89.54EducationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.euromonitor.com/womdas/12:57:4412/3/2004

Off CampusNon-UConn134.241.135.70Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:28:5212/3/2004

Off CampusNon-UConn134.74.42.82Business AdministrationOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:56:4612/3/2004In the LibraryUConn Faculty137.99.1.122

Liberal Arts & SciencesOther Activitieshttp://www.siam.org/journals/simax/simax.htm12:52:1712/3/2004On Campus - StorrsUConn Graduate Student137.99.1.122

EngineeringInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:04:3112/3/2004On Campus - StorrsUConn Graduate Student137.99.1.219

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/pqdweb?RQT=31812:16:3312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.105

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.jstor.org/journals/07322399.html12:16:5212/3/2004On Campus - StorrsUConn Graduate Student137.99.100.105

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:29:5312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:48:4112/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/login?COPT=SU5UPTAmVkVSPTImREJTPTE3MjErMysxNkJD&clientId=4899612:04:2312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175


MINES for Libraries® Implemented via StatsQUAL®

Dataflow Diagram

StatsQUAL®

Library PatronLibrary Administration/

Personnel

ARL Personnel

EZProxy®or other authorization/

access gateway

Resource Providers

- Resource Request- Browser cookie contents

- Requested Resource- Browser Cookie

Requested Resource

Resource Request

- Survey Form- Browser Cookie (session tracking)

- Survey Response- Browser Cookie contents

- Patron redirect- Target /Referring URLs

Survey respondent redirect

Master Survey Parameters

Administrative Interface

- Survey Monitoring Interface- Survey Customization Interface- Formal Written Report- Raw Data Download- Online Analytical Interface

Survey Customization Preferences


MINES for Libraries® Implemented via StatsQUAL®

Cross-Functional Flowchart

MINES for Libraries ® Survey Data Collection Process (proposed)

EZProxyEZProxy StatsQUALStatsQUAL Resource VendorResource VendorUserUser

User makes resource request

(clicks on hyperlink)

Does the requested resource require

proxing?

Send requested document

No

Request document from vendor and return to patron

with NO URL re-writing

User receives document. All

subsequent requests to resource are

unproxied.

End

Subsequent requests

Is the requester authorized?

Yes

Begin

User Authentication

process No

Does an active session exist for this patron? – check for StatsQUAL® placed

browser cookie to determine how to survey user...

YesRedirect to StatsQUAL

User complete MINES survey

Active Session, different target URL from Last. Send shortened survey (request purpose only)

Expired/New Session. Send full survey.

Request document from vendor and

return to patron with URL re-writing

Active Session & Same target URL. No data point recorded.

User receives document. All

subsequent requests to resource are

proxied.

End

StatsQUAL writes responses to database. If purpose only, retrieves and replicates other response fields

from last record for the active session.


Reporting

Streamline data collection Standardized reporting

General overview Frequencies and cross tabulations for

common variables Who is accessing electronic resources? (user

status and affiliation) Where are they accessing electronic resources?

(location) Why are they accessing electronic resources?

(purpose/reason of use)


26%3%

5%

SponsoredResearch

Instruction

Other

Other SponsoredActivities

*72% of sponsored research usage of electronic resources occurred outside the library; 83% took place on campus.

63%

14%2%

21%

11%

62%

25%2%

6%

58%

34%2%

On-Campus, Not in the Libraryn = 9,460

In the Libraryn = 9,733

Off-Campusn = 7,790

Overall Usen = 26,983

66%

MINES for Libraries®

Purpose of Use by LocationU.S. Main Campus Libraries 2003-05

Sponsored Research

Instruction

Other

Other SponsoredActivities


Online Data Analysis

Expand data analysis capabilities Multidimensional data views “Drag and

drop” Cross tabulations Frequencies Longitudinal analysis


Web Interface


Post Survey Consulting

Help institutions: Identify common themes in their data Identify answers to research questions Identify next steps according to initial goals

and/or research questions


Resources Web interface

http://minesforlibraries.org

ARL New Measures and Assessment Initiatives: MINES for Libraries®

http://www.arl.org/stats/initiatives/mines/index.shtmlplace

Articles & Presentationshttp://www.arl.org/stats/initiatives/mines/minesresources.shtml

http://minesforlibraries.org/

http://www.arl.org/stats/initiatives/mines/index.shtmlplace

http://www.arl.org/stats/initiatives/mines/minesresources.shtml

Measuring the Impact of Networked Electronic Resources: Developing an Assessment Infrastructure for...

Documents

Transcript of Measuring the Impact of Networked Electronic Resources: Developing an Assessment Infrastructure for...