LibMeterLibMeter

Striving for Excellence with Electronic Library Statistics

Indicators for Performance EvaluationIntroduction – Current Practice

Institut für Informationswissenschaft, University of Applied Sciences, Cologne (FH-Köln IIW)

Seminar on Library Evaluation

(S. FÜHLES-UBACH)

Peter Ahrens Guest Lecture

V 1.00a – 2009-05-18

Cologne, 7th May 2009

LibMeterLibMeter

LibMeter Seminar

This LibMeter Introduction Seminar is a continuation from the following

Slideshare Presentation: Current Practice – Basics

It also relates to: LibMeter Case Study #1(University of Düsseldorf)

LibMeterLibMeter

Roadmap of this PresentationIntroduction - Current Practice

I. Introduction - Basics A Quick Futuristic Web Tour Usage Stats for the Rest of Us (No Maths please!) Library Role, Services & Usage

II. Introduction – Current Practise Where to get Usage Figures from ? How to interpret usage Figures ? LibMeter Case Studies

III. Perspectives

LibMeterLibMeter

Where to get Usage Figures for

Electronic Services?

LibMeterLibMeter

German Library Statistics

DBS 2007+

LibMeterLibMeter

Which Parties are involved in Generating Service and Stats ?

Sorry, we need to become technical again – Stats Source One Party-Scenario (Examples: OPAC hits, Homepage)

Service Provider = Library += Raw Statistics Provider = Webserver/ IT Department

(+/- allied special statistics service provider) Two Party-Scenario (Example: COUNTER stats)

Lib. + Content Provider = Online Publisher/ Service Host Three Party Scenario (Examples EZB; OpenURL)

Lib. + Content + Web Service provider - Library Computing Service usually Server with powerful integrated specific statistic tools

Multi Party-Scenario: (Example BIX Virtual Library Use) Lib. + Content + Web Service + Stats Provider

LibMeterLibMeter

DBS Starts reporting on Electronic Usage in 2007

DBS Variable Analysis (VA) is available since 2004 DBS-VA contains annual data back to 1999 DBS reports on 6 electronic „OUTPUT“-parameters

(=use or usage) since 2007

LibMeterLibMeter

Use of Electronic Resources & Online Services in DBS (2007)

General Services (1 Party or 1 Party+) Visits of Library-Homepage (Zählpixel) Visits of OPAC (Zählpixel) *1.)

Content Delivery (2 Parties) *2.)

Full Text Downloads of locally licensed articles (COUNTER) Database Search Sessions (+/-COUNTER) Fulltext Views of locally managed Electronic Documents

Context –Pointing –Linking Services (3 Parties) *2.)

Links to licensed eJournals (Homepages) (EZB=A-Z List)

LibMeterLibMeter

Availability of Electronic Usage Data is Still Limited (DBS 2007)

Compiled from Data downloaded from: http://www.bibliotheksstatistik.de/eingabe/dynrep/index.php [2009-04-15]

LibMeterLibMeter

Sample University Library Profile (DBS)Basics on Library-“Size“, -Input & Output

Actions Raw Numbers

Often very high values Go beyond imagination

Work Basis: User Numbers Different Types Can be combined to

Primary User NumberPrimary User Number Standardization

Key Step: Calculation of „Value Value per User per Yearper User per Year“ = Normalization

Benefits Numbers become imaginable Numbers can be memorized Numbers become comparable

Data downloaded from www.bibliotheksstatistik.de into PC-Spreadsheet Program, [2009-05-03]

LibMeterLibMeter

Looking at one Parameter (EZB)

reported by 2 statistics hosts EZB Admin versus

DBS 2007

LibMeterLibMeter

Linking to „Electronic Journals“What it means ? What is counted ?

Different reporting definitions – CAVE, Don‘t compare apple and pears ! Standard EZB reports for a member institution (library)

A+B+C+D = EZB_Standard (include all EZB-Requests issued from one Institution) Special EZB reporting for DBS/Par. #181 (automatic upload 2007 onwards)

B+C = EZB_for_DBS (exclude A+D = Open Access, NatLis, free Grey Lit + PPV) What does this mean in actual numbers (e.g. for ULB Düsseldorf for 2007)

EZB_for_DBS = 124.421; Total Counts: EZB_Standard = 185.361 => +/- 60.940

A = free accessB = locally licensedC = As B but only certain yearsD = paid full text, not licensed

Access Category of Publication

LibMeterLibMeter

Pros and Cons of counting EZB „free“ eJournal-Linking activity ?

NO (B&C only) - Controlling & accounting approach (DBS 2007) „Free“ eSources do not comply with usage indicator DBS#181 definition - to

have clearly locally attributable cost & effort Direct Cost is a pre-requisit for DBS#181 = No direct cost, no count !

„Maßgeblich ist immer die Prämisse, dass diese Angebote der Bibliothek Aufwand (Geld und Arbeit) verursachen und damit eine mit den vorhandenen Ressourcen erbrachte Leistung darstellen“ (*)

Content 1-to-1 correlated with (paid!) COUNTER articles DBS#183

YES (+A&D): Service monitoring approach (EZB basic Report) Library participates in community effort of coop. cataloguing to keep EZB alive Service is clearly attributable to local users of a given institution Many additional Service events for similar service (+50%)

Reporting would allow analysis of trends in usage of electronic scholarly information and significant related library services (abd community efforts) What is the bulk use of all NatLis, all Open Access journals ? Are there differences between Institution(s) (Types) Are there Proportion Shifts between A, B, C & D over time (Berlin-Declaration, DFG)

LibMeterLibMeter

Looking annually at 1 Parameter (EZB)

at 1 time (DBS 2007) for many libraries

LibMeterLibMeter

Comparing EZB Use (B+C) among University Libraries in 2007

Compiled from DBS VA: http://www.bibliotheksstatistik.de/eingabe/dynrep/index.php [2009-04-15]

EZB (B+C only) call as per DBS 2007Standardized per Primary User per Year

LibMeterLibMeter

Comparing Usage at different Types of

Institutions

LibMeterLibMeter

2007 EZB Usage per Primary User at Universities versus Polytechs

EZB Statistics Universities (n=57)

Median = 6,2 Mean / Average = 6.9 Standard Deviation = 5.6 Variation Coefficient = 0.8

Polytechs (n=34) Median = 0,42 (sic!) Mean / Average = 2.0 Standard Deviation = 6.1 Variation Coefficient =3.0

Düsseldorf: Mean / Average 7.0 = 0.13 Std. Dev. above Mean

While you may calculate Mean and St. Dev. for a population, make sure it is normally

distributed before drawing further conclusions (check Median) !

LibMeterLibMeter

Looking at Distribution among University Libraries & Polytechs ?

A: Number of EZB calls correlated with conventional Loans* ? NO, there is no correlation

B: Number of EZB calls correlated with the Staff/Student ratio* ? YES, there seems to be a weak

positive correlation i.e. higher staff proportion goes

along with higher the EZB usage In other words (Hypothesis):

Staff uses EZB eJournal-Catalogue more than Students

This is not astonishing, but for the first time it becomes quantifiable

A

B

LibMeterLibMeter

BIX-2008+Performance Indicator

„Virtual Use“

LibMeterLibMeter

HDM Usage Counts for Electronic Library „Traffic“

Data: Courtesy of I. SIEBERT & J. KREISCHE, University Library Düsseldorf from private Website at HDM [2009-04-30]

LibMeterLibMeter

BIX Performance-Indicator „Virtual Library Use“ New (per 2008*) in German Library Performance Index „BIX“ Normalized Combined Parameter: „Virtual Annual Use“ Definition: (OPAC & Homepage) / Number of Primary

Users Simple Interpretation

Year Source OPAC Homepage OPAC + HP Primary Users2007 DBS 684.097 3.477.400 4.161.497 16.991

HDM 1.116.511 2.360.890 3.477.401 16.991

Hits per User 65,7 138,9 204,7 = Virtual Use

LibMeterLibMeter

Electronic Usage enters the „Main Stage“ in 2008 (with BIX)

„Simple“ Standardized Parameters (meaningful !)

Annual Library „Use“ per Primary User: Physical visits: 110,8 Virtual Use: 204,7

Simple Statements Average User comes every

second workday Average virtual user comes

daily CAVE: Only „front door“

LibMeterLibMeter

From Electronic Library Output to „Virtual Use“ (DBS2007 > BIX2008)

Principle Take reliable Usage figures from

different relevant Services with high number of Events per User per Year

Combine to new parameter Benefits

Comprehensive & Easy to understand

Easy to compare E.g. with physical use

Limits Difficult to standardize

Homepage & OPAC counts across institutions (different software, no established counting standard)

HDM-Zählpixel-Provider limits

LibMeterLibMeterHow SMART is „Virtual Use“ ?Specific: Quite general Measurable: YES Community Standard with help

of HDM ZählpixelAchievable and Attributable: Library is actively producing Service Today every library can produce #Relevant and Realistic: YES Users frequently go there Closely tied to core services How representative for total spectrum

of electronic Library Services ?Time-bound, Timely, Trackable, and

Targeted: No log-files publicly available (yet?) Not available for all periods

LibMeterLibMeter

DBS & BIX StatisticsBenefits & Limits

Benefits Broad Institutions Base Fairly Comprehensive Creates Awareness

Community Public

Online Data Warehouse Transparency Allows Meta Studies

Limits Few Parameters Limited Scope

i.e. more detailled services missing

Standardization Problems Incomplete data Only Annual Data Time Lag International Comparability ?

LibMeterLibMeter

Journal & Database Vendors

(COUNTER)

LibMeterLibMeter

COUNTER provides „standardized“ Usage Stats

LibMeterLibMeter

Example: Usage of Subscribed JournalsVia COUNTER & SUSHI & ScholarlyStats

LibMeterLibMeter

COUNTER StatisticsBenefits & LimitsBenefits Working towards Standardization

of Online Content Usage Counting Accepted international

(commercial Providers) Code of Conduct

Available for hundreds of publishers & Database providers Available for most e-Journals

Basis for Commercial Discussions with Publishers and Hosts External: With providers Internal: With faculty

Limits Commercial Content only

OpenAccess, PubMed, Google Scholar missing

Expertise & work needed to handle the hundreds and thousands of spreadsheets

Needs Tools to handle Data XML-Harvester (SUSHI) Warehouse: ScholarlyStats

Standardization Problems No ISO Standard !

Focus on Publishers Content, not Library e-Services

Can you trust your Providers ?

LibMeterLibMeter

StandardzedLibrary Services(e.g. OpenURL)

LibMeterLibMeter

INTERNET

1. Generation: Static Links (1992+)

Sources TargetsStatischer Link: URL

LibMeterLibMeter

INTERNET

BIBLIOTHEK

2. Generation: Persist. Identifier Links

Sources Targets

Datenbasesof Links

CrossRefDOI

Too dynamic,Too much work

For Library

LibMeterLibMeter

INTERNET

Virtual Library

3. Generation: Context sensitive LinksOpenURL Resolver

Sources Targets

OpenURL-ResolverLinkServer

Service MenuLink to fulltext

Document DeliveryInterlibrary Loan

OPAC Entry Store in PBS

Lookup in WoS…

CATA

LOG

INDE

X

BOOK

JOUR

NAL

LibMeterLibMeter

http://sfx.aaa.edu/menu?genre=article&issn=1234-5678&volume=12&issue=3&spage=1&epage=8&date=1998&aulast=Smith&aufirst=Paul

Linking with OpenURL ResolverPrinciple & URL-Syntax

LibMeterLibMeter

Commercial Introduction of LinkResolvers (2000-2004)

Time since Market Introduction (in Years per Sept. 2004)(Source: Computers in Libraries, Okt. 2004)

0 1 2 3 4 5

Ex Libris - SFX

Elsevier/Endeavor - LinkFinderPlus

Openly - 1Cate

Innovative Interfaces - WebBridge

Geac - Vlink

Fretwell-Downing - OL2

Sirsi - Sirsi Resolver

Serials Solutions - Article Linker

EBSCO - LinkSource

Ovid - LinkSolver

TDNet - TOUR

LibMeterLibMeter

OpenURL Dynamically Links to the „Appropriate Copies“

LibMeterLibMeter

OpenURL Statistics available ad hocwith Breakdowns - SFX Admin Center

•Ermittlung der Link-Nutzung in der Datenbank CAPlus für Feb.-Apr. 06•zusammengefaßt monatlich•Eingeschränkt nur auf die Gruppe der (echten) universitären Nutzer•Ausgabe differenziert nach Dokumentenzahl zu denen elektronischer Volltext geboten werden konnte (incl. Nationallizenzen) versus kein elektronischer Volltext (nur andere SFX-Dienste der Bibliothek) •Anmerkung: Dieser Statistik-Typ erlaubt das einfache Abschätzen und Monitoring von Volltext-Abdeckungsgraden für verschiedene Datenbanken, Fachbereiche oder Anwenderungruppen.

LibMeterLibMeter

OpenURL StatisticsBenefits & LimitsBenefits Global ISO Standard Counts fully comparable Very wide scope of service Built-In Report Generators

Easy, fast, flexible, reliable distinguishes licensing types

(local, consortial, national, free) Multiple Facets breakdown

possible Sources, Targets, Services … Variable Periods Sub-Groups of Users (IP-based or

other) Service and Context focussed (not

just content)

Limits Not all libraries have OpenURL

resolver yet (about 95 % of University Libraries do)

Not official DBS-parameter (yet?) Extra Cost of OpenRUL Resolver

Service Not all Links go through OpenURL

(same applies for other link types) Some sources do not send

source-ID -> remain unidentified

LibMeterLibMeter

Working more with Usage Figures

Case Studies

LibMeterLibMeter

LibMeter Case_Study #1

Data: Personal Communication May 2009, Courtesy of Dr. Kreische, ULBD Düsseldorf

Basics of Electronic UseBIX „Virtual Use“

@ ULBD Uni DüsseldorfSee separate Slide show

LibMeterLibMeter

Selected Readings to broaden and to deepen your Understanding

1. BIX - Der Bibliotheksindex 2008. B.I.T.online Sonderheft. 2008 Juni ;1-56.   2. Blecic DD, Fiscella JB, Wiberley Jr SE. Measurement of Use of Electronic Resources: Advances in Use

Statistics and Innovations in Resource Functionality. College and Research Libraries. 2007 ;68(1):26-44.   3. Ceynowa VK, Coners A. Balanced Scorecard für wissensch. Bibliotheken. Vittorio Klostermann; 2002.   4. Gallagher J, Bauer K, Dollar DM. Evidence-based librarianship: Utilizing data from all available sources to

make judicious print cancellation decisions. Lib. Collections, Acq. & Tech. Serv.. 2005 ;29(2):169-179.   5. George Boston, Whang M. E-Resources Usage Data: Apples to Oranges and Fixing Holes   [Internet].

Atlanta: 2008. [zitiert 2009 Apr 22] Available_from:http://209.85.129.132/search?q=cache:mj5wfYDT2BoJ:https://smartech.gatech.edu/bitstream/

1853/20874/13/Boston_Whang_ERL_final.ppt+scholarly-stats+sfx&cd=4&hl=de&ct=clnk&gl=de&client=firefox-a 6. Hutzler E. Bibliotheken gestalten Zukunft : Kooperative Wege zur digitalen Bibliothek. Dr. Friedrich

Geißelmann zum 65. Geburtstag. Universitätsverlag Göttingen; 2008.   7. Kreische J. Die Messung von Vernetzung. Nutzungsstatistiken mit SFX [Internet]. Münster: 2006.

1. Available from: http://docserv.uni-duesseldorf.de/servlets/DocumentServlet?id=9728 7.8. Kreische J. Zwischen Ranking und Qualitätsmanagement: BIX WB [Internet]. Mannheim: 2008.

1. Available from: http://www.opus-bayern.de/bib-info/volltexte/2008/583/pdf/BIX%20Mannheim%20(Word%202003).pdf9. Poll R, te Boekhorst P, Measuring quality, Saur, München, 2007.   10. Lossau N. digital services in academic libraries. In: Digital Convergence - Libraries of the Future: Libraries

of the Future. Springer; 2008.   11. Shepherd P. The feasibility of developing and implementing journal usage factors: a research project

sponsored by UKSG. Serials. 2007;20(2):117-123.

LibMeterLibMeter

Special Thanks for critical & constructive discussions to: Joachim KREISCHE, ULB Düsseldorf Simone FÜHLES-UBACH, FH Köln Eric MULDER, Den Haag Marion MÜLLER , ULB Düsseldorf Jessica BUSCHMANN, Dortmund Nol VERHAGEN, UvA, Amsterdam Sylvia THIELE , UB Dortmund Britta RIEBSCHLÄGER , ULB Düsseldorf Peter KOSTÄDT, UB Köln

LibMeterLibMeter

About me – Key Fields of InterestsLibrariesEarly 1970s – Founding highschool libraryEarly 1980s - Converting Card Catalog to electronic (Mainframe & punchcards)Late 1980s creating bibliographic database for 5.000+ article reprints (Mac) Since 1990 consulting scholarly libraries on retrieval systems, End user friendynessHospital Libraries

Life Sciences& Statistics

Ph.D. thesis on pattern recognition in brain Several peer-reviewed publicatios

Presentations

Publishing & SoftwareEditing & Managing highschool journal for 5 yearsHeading Cycling Campaign Newsletter for 3 years

Knowledge Finder – Biomedical Search Engine •Distribution & Localization for Europe

12 years e- Ressources•Elsevier / Kluver / Ex Libris

•Science Direct,Embase.com, Ovid, SFX, MetaLib

Knowledge Management1990s Scientific Knowledge Management Software • Development & DistributionLate 1990s e-library transformation• consulting, Projects, trainingsMid 2000s Library e-services•Development of Cooperative strategies•Market Analysis & Business Planning

Information Technology in the SciencesLate 1970s first own programming on handheld computers, then Mainframes, then Apple II, then MacLate 1980s German Academic Software Price for co-developping a relational bibliographicpProgram - PARiS1990s Fuzzy logic, natural language & relevance ranked output retrievalLate 1990s – Online e-Journal and integrated search platformsMid 2000s – OpenURL, Context sensitive linking, Metasearch, ERM, Recommenders, Library 2.0

LibMeterLibMeter

LibMeter Seminar

x

This LibMeter Introduction Seminar is a continuation from the following

Slideshare Presentation: Current Practice – Basics

It also relates to: LibMeter Case Study #1(University of Düsseldorf)

LibMeterLibMeter

LibMeter Seminar

x

Personal Announcement:It is intended to continue this series slowly with further modules amongst others on:

Detecting, following and predicting Usage-Trends Measuring Impact of e-Service Marketing Events

International Library Comparisons Comparing Academic & Library Excellence

LibMeterLibMeter

LibMeter Seminar

x

The End !

Thank YouPeter Ahrens

libmeter-at-gmail.com