Post on 15-Nov-2014
description
Usage Factor:
Final Report & Next Steps
John McDonald, Director, Libraries of the Claremont Colleges
Usage Statistics: New Developments and Practical ApplicationsNFAIS Committee on Usage Statistics
New York City, October 21, 2008
Usage Factor Genesis
COUNTER statistics provide a reliable comparison of amount of use between journals
But they don’t provide a meaningful usage-based measure of relative quality or value
ISI's Impact Factor compensates for the fact that larger journals will tend to be cited more than smaller ones
Who will be interested? Is there a demand for it among:-
Publishers Authors Librarians
What are the practical issues that would need to be addressed?
Who will do it & how much will it cost to develop and maintain?
Research review
From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005
Some initial evidence……
From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005
Addressing the challenge…….
ISI's Impact Factor provides an industry standard for journal quality.
Can we do something similar for usage?
In other words, should we seek to develop a usage-based measure of journal quality or value?
What metrics should be used?
The Research…….
In 2007 UK Serials Group and COUNTER published research which examined:- the various ways in which journal quality is
currently assessed the degree to which any additional usage-
based metrics might prove valuable to each stakeholder community
practical ways in which such metrics might be derived and constructed to provide the maximum utility for all, within defined resource constraints
Stage 1 Phase 1 October 2006 – January
2007 COUNTER Director Peter Shepherd carried
out a series of in-depth interviews with:- 7 authors 9 librarians 13 publishers
Phase 2 March 2007 Broader web based survey of:-
155 librarians 1400 academic authors
Results June 2007 Final report published by UKSG
Stage 1, Phase 1 Results - Users
Would Journal Usage Factors be helpful to you in assessing the value, status and relevance of a journal?
Librarians – YES 100% Authors - YES 100%
Phase 1 Usage factor advantages
A useful counterweight to Impact Factors
Especially helpful for journals and fields not covered by ISI
Especially helpful for journals with high undergraduate or practitioner use
Especially helpful for journals publishing relatively few articles
Data available potentially sooner than with Impact Factors
Phase 1 Usage factor advantages
Authors select journals that will give their articles prestige and reach. Impact Factor is a widely used surrogate for the former, while perceived circulation and readership reflect the latter. But usage is becoming more important as a measure of reach.
Carol Tenopir
A simple, usage based metric would make usage more understandable to editors and authors as a measure of value. There is currently much talk of usage and a lot of data, which the non-librarians find confusing.
Publisher
Phase 1 Issues to address
Impact Factors are well known and relied upon in the industry
Not all publishers are COUNTER compliant.
COUNTER data may not be robust enough for this.
How would print usage, still significant for many journals, be taken into account?
Would another global measure, such as usage half-life per journal or per discipline, be of greater value?
Usage based metrics could stimulate publishers to inflate their usage or be influenced by sales forces and technology investments.
Many journals publish in multiple hosting sites, making calculation of a single usage metric problematic.
Phase 1 Who will construct the measure?
Publishers are, on the whole, unwilling to provide their usage data to a third party for consolidation calculation of Usage Factors.
The majority appear to be willing to calculate UFs for their own journals and to have this process audited.
This is generally perceived as a natural extension of the work already being done for COUNTER.
While it may have implications for systems, these are not seen as being problematic.
Some publishers already consolidate their usage data from several sources and also maintain figures on the total number of articles published in each of their journals.
Basis for Usage Factor
Usage Factor =
Total usage over period ‘x’ of articles published during period ‘y’
÷Total articles published during period ‘y’
Phase 1 Definitions
Many views on how UF components should be defined.
In particular, how to define total usage specified usage period total number of articles published online specified publication period
Tests with real usage data will be required to refine the definitions for these terms.
Phase 1 What to measure
TOTAL USAGE There was agreement that usage should be
total downloads, as specified in COUNTER JR1.
SPECIFIED USAGE PERIOD There was overwhelming agreement that the
specified usage period be one calendar year.
TOTAL NUMBER OF ARTICLES PUBLISHED ONLINE Need to account for journals not covered by ISI Is the ISI definition of “source items” too narrow
when measuring usage?
Phase 1 What to measure
SPECIFIED PUBLICATION PERIOD There was a diversity of responses to this
question, with no clear consensus on any time period among authors, librarians or publishers.
Insufficient data to support the selection of
any one specific option
Tests using real data will be required before any firm conclusions can be drawn.
Phase 2 Librarian results: new journals
Ranking without Usage Factor Ranking with Usage Factor
1. Feedback from library users 1. Feedback from library users
2. Price 2. Usage Factor
3. Reputation/status of publisher 3.Price
4. Impact Factor 4. Impact Factor
5. Reputation/status of publisher
Phase 2 Librarian results: existing journals
Ranking without Usage Factor
Ranking with Usage Factor
1. Feedback from library users 1. Feedback from library users
2. Usage 2. Usage
3. Price 3. Usage Factor
4. Cost per Download 4. Price
5. Impact Factor 5. Cost per Download
6. Reputation/status of publisher
6. Impact Factor
7. Reputation/status of Publisher
“I would view Usage Factor as an aid for collection rather than cancellation decisions. Usage per se is a more suitable tool for us when considering cancellation.”
Stage 1 recommendations
UF be developed to the point of testing as a practical, implementable measure of journal quality, value and status.
On May 18, 2007 the UKSG Committee accepted these recommendations.
Next Steps: Stage 2
Test each of the individual elements in the UF equation using real publisher usage data.
Compare UF ranks with IF ranks.
Refine and further define the workflow/organization scenarios for the definition, calculation and dissemination of the Usage Factor.
Stage 2 – The Plan
Project Steering Group established (6 publishers, 1 aggregator, 1 hosting service, wider geographical library representation
Usage logs to be converted to uniform standard report format for analysis by expert third party
RFP currently in DRAFT form for third party selection process
Stage 2 – The Deliverables A report (early 2009) which will:
Outline the various metrics assessed
Recommend which of them prove consistent and robust enough to be adopted for scaled up onward monitoring
Suggest any ways in which data providers might amend the way they capture, structure, label, and maintain their data which would make the measurement of Usage Factors easier and more reliable.
Propose ways to audit Usage Factors for accuracy
Importance….. “Currently journal publishers are under
a lot of pressure to demonstrate the value they provide. By participating in this process, publishers will influence it, helping to develop useful measures in which they can have confidence.”
“This is going to happen in any event, so it is best that UF is developed and implemented by a trusted organization in which publishers are represented.”
Usage Factor Project
More information at: http://www.uksg.org/usagefactors
Contact: Richard Gedye, Project Director
richard.gedye@oxfordjournals.org