Bibliometrics in the libraryWageningen UR library experience

at Milton Keynes, March 5th 2013

Wouter Gerritsma

Contents

Research Evaluation in the Netherlands

CRIS & Repository @Wageningen

Bibliometrics module

Research questions

Developments in the marketplace

Lessons learned

Some advice

Research assessment in the Netherlands

Supervised by VSNU/QANU

●6 year cycle for external peer reviews

●After 3 year midterm review

●Unit of analysis (in Wageningen): Graduate schools

Citation analyses are not stipulated in the current Standard Evaluation Protocol (SEP).

●This has become mandatory at Wageningen UR, als at the social sciences department and for the research institutes

SEP criteria

quality (including international academic reputation and PhD training)

productivity (the relationship between input and output)

societal relevance (including valorisation)

vitality and feasibility (the ability to react adequately to important changes in the environment).

Metis, our CRIS

Metis is a Current Research Information System (CRIS)

Data entry at chair group level

Quality control by the library

●Locating full text (uploading to e-depot)

●Maintenance journal lists

●Document type assignation and inclusion of DOI's

Compulsory output registration

●Research assessments only on metis registered publications

Information on all labour relations of faculty and staff

Information on all projects

Repository or Institutional Bibliography?

Wageningen Yield (WaY) is the repository of Wageningen UR

●Synchronized overnight with the updates from Metis

●WaY contains metadata descriptions of all Wageningen UR publication output, >190.000 items

●WaY is our OA repository, >40.000 items

●WaY is our tool for citation analyses, >22.000 publications

●Advanced bibliometrics

Full screen image with title

How do we compare numbers

Scientist Z. Math has a publication from 2002 with 17 citations

Scientist M. Biology has a publication from 2008 with 32 citations

Baselines for Mathematics

Baselines for Molecular Biology

For a single publication

Zee, F.P.v.d., G. Lettinga & J.A. Field (2001) Azo dye decolourisation by anaerobic granular sludge. Chemosphere 44:1169-1176.

●Citations from WoS: 94

Journal: Chemosphere

Categorised by ESI in Environment/Ecology

Baseline data for Environment/Ecology.

●Article from 2001 in Environment/ecology:

●On average: 19.36 citations;

●Top 10%: 44 citations; Top1%: 141 citations

Relative Impact: 94 / 19.36 = 4.9

van Veller, M.G.P et al. (2010). Bibliometric analyses on repository contents for the

evaluation of research at Wageningen UR. In: Qualitative and Quantitative

Methods in Libraries: Theory and Applications. A. Katsirikou and C. H. Skiadas.

p.19-26. http://edepot.wur.nl/7266.

Example of results

The actual publications and their impact are provided

Research credits report (partly)

Slice and dice any way you want

Advanced bibliometric indicators

Follow Moed (1995) as closely as possible; but.....

Web of Science is used for citation data

●We can’t make corrections for self citations

Essential Science Indicators for baseline data (World average, Top 10% and Top 1%)

●Limited number of research fields (22)

BUT:

We can determine the representativeness of the citation analysis!

Representativeness

Publication type #Pubs

Refereed articles 324

Non-refereed articles 7

Books 1

Refereed book chapters 36

Non-refereed book chapters 13

PhD Theses 45

Conference papers 137

Total Academic Publications 563

Representativeness

Publication type #Pubs WoS Repr.

Refereed articles 324 288 89%

Non-refereed articles 7

Google Scholar 1

Refereed book chapters 36

Non-refereed book chapters 13

PhD Theses 45

Conference papers 137

Total Academic Publications 563

Prospective versus Retrospective analyses

CWTS performs normally Prospective analyses

●Current researchers, 10 years back

●Missing some retired bigshots!

You need to keep track of the actual publication record for retrospective analyses. This is difficult for external parties.

●Head-tail problems

No research on differences in outcomes of prospective versus retrospective analyses

●We need research in this area!

Self citations

CWTS performs corrections for self citations

Correcting for self citations in Web of Science is incomplete

●As long as the RsearcherID is not fully introduced this will be impossible in WoS

Correcting for Self citations in Scopus is possible

Belgian research has shown that it has not a tremendous influence

Glänzel, W., K. Debackere, B. Thijs & A. Schubert (2006). A concise review on the role of author self-citations in information science, bibliometrics and science policy. Scientometrics, 67(2): 263-277 http://dx.doi.org/10.1007/s11192-006-0098-9

Some observations on commercial analytical tools

Web of Science

Citation data (you can include citations from other databases on Wok)

API to download citation data

Baselines from ESI

InCites more advanced, but where do you manage the information?

Author disambiguation is a major problem

Scopus

Citation data obtainable through an API

Benchmarking with SciVal Strata, no API yet

Not yet fully developed, major changes coming up.

CWTS monitor

So far the most elegant and comprehensive citation analysis tool (still in beta) to be launched soon.

Citation database agnostic!

Google Scholar

Very popular by social scientists and arts & humanities

Have you ever retrieved more than 1000 results from any Google product?

Google Scholar can't count

Harzing's Publish or Perish software does a decent job.

Altmetrics

Quickly developing

●ScienceCard

●Total-Impact

●Readermeter

●Microsoft Academic Search

●etc.

We look into inclusion on top of the WaY for article level metrics

Wouters, P. & R. Costas (2012). Users, narcissism and control. Utrecht, NL: SURFfoundation. http://www.surffoundation.nl/en/publicaties/Pages/Users_narcissism_control.aspx.

CRIS and Bibliometric analysis tools

If you maintain a CRIS, why should you maintain your researchers and organisation structure in a bibliometrics analysis tool as well?

Do commercial packages have ways to publicize the results for scrutiny by the researchers?

Lessons learned

Matching Wageningen Yield and WoS

WoS: 9577 articles

WaY: 10933 articles

Missing in Way: 807 articles

Missing in WoS: 1159 articles

1161 peer reviewed articles not in ISI journals

It is a lot of hard work to keep track of all publications. The library can a should do a better job than commercial service providers

Journal selection and impact @WUR

Why in the library?

Library is the functional manager of Metis / WaY because of wide experience with bibliographic metadata

Library manages contracts with publisher(s) of external databases that are being used

Library has experience in developing and maintaining large databases

Library has ample experience in searching complicated databases such as Web of Science

Advantage of using Metis / WaY

Improvements in publication lists, etc. recorded

Knowledge of, and experience with bibliometric analyses is better institutionalized

More visibility through Open Access management

Clarity / transparency for researchers

Analysis of a single unit within the institute offers advantages for the organization as a whole

Better understanding of our own researchers

●We know where they publish

●We know what they cite

●We know something about their impact

Raising library awareness

Improvement of the (metadata) quality in the repository

Quality has lead to compulsory registration for research assessments

Presentations for research groups during the preparation for peer reviews

Presentations based on detailed studies of single groups

Library gives advice on elements for publication strategies for groups and individuals

●there is a huge demand for these workshops

Closing the circle

My advise

Start small, gain experience

Show you can pull it off

Be transparent!

How much is your university spending research evaluations?

Invest those resources in your own systems

Thank you

http://viaf.org/viaf/285392263/

http://orcid.org/0000-0001-7274-0698

http://wu.academia.edu/WouterGerritsma

http://www.researcherid.com/rid/A-4161-2008

http://www.mendeley.com/profiles/wouter-gerritsma

http://www.researchgate.net/profile/Wouter_Gerritsma

http://scholar.google.com/citations?user=3iDBE-MAAAAJ

http://academic.research.microsoft.com/Author/34373815

http://www.narcis.nl/person/info:eu-repo/dai/nl/33714253X