Citation analysis in research evaluation

Wouter Gerritsma, Marianne RenkemaInformation Specialists, Wageningen UR Library

Introduction

Introduction Citation Indexes Citation analysis of scientific journals

Excercises Citation analysis of researchers and groups

of researchers

When research evaluation/citation analysis? Research assement of graduate schools (5

year cycle, stipulated by the VSNU) Other research groups are reviewed in a similar

way Benchmarking becomes more important

Grant applications Job positions

Sooner of later, you will be subjected

to such an analysis

Objective

How are citation analyses performed, and what are their shortcomings or weaknesses?

What can you do to enhance your results

References and citations what is the difference?

If paper R contains a bibliographic footnote using and describing Paper C, then R contains a reference to C, or R cites C, and C has a citation from R or is cited by R.

C is a (cited) reference contained in R. R is the source of the reference.

Source: Moed (2005) based on Price (1970)

Citation Indexes

Institute for Scientific Information Founder Eugene Garfield Part of the Thomson group (commercial publisher)

Index articles of about 7800 journals, standard bibliographic information + reference lists

Sell the same data in various databases to libraries and others (CC, WoS, JCR, ESI)

They have monopoly position for citation data; they are the Standard

CC, WoS, JCR, ESI

CC: Current Contents, weekly updated bibliographic database covering the top of Sciences (no citation data)

WoS: Web of Science, weekly updated database, that includes references and citation data as well (slower than CC)

JCR: Journal Citation Report, yearly analysis of WoS data, presenting perfomance measures for journals.

ESI, Essential Science Indicators, bimonthly updated analytical database giving performance measures for Countries, Institutes, Scientists and Journals next to research fronts, hot papers and baselines.

Database Coverage WoS/JCR/CC

Excellent Good Moderate

Molecular biology & Biochemistry

Applied physics & Chemistry

Other Social Sciences

Biological Sciences related to humans

Biological Sciences related to plants & animals

Humanities

Clinical medicine Psychology & psychiatry

Physics & Astronomy Other social sciences related to medicine & health

Mathematics / Engineering / Economics

Source: Moed (2005)

Are there competitors ?

Scopus (citation tools are being tested) Google Scholar (http://scholar.google.com) PsychInfo/SciFinder (article indexes in Desktop

library) ArXiv (Physics) Spires (high energy physics) Citeseer (ICT) OA Initiatives

Citation analysis of journals

Warning !

Often it has been attempted to use Journal Impact factors in some way to judge the performance of a group or individual scientist.

This is proven very bad practice!

Journal citation analysis

What are impact factors Are there other journal performance

measures? What are the pitfalls of the IF How can I make use of this knowledge?

Journal Citation Reports

Report 3 measures Impact factor Immediacy Index Cited half life

0 2 4 6 8 10

Years after publication

Cita

tions

/yea

rCited half-life

50% citations 50% citations

31

Immediacy index Window

Impact Factor Window

Definition of Impact factor

The impact factor of a journal J in year T is defined as follows:

“The number of citations received in year T by all documents published in J in the years T-1 and T-2”

÷“The number of citable documents published in J in

the years T-1 and T-2”

Citable items

Cited items (nominator) include: Research articles, review arrrticles, notes,

letters, editorials, news items, corrections and meeting abstracts

Citable items (by definition) Articles, notes and reviews

Lancet (2002)

Type of document

No. Docs Cites Cites/Doc

Articles/reviews (citable items)

1,544 (a) 13,106 8.5

Other types 4,899 2,564 0.5

Total 6,443 15,670(b) 2.4

JCR Like IF = (b)/(a) = 10.2

Impact factor

Performance measure for journals“… it is also used for assessment of the quality of

individual papers, scientists and departments. For the latter a scientific basis is lacking, as we will demonstrate in this contribution” (Opthof, 1997)

Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research 33(1): 1-7. http://dx.doi.org/10.1016/S0008-6363(96)00215-5

50 % of articles generate 90% of all cites

Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ 314(7079): 497-502. http://bmj.bmjjournals.com/cgi/content/full/314/7079/497

Some other points

IF measures citation impact of articles in 2nd & 3rd year after publication. Therefore biased towards journals revealing rapid maturing.

Reference practices, particularly the number of references per article and their age distribution, vary considerably among subfields. IF between subfields can therefore not be compared.

Life: JCR & ESI

When do you make use of IF

Answering the questions where do you want to publish your manuscript.

Other questions, to be asked Reach colleagues ? In which article indexes included? Open access?

Citation analysis of paper, scientist or group

Science Citation Indexes

Web of Science is comprised of three Citation Indexes

Science Citation Index Exanded Social Science Citation Index (SSCI) Arts & Humanities Citation Index (AHCI)

Searching WoS

General search: Searches bibliographic data: Title, Abstract, keywords, Authors, Addresses Not all citations are found due to errors in citing,

indexing etc. Cited reference search: searches in the

references of source data Not all articles are found. Non cited articles are

missing

Author searches

Use the index !

But what does it mean?

Publication A (2003), 25 citations (Mol. Bio.) Publication B (1996), 17 citations (Math)

Essential Science Indicators (ESI)

Analytical database of SCI, covering 10 years + current year building

Comparison between Countries, Institutes, Scientists and Journals

Hot papers / Highly cited papers Research fronts Baselines

Publication age : world averages citations (1)Citaties in verchillende onderzoeksvelden

0,00

50,00

100,00

150,00

200,00

250,00

300,00

350,00

400,00

450,00

199319951997199920012003

Publicatie jaar

Cit

ati

es

Agricultural Sciences

Molecular Biology &Genetics

1.00 %

10.00 %

1.00 %

10.00 %

Publication age : world averages citations (2)Citaties in verchillende onderzoeksvelden

0,10

1,00

10,00

100,00

1000,00

199319951997199920012003

Publicatie jaar

Cit

ati

es

Agricultural Sciences

Molecular Biology &Genetics

1.00 %

10.00 %

1.00 %

10.00 %

In practiceAgricultural sciences

0,1

1

10

100

199819992000200120022003

Publication year

Cit

atio

ns

1.00 %

10.00 %

AVG

Group 1

Group 2

Group 3

In practice (WIAS, plant & animal sciences)

0,1

1

10

100

1000

1994 1996 1998 2000 2002 2004

Publication year

Cit

atio

ns Citations

Average

Best 10%

Best 1%

Relative impact of a research institute

Subfield All groups Group 1 Group 2 Group 3

Agricultural Sciences

3,82 3,86 3,87 3,60

Biology & biochemistry

0,91 1,55 0,44 1,09

Chemistry 1,76 1,76

Clinical medicine 1,73 1,81 1,11

Microbiology 1,70 0.57 1,73

All subfields 2,06 2,08 2,26 1,84

Wageningen in Dutch perspective (2004 data)Institute

No. highly cited papers Papers Citations

Cites/ paper

World Rank

Univ Utrecht 343 23.938 286.385 11,96 68

Univ Amsterdam 338 20.730 262.183 12,65 75

Leiden Univ 280 18.565 262.183 14,11 77

Vrije Univ Amsterdam 248 15.353 200.385 13,05 111

Wageningen UR 230 14.707 151.227 10,28 171

Erasmus Univ Rotterdam 217 13.450 208.290 15,49 103

Univ Groningen 217 15.800 158.830 11,76 128

Univ Nijmegen 152 14.700 172.604 11,74 140

Delft Univ Technol 145 10.105 70.204 6,95 362

Univ Maastricht 112 8.636 95.775 11,09 276

Eindhoven Univ Technol 92 6.199 43.567 7,03 522

Univ Twente 75 5.679 39.144 6,89 562

Tilburg Univ 11 1.818 7.175 3,95 1714

Strength & Weaknesses of Wageningen UR

Subfield

Highly cited papers

Papers CitesCites/Paper

World rank (citations)

ENVIRONMENT/ECOLOGY 40 1878 20187 10,75 3

AGRICULTURAL SCIENCES 39 2212 14502 6,56 3

PLANT & ANIMAL SCIENCE 99 4249 39280 9,24 5

MICROBIOLOGY 8 951 15239 16,02 24

PHARMACOLOGY & TOXICOLOGY 2 189 2946 15,59 170

BIOLOGY & BIOCHEMISTRY 2 1421 15408 10,84 197

GEOSCIENCES 7 299 3051 10,20 222

MOLECULAR BIOLOGY & GENETICS 0 467 8237 17,64 263

CHEMISTRY 8 995 9616 9,66 284

SOCIAL SCIENCES, GENERAL 3 203 692 3,41 352

CLINICAL MEDICINE 14 791 15563 19,68 404

ENGINEERING 3 269 943 3,51 601

Example of selection of candidates for a professor in 2004

Candidate# Papers 1994-2003

#Citaties Relatieve Impact

RI 1994-1998

RI 1999-2003

#papers bij top 10%

#papers bij top 1%

a 80 1565 1,64 1,76 1,52 4 2

b 65 498 1,93 1,84 1,95 17 1

c 93 972 1,15 1,39 0,9 8 0

d 88 1886 1,86 1,69 1,94 16 3

e 57 346 0,75 0,58 0,83 3 0

Life WoS & ESIExercises WOS & ESI

What questions should you ask

Database coverage Accuracy of the counts Time period of the exercise Self citations Normalization? Verify the collected data

Winding down

What is in a name? Persons (female PhD students!, initials!) Groups (Chair groups, Business units) Wageningen UR !