Modern Developments in HRM Practices: Performance Management
Citation analysis: State of the art, good practices, and future developments
-
Upload
ludo-waltman -
Category
Science
-
view
559 -
download
0
Transcript of Citation analysis: State of the art, good practices, and future developments
![Page 1: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/1.jpg)
Citation analysis: State of the art, good practices, and future developmentsLudo WaltmanCentre for Science and Technology Studies, Leiden University
Bibliometrics & Research Assessment: A Symposium for Librarians & Information ProfessionalsBethesda, MarylandOctober 31, 2016
![Page 2: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/2.jpg)
2
Centre for Science and Technology Studies (CWTS)• Research center at Leiden
University in the Netherlands• History of more than 25 years in
bibliometric and scientometric research
• Commercial bibliometric analyses for clients worldwide
![Page 3: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/3.jpg)
3
State of the art
![Page 4: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/4.jpg)
4
What do citations measure?
Quality Scientific impact
Relevance
Visibility
Other factors
Reputation
Citations
![Page 5: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/5.jpg)
5
Citizen bibliometrics vs. professional bibliometrics
Citizen bibliometrics• Do-it-yourself bibliometrics by
researchers, research managers, and librarians
• Publication and citation counts, impact factor, h-index
• Web of Science, Scopus, Google Scholar
• Mainly at the level of individual researchers
Professional bibliometrics• Bibliometric analyses
supported by professional bibliometricians or specialized bibliometric software tools
• Field-normalized indicators
• Web of Science, Scopus
• Mainly at the level of research institutes
![Page 6: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/6.jpg)
6
Impact factor
Impact factor
![Page 7: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/7.jpg)
7
Common criticism on impact factor
• Too short citation window• Too sensitive to highly cited publications• Inflation by review articles• Numerator/denominator inconsistency• Vulnerability to manipulation• Incomparability between fields• Impact factor not suitable for assessing individual
publications and their authors
![Page 8: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/8.jpg)
8
Skewness of citation distributions
![Page 9: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/9.jpg)
Skewness of citation distributions
9
![Page 10: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/10.jpg)
10
San Francisco Declaration on Research Assessment (DORA)
![Page 11: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/11.jpg)
11
![Page 12: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/12.jpg)
12
![Page 13: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/13.jpg)
13
h-index
A scientist has index h if h of his papers have at least h citations each and the other papers have
at most h citations each
![Page 14: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/14.jpg)
14
Common criticism on h-index
• Undervaluation of highly cited publications• Dependence on career length• Inflation by co-authored publications• Incomparability between fields
![Page 15: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/15.jpg)
15
Arbitrariness of the h-index
![Page 16: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/16.jpg)
Consistency of an indicator
16
If two scientists achieve the same performance improvement, their ranking relative to each other
should remain unchanged
![Page 17: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/17.jpg)
17
Inconsistency of the h-index
Source: Waltman and Van Eck (2012). JASIST, 63(2), 406-415.
![Page 18: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/18.jpg)
18
Differences in citation density between fields
![Page 19: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/19.jpg)
19
Field normalization
• Professional bibliometrics is focused mainly on the institutional level, where field normalization is essential
• Standard field normalization approach:– Consider a publication from 2010 in an oncology journal– The publication has 45 citations– On average, publications from 2010 in oncology journals have
15 citations– Normalized citation score of the publication is 45 / 15 = 3
![Page 20: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/20.jpg)
20
Adoption of standard field normalization approach
![Page 21: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/21.jpg)
21
Limitations of standard field normalization approach• Inaccuracies in journal-based field classification
systems• Multidisciplinary journals• Fields are too broad
![Page 22: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/22.jpg)
22
Fields are too broad (Clinical Neurology)
Neurosurgery
Neurology
![Page 23: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/23.jpg)
23
Alternative field normalization approaches• Publication-level rather than journal-level field
classification systems (CWTS Leiden Ranking)• Source normalization (SNIP)• Flexible field definitions, for instance based on co-
citations (Relative Citation Ratio)
![Page 24: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/24.jpg)
24
4000 fields in publication-level classification system
Social sciences and
humanities
Biomedical and health sciences
Life and earth sciences
Physical sciences
and engineering
Mathematics and computer science
![Page 25: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/25.jpg)
25
CWTS Leiden Ranking 2016 (www.leidenranking.com)
![Page 26: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/26.jpg)
26
Source normalization (SNIP)
![Page 27: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/27.jpg)
27
Relative Citation Ratio
![Page 28: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/28.jpg)
28
Challenges in constructing field normalized indicators• Accuracy:
– Indicators should provide accurate field-normalized citation scores
• Transparency/simplicity:– Indicators should be transparent and easy to explain
• Validity:– Being cited should never be harmful– Indicators should behave consistently
![Page 29: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/29.jpg)
29
Good practices
![Page 30: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/30.jpg)
30
Diana Hicks, Paul Wouters, Ludo Waltman, Sarah de Rijcke, and Ismael Rafols
Nature, April 23, 2015www.leidenmanifesto.org
![Page 31: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/31.jpg)
33
Keep analytical processes open, transparent and simple
![Page 32: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/32.jpg)
34
Keep analytical processes open, transparent and simple• Using non-transparent indicators to support
expert judgment is difficult• Documenting an indicator in a scientific paper
doesn’t give full transparency• Citizen bibliometrics may be more transparent
than professional bibliometrics
![Page 33: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/33.jpg)
35
Social sciences and
humanities
Biomedical and health sciences
Life and earth sciences
Physical sciences
and engineering
Mathematics and computer science
Field normalization based on 4000 publication-level fields: Transparent?
![Page 34: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/34.jpg)
36
Recognize systemic effects of indicators: Impact factor
Source: Hicks et al. (2015). Nature, 520, 429-431.
![Page 35: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/35.jpg)
37
Recognize systemic effects of indicators: Impact factor
% of journals with more than threefold overrepresentation of self-citations to past two years
![Page 36: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/36.jpg)
38
Future developments
![Page 37: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/37.jpg)
39
New data sources: Altmetrics
![Page 38: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/38.jpg)
40
New data source: Full text
Source: Bertin et al. (2016). JASIST, 67(1), 164-177.
![Page 39: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/39.jpg)
41
New data sources: Full text
Source: Rahulja et al. (in press). Natural Language Engineering.
![Page 40: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/40.jpg)
42
Contextualized scientometrics: Beyond just indicators• Indicators should be used to complement expert
judgment• Complex black-box indicators tend to replace
expert judgment rather than to complement it• Indicators should be presented together with
contextual information; one should be able to see what is behind the numbers
![Page 41: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/41.jpg)
Contextualized scientometrics:What is behind the numbers?
43
![Page 42: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/42.jpg)
44
Contextualized scientometrics:What is behind the impact factor?
PNAS EMBO Journal
![Page 43: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/43.jpg)
45
Contextualized scientometrics: Altmetric dashboard
![Page 44: Citation analysis: State of the art, good practices, and future developments](https://reader036.fdocuments.net/reader036/viewer/2022070514/587edde01a28abdb198b7d75/html5/thumbnails/44.jpg)
46
Contextualized scientometrics:Visual user interfaces