Evaluating Greek Departments of Computer Science/Engineering using Bibliometric Indices Dimitris...
-
date post
21-Dec-2015 -
Category
Documents
-
view
222 -
download
4
Transcript of Evaluating Greek Departments of Computer Science/Engineering using Bibliometric Indices Dimitris...
Evaluating Greek Departments of Computer Science/Engineering using Bibliometric Indices
Dimitris Katsaros1,2
Vassilis Matsoukas1
Yannis Manolopoulos1
12th Panhellenic Conference on Informatics Samos Island, August 28-30, 2008
1Informatics Department, Aristotle University, Thessaloniki2Department of Computer & Communications Eng, University of Thessaly, Volos
2
Scientometrics: the Science of Measuring Science
• Careers in science are not scientific; they depend on: • luck• social connections• the ability to impress influential people and
referees • the foresight to join the right lab at the right
time• the foresight to associate oneself with
prestigious people and prestigious projects
• Such systems waste scientific talent and produce resentment
3
Scientometrics: the Science of Measuring Science
• Promotion strictly according to scientific merit would revolutionize scientific career
• Scientific production: the basis for any measurement of scientific merit
• Scientific production consists of:• published articles, and• their impact
• Scientometrics: the science of measuring the “quality” of science
4
Why Measuring Science?
• Numerical indices for quantification of published research output are being increasingly used by:• employers for hiring personnel• promotion panels – promotions, tenure
• funding agencies – “Funding does not regenerate funding. But reputation
does.”
• Australia: Research Quality Framework (RQF)• UK: Research Assessment Exercise (RAE)• Attempts to quantify the quality of science
are always fraught with difficulty, but it is worthwhile …
5
Early Approaches: the Impact Factor
• Eugene Garfield (Science, 1972) described the Impact Factor (IF) for journals:Impact factor for Journal X, 2007►A = # citations in all ISI articles during 2007 to
papers published in X during 2005–2006►B = # of articles published in X during 2005–2006►Impact Factor = A/B
• The IF is computed from data gathered by the Institute for Scientific Information (ISI), which publishes the Science Citation Index
6
Impact Factor Heavily Criticized …
• Few articles make the difference:• Philip Campbell – Editor-in-Chief of the journal
Nature – concerned about IF’s crudeness (ESEP, 2008):The value of Nature’s impact factor for 2004 was 32.2When he analyzed the citations of individual Nature papers over the relevant period (i.e., citations in 2004 of papers published in 2002 to 2003), he found that 89% of the impact factor was generated by just 25% of the papers!
7
Impact Factor Heavily Criticized …
• Why papers from a two-year period & citations from a single year:• John Ewing concerned about IF’s “parameters”
(NOTICES OF THE AMS, 2006):Looking at citations for only two years after
publication may produce faulty resultsIn some fields (e.g., mathematics) citations
frequently follow several years after publicationWhy two years?And why choose citations from journals published
in a single year?Both are somewhat arbitrary choices, without any
obvious justification
8
The Hirsch h-index
• Jorge Hirsch (PNAS, 2005) defined the h-index:• An author scores h if h of their N papers each
have at least h citations, with the remaining (N– h) papers each having fewer than h citations
• Quantifies both the actual scientific productivity and the apparent scientific impact of a scientist
• Resists to the power-laws followed by the evaluation metrics based on simple arithmetics
9
h-index graphical illustration
The intersection of the 45° line with the curve gives h
10
h-index example
• A scientist with 5 articles
• When ranked, have 6,4,4,2,1 citations
A Ferrers graph representation
The h-index is equal to the
length ofthe side of the Durfee square
11
h-index’s shortcomings
• is bounded by the total number of publications: hÉvariste Galois= 2, hAlbert Einstein=5
• does not consider the context of citations:• citations are often made simply to flesh-out an
introduction• citations made in a negative context• citations made to fraudulent or retracted work
• does not account for confounding factors• practice of "gratuitous authorship"• the favorable citation bias associated with
review articles
12
h-index’s shortcomings (2)
• is affected by limitations in citation data bases
• is a natural number and thus lacks discriminatory power
• does not account for the age of the articles and the age of citations
• does not account for the number of authors of a paper
• …and many more
13
The g-index by Egghe
• The h-index de-emphasizes singular successful publications in favor of sustained productivity. But, it may do so too strongly!
• Two scientists may have the same h-index, say, h = 30, but one has 20 papers that have been cited more than 1000 times and the other has none
• g-index (Scientometrics, 2006): the (unique) largest number such that the top g articles received (together) at least g2 citations
14
Age-decaying approaches to h-index
• The h-index favors senior scientists• Contemporary h-index (Scientometrics, 2007)
• A researcher has contemporary h-index hc if
• hc of his Np articles have Sc(i)≥hc
• the rest Np-hc articles have Sc(i)≤hc • Sc(i)=γ * (Y(now) - Y(i) + 1)-δ |C(i)|• Usually: γ=4 and δ=1
• The Age-Weighted Citation Rate (AWCR) and AW-index• AWCR measures the average number of citations to an
entire body of work, adjusted for the age of each individual paper
• An old article gradually loses its “value”• Show how “active” a researcher is
15
Accounting for multi-authorship
• Individual h-index • It divides the standard h-index by the average
number of authors in the articles that contribute to the h-index, in order to reduce the effects of co-authorship
• PoP Individual h-index• instead of dividing the total h-index, it first
normalizes the number of citations for each paper by dividing the number of citations by the number of authors for that paper, then calculates the h-index of the normalized citation counts
16
More extensions to the h-index …
• Can be extended to measure the performance of:• Research groups• Universities/institutions• Countries
• The notion of successive h-indexes• Scientometrics, 70(1), 2007: their definition• Scientometrics, 73(1), 2007: for countries• Scientometrics, 75(1), 2008: for institutions• JASIST, 59(8), 2008: for groups of authors• JASIST, to appear, 2008: mathematical foundation
17
Departmental h-index: the Greek Case
• the h-index of a specific department equals h if • h of his/her Np faculty members have a value of
h as h-index, and • the rest (Np−h) faculty members have no more
than h value as h-index
• Inspired by the current debate on Greek universities evaluation
• Confirm excellence• Discover patterns and exceptions
18
Data Collection & Processing
• Included 17 departments hosting 552 staff members of all ranks
• Excluded recently founded departments since they do not have an adequate number of staff members
• Paid special attention to avoid name ambiguities
• Use of the tool Publish or Perish
19
Results for the Dept of Informatics (AUTH)
20
Depts. of Computer Science/Engineering
21
The “toppers”
• NTUA, UAthens and UPatras stand high• UCrete
• is ranked 1st in (almost) all examined bibliometric indices, even though it is not the most productive (ranked 8th) due to its medium size (ranked 11th)
• is not very popular among the prospective students (ranked 14th), which might be attributed to its distance from the Greek mainland
22
Productivity and Impact Keep Pace
• UPatras (Dept of Elect & Comp Eng) and UThrace are on the opposite end
• UCrete and AUEB show an outstanding relative quality in comparison to their production
23
h-index & hc-index Keep Pace
AUEB, UAegean and UPiraeus demonstrate a deviate from the rule
Due to a recent successful recruitment policy?
24
More Profs Higher h-index
UCrete and UThessaly are positive exceptions to the rule
UThrace and UMacedonia are negative exceptions
25
Scientific excellence and popularity (among the prospective students) keep pace
UCrete and AUTH (Dept. of Informatics) seem under-preferred
UThrace, UMacedonia and AUTH (Dept. of Elect & Comp Eng) are over-preferred
26
Summary and Contributions
• Ranking of research institutions by bibliometric methods: popular tool used by• policy makers• public media • scientific world
• Departmental h-index: the Greek case• Marked the “best” department• Recorded some generic patterns • Discovered “outliers” to these patterns