Evaluating Greek Departments of Computer Science/Engineering using Bibliometric Indices

Dimitris Katsaros1,2

Vassilis Matsoukas1

Yannis Manolopoulos1

12th Panhellenic Conference on Informatics Samos Island, August 28-30, 2008

1Informatics Department, Aristotle University, Thessaloniki2Department of Computer & Communications Eng, University of Thessaly, Volos

2

Scientometrics: the Science of Measuring Science

• Careers in science are not scientific; they depend on: • luck• social connections• the ability to impress influential people and

referees • the foresight to join the right lab at the right

time• the foresight to associate oneself with

prestigious people and prestigious projects

• Such systems waste scientific talent and produce resentment

3

Scientometrics: the Science of Measuring Science

• Promotion strictly according to scientific merit would revolutionize scientific career

• Scientific production: the basis for any measurement of scientific merit

• Scientific production consists of:• published articles, and• their impact

• Scientometrics: the science of measuring the “quality” of science

4

Why Measuring Science?

• Numerical indices for quantification of published research output are being increasingly used by:• employers for hiring personnel• promotion panels – promotions, tenure

• funding agencies – “Funding does not regenerate funding. But reputation

does.”

• Australia: Research Quality Framework (RQF)• UK: Research Assessment Exercise (RAE)• Attempts to quantify the quality of science

are always fraught with difficulty, but it is worthwhile …

5

Early Approaches: the Impact Factor

• Eugene Garfield (Science, 1972) described the Impact Factor (IF) for journals:Impact factor for Journal X, 2007►A = # citations in all ISI articles during 2007 to

papers published in X during 2005–2006►B = # of articles published in X during 2005–2006►Impact Factor = A/B

• The IF is computed from data gathered by the Institute for Scientific Information (ISI), which publishes the Science Citation Index

6

Impact Factor Heavily Criticized …

• Few articles make the difference:• Philip Campbell – Editor-in-Chief of the journal

Nature – concerned about IF’s crudeness (ESEP, 2008):The value of Nature’s impact factor for 2004 was 32.2When he analyzed the citations of individual Nature papers over the relevant period (i.e., citations in 2004 of papers published in 2002 to 2003), he found that 89% of the impact factor was generated by just 25% of the papers!

7

Impact Factor Heavily Criticized …

• Why papers from a two-year period & citations from a single year:• John Ewing concerned about IF’s “parameters”

(NOTICES OF THE AMS, 2006):Looking at citations for only two years after

publication may produce faulty resultsIn some fields (e.g., mathematics) citations

frequently follow several years after publicationWhy two years?And why choose citations from journals published

in a single year?Both are somewhat arbitrary choices, without any

obvious justification

8

The Hirsch h-index

• Jorge Hirsch (PNAS, 2005) defined the h-index:• An author scores h if h of their N papers each

have at least h citations, with the remaining (N– h) papers each having fewer than h citations

• Quantifies both the actual scientific productivity and the apparent scientific impact of a scientist

• Resists to the power-laws followed by the evaluation metrics based on simple arithmetics

9

h-index graphical illustration

The intersection of the 45° line with the curve gives h

10

h-index example

• A scientist with 5 articles

• When ranked, have 6,4,4,2,1 citations

A Ferrers graph representation

The h-index is equal to the

length ofthe side of the Durfee square

11

h-index’s shortcomings

• is bounded by the total number of publications: hÉvariste Galois= 2, hAlbert Einstein=5

• does not consider the context of citations:• citations are often made simply to flesh-out an

introduction• citations made in a negative context• citations made to fraudulent or retracted work

• does not account for confounding factors• practice of "gratuitous authorship"• the favorable citation bias associated with

review articles

12

h-index’s shortcomings (2)

• is affected by limitations in citation data bases

• is a natural number and thus lacks discriminatory power

• does not account for the age of the articles and the age of citations

• does not account for the number of authors of a paper

• …and many more

13

The g-index by Egghe

• The h-index de-emphasizes singular successful publications in favor of sustained productivity. But, it may do so too strongly!

• Two scientists may have the same h-index, say, h = 30, but one has 20 papers that have been cited more than 1000 times and the other has none

• g-index (Scientometrics, 2006): the (unique) largest number such that the top g articles received (together) at least g2 citations

14

Age-decaying approaches to h-index

• The h-index favors senior scientists• Contemporary h-index (Scientometrics, 2007)

• A researcher has contemporary h-index hc if

• hc of his Np articles have Sc(i)≥hc

• the rest Np-hc articles have Sc(i)≤hc • Sc(i)=γ * (Y(now) - Y(i) + 1)-δ |C(i)|• Usually: γ=4 and δ=1

• The Age-Weighted Citation Rate (AWCR) and AW-index• AWCR measures the average number of citations to an

entire body of work, adjusted for the age of each individual paper

• An old article gradually loses its “value”• Show how “active” a researcher is

15

Accounting for multi-authorship

• Individual h-index • It divides the standard h-index by the average

number of authors in the articles that contribute to the h-index, in order to reduce the effects of co-authorship

• PoP Individual h-index• instead of dividing the total h-index, it first

normalizes the number of citations for each paper by dividing the number of citations by the number of authors for that paper, then calculates the h-index of the normalized citation counts

16

More extensions to the h-index …

• Can be extended to measure the performance of:• Research groups• Universities/institutions• Countries

• The notion of successive h-indexes• Scientometrics, 70(1), 2007: their definition• Scientometrics, 73(1), 2007: for countries• Scientometrics, 75(1), 2008: for institutions• JASIST, 59(8), 2008: for groups of authors• JASIST, to appear, 2008: mathematical foundation

17

Departmental h-index: the Greek Case

• the h-index of a specific department equals h if • h of his/her Np faculty members have a value of

h as h-index, and • the rest (Np−h) faculty members have no more

than h value as h-index

• Inspired by the current debate on Greek universities evaluation

• Confirm excellence• Discover patterns and exceptions

18

Data Collection & Processing

• Included 17 departments hosting 552 staff members of all ranks

• Excluded recently founded departments since they do not have an adequate number of staff members

• Paid special attention to avoid name ambiguities

• Use of the tool Publish or Perish

19

Results for the Dept of Informatics (AUTH)

20

Depts. of Computer Science/Engineering

21

The “toppers”

• NTUA, UAthens and UPatras stand high• UCrete

• is ranked 1st in (almost) all examined bibliometric indices, even though it is not the most productive (ranked 8th) due to its medium size (ranked 11th)

• is not very popular among the prospective students (ranked 14th), which might be attributed to its distance from the Greek mainland

22

Productivity and Impact Keep Pace

• UPatras (Dept of Elect & Comp Eng) and UThrace are on the opposite end

• UCrete and AUEB show an outstanding relative quality in comparison to their production

23

h-index & hc-index Keep Pace

AUEB, UAegean and UPiraeus demonstrate a deviate from the rule

Due to a recent successful recruitment policy?

24

More Profs Higher h-index

UCrete and UThessaly are positive exceptions to the rule

UThrace and UMacedonia are negative exceptions

25

Scientific excellence and popularity (among the prospective students) keep pace

UCrete and AUTH (Dept. of Informatics) seem under-preferred

UThrace, UMacedonia and AUTH (Dept. of Elect & Comp Eng) are over-preferred

26

Summary and Contributions

• Ranking of research institutions by bibliometric methods: popular tool used by• policy makers• public media • scientific world

• Departmental h-index: the Greek case• Marked the “best” department• Recorded some generic patterns • Discovered “outliers” to these patterns