Evaluating research on a massive scale in the ANVUR experience. Approaches and controversies Andrea...

Evaluating research on a massive scale in the ANVUR experience.

Approaches and controversies

Andrea BonaccorsiBoard of ANVUR

In collaboration with Marco MalgariniDirector, Research Assessment

2

Outline

1.The Italian National Agency for the Evaluation of Universities and Research

Institutes

2.The Research Evaluation for the period 2004-2010 (VQR 2004-2010)

3.The National Scientific Habilitation Program

4.The ANVUR bi-annual Report on the Italian Research system

5.Classification of journals in Humanities and Social Sciences

3

The Italian National Agency for the Evaluation of Universities and Research Institutes

According to the legislation established in 2010, the Italian National Agency for the

Evaluation of Universities and Research Institutes (ANVUR) is in charge of:

•External evaluation of the quality of activities of Universities and Public Research

Organisations (PROs)

• (a) accreditation and quality assurance of academic courses according to

European Guidelines (ENQA)

• (b) research assessment

•

•Giving general advices for the activities of internal evaluation bodies

•Evaluate public funding programs aimed at enhancing research and innovation

activities

4

The Italian National Agency for the Evaluation of Universities and Research Institutes

More specifically, since 2011 ANVUR has been involved in several projects of

research assessment or which have used the results of research assessment

•The quality of research evaluation exercise for the period 2004-2010 (VQR 2004-

2010, or Valutazione della Qualità della Ricerca), results of which have been

presented on July 16, 2013

•The National Habilitation program, which is currently completed (1st wave)

•The bi-annual Report on the Italian Research system, published in March 2014

•The accreditation of doctoral courses, due to be published in June 2014.

5

The Evaluation of Research 2004–2010 (VQR 2004-2010)

The Decree regulating the second Italian research quality evaluation exercise has been

issued in 2011.

According to the Decree, all the Italian Assistant, Associate and Full Professors participate

to the evaluation exercise, together with Researcher and Technicians of PROs directly

funded by the Ministry of University and Education (MIUR)

Evaluation refers to Areas of research, Universities and Departments – it does not

concern individual researchers

Evaluation has already affected the distribution of public funding to the Italian research

system (2013).

6

The Evaluation of Research 2004–2010 (VQR 2004-2010)

Each University has chosen the three best research products (six products for PROs)

published in the 2004-2010 period by each researcher

Research products considered for evaluation are:

•Articles

•Books, books chapters and proceedings (with ISBN code)

•Critical editions, translations, scientific comments

•Patents

•Design, Performance, Dataset, Software and other research products

Research evaluation is classified according to official organisation of Scientific Areas of

research established by the National University Council (CUN)

7

Italian Scientific Areas of Research

•More specifically, Italian academia is officially divided up in 14 Scientific Areas (Aree CUN)•The 14 Areas may be distinguished into:•Areas for which is common to evaluate research quality on the basis of bibliometric indicators regarding the impact of the Journal on which the article is published and the number of citations of the article (so called Bibliometric sectors)•Mathematics (Area 1)•Physics (Area 2)•Chemistry (Area 3)•Earth Sciences (Area 4)•Biology (Area 5)•Medicine (Area 6)•Agricultural and Veterinarian Sciences (Area 7)•Part of Architecture and Civic Engineering (Area 8)•Industrial and communication Engineering (Area 9)

Humanities and Social Sciences•Among Social Sciences, bibliometric indicators are also used in Economics and Statistics (Area 13) and in a small part of Psychology (Area 11)

8

Institutions and experts involved

•Institutions involved in the VQR 2004-2010 include:•67 + 28 public and private Universities•12 PROs under the control of the Ministry of Education and University•27 Research Institutes, Consortia and Foundations which have voluntary submitted an application to be evaluated by ANVUR

•Evaluation has been performed by:•450 national and international Scientific experts from all the 14 Areas of Research•Almost 14,500 national and international reviewers

9

Research products submitted for evaluation

•A total of 184,742 research products have been submitted for evaluation•Among them we find:•73,5% of review articles•19,9% of books, books chapters•5,8% proceedings, translations, scientific comments•0,9% miscellaneous other products

•Research products may be classified according to the area to which they are submitted for evaluation and to the area of the subject submitting them for evaluation

10

Evaluation Methods and Criteria

•Evaluation is based on a combination of bibliometric and peer review methods

•In any case, overall at least 50%+1 of the products of research submitted for evaluation is subject to peer review (by legal obligation in the decree creating ANVUR)

•In the bibliometric sectors defined above, there is an intense use of bibliometric methods

•Bibliometric evaluation concerns research products indexed either in Scopus or Web of Sciences databases

11

The bibliometric method of evaluation

•According to the VQR definition, a research product may be defined as follows:•Excellent, when it falls in the top 20% in terms of quality according to the international scientific community in the field•Good, when it falls in the 60-80% segment in terms of quality according to the international scientific community in the field•Adequate, when it falls in the top 50-60% segment in terms of quality according to the international scientific community in the field•Limited, when it falls in the bottom 50% in terms of quality according to the international scientific community in the field

•According to the aforementioned criterium:

A product of research randomly extracted in an homogeneous set of all the products published in the world in a given sector and a given year has a

probability of being classified as Excellent, Good, Fair or Limited respectively equal to 0,2; 0,2; 0,1; 0,5

12

The calculation of bibliometric indicators

Threshold (# citations) for each evaluation class for some subject category according

to the ISI database, year 2009

•According to this methodology, the classification of research products is performed in two steps:

1.Definition of the appropriate bibliometric indicators to be used2.Definition of an appropriate algorithm assigning the value of the chosen indicator(s) to one of the defined class of merit (E; G; A; L)

13

The calculation of bibliometric indicators

•The indicators chosen for the VQR 2004-2010 are the following:

1.An indicator of the impact of the journal on which the research product has been published, such as Scimago Journal Rank or the Journal Impact Factor. Such an indicator provides information on the quality of the publication linked to:a)Referee procedureb)Reject ratec)Diffusion of the journald)Reputation of the Journal

2.The number of citations received by the research product

14

The bibliometric algorithm - 1

•For each article, we first identify the Journal, the year of publication and the scientific field

•We then calculate the empirical cumulative distribution of the SJR for all the Journals in a given scientific field in a given year

•The distribution is divided up in four areas, with probability of inclusion respectively equal to 0,2; 0,2; 0,1 and 0,5

1

IF

0.6

0.8

0.5

Th3 Th2 Th1

Cum. Distrib.

15


•We then calculate the empirical cumulative distribution of the number of citations for all the articles in a given scientific field in a given year

•Also this distribution is divided up in four areas, with probability of inclusion respectively equal to 0,2; 0,2; 0,1 and 0,5

# of citations

1

0.6

0.8

0.5

Th3 Th2 Th1

Cum. Distrib.

16


•As a result, we have two different classifications respectively based on the quality of the Journal (as measured by the SJR) and the number of citations of the paper•Starting from these information, we build up the 4x4 matrix showed here•When the two indicators give equal indications, the class is settled•When the two indicators give diverging signals, we may choose to give an higher weight either to the impact or to the citational indicator

A

B

C

D

# Citations

1

2

3

4

1 2 3 4

Bibliometry (SJR)

17


•If classification according to the two criteria diverges, the final class may depend:–On choosing whether to give more emphasis to one of the two criteria–On the value of the difference among the two different classifications•As an example, the matrix on the right present a possible classification for older articles, in which more emphasis is given to the number of citations criteria•The “U” cell will be attributed with “informed peer review”•The choice about the allocation of undetermined cells is left to the responsibility of each group of experts for each scientific disipline

A A U UB B B UU C C CU U D D

# Ci

tatio

ns

1

2

3

4

1 2 3 4Bibliometry (IF,…)

18

Impact of VQR

• Funding of universities• Share of block granting (Fondo di Finanziamento Ordinario, FFO) allocated

according to the relative merits of universities as stated by VQR• Decree August 2013- share will increase steadily in following years• However, current maximum change +/- 5% with respect to previous year• Decree 2014 under approval

• Internal allocation of resources of universities• Use of VQR indicators by department and/or scientific areas used to

allocate resources (e.g. PhD grants, post-doc grants, research funds)• Use for allocation of new positions more sensitive

• Accreditation of doctoral courses• Reform of doctoral courses aimed at strenghtening the international

orientation and rationalizing the offer• Accreditation criteria include the «scientific quality of the faculty»• ANVUR is responsible for accreditation- used VQR data to evaluate the

quality of members of the faculty

19

Critical issues in Humanities and Social Sciences

• Peer review• Need to test extensively the Evaluation form before submitting to referees• Qualitative comments should be mandatory and communicated to the

evaluated subjects• List of referees made public

• Meaning of internationalization• Not all disciplines share the same meaning of «internationalization»• Avoid an interpretation by referees as a negative score on products in

Italian language and/or in Italian journals/ books• Yet there is no evidence of a systematic bias

• Classification of journals in the area of Economics and Business• Journals classified according to the percentiles of the world distribution of

IF in four broad areas (Economics, Accounting and management, History, Mathematics and Statistics)

• No Italian journals in class A; only 20 in class C• Yet the low score is mainly due to books and book chapters

20

The National Habilitation

•Law n. 240/2010 has innovated the recruitment procedures for Italian Academia

•Candidates aspiring to become Professor in Italy are now selected through a two stage procedure:•In a first stage, they apply to be granted a National Habilitation to become either Associate or Full Professor in their scientific field•Only those being granted the National Habilitation may participate to local selection procedures promoted by Universities

•In 2012, Ministerial Decree n. 76 has established criteria and parameters for the National Habilitation

•In July 2012, the Italian Ministry of Education has started the selection procedure for the National Habilitation Program

21

The National Habilitation Program

•According to the Ministerial Decrees, ANVUR:

•Defines parameters and criteria to be used for the selection of both Full Professors participating to the Commissions and of candidates in the National Habilitation Program

•Selects the Italian and Foreign candidates to be member of the sector-specific Commissions in charge of granting the National Habilitation

22

The National Habilitation Program: parameters and indicators

•In the bibliometric areas the indicators are:

•Number of articles published either in the Scopus or ISI-Web of Science databases in the last 10 years•Number of citations received over the entire academic career of the candidate from all indexed sources•H-index

In the non-bibliometric sectors the indicators are:• Number of books• Number of book chapters and journal articles• Number of articles in A-rated journals

• All the above indicators are normalised for the academic age of the candidate. The academic age is defined as the number of years since the first publication included in the submission

•Normalization of h-index is fone using the contemporary h-index formula

23

The National Habilitation Program: parameters and indicators

•For each indicator, the Median values of the distributions for the two Populations of Italian Associate and Full Professors are calculated

•According to the Decree, to be part of the Commissions granting the National Habilitation an Italian or Foreign Full Professor should fall in the top 50% of the distribution in two out of three indicators, calculated for the Population of Italian Full Professors (only one for Humanities and Social Sciences)

•The same criterium applies for the selection of candidates to be appointed as Associate or Full Professors, respectively

•However, according to the Decree, Judging Commissions are allowed to grant Habilitation also to candidates not fully complying to these requirements, provided that the decision is adequately motivated ex ante (= before opening the list of candidatures)

24

The National Habilitation Program: the information set

•The information set on which the indicators are calculated is obtained from the combination of:

•The Archive of all the Italian Associate and Full Professors. Data are drawn from access to personal websites of professors, held by the Minister of University and Research and managed by a university consortium of IT infrastructure and services (Cineca)

•Bibliometric information obtained from the Scopus and ISI-Web of Science databases

•Distributions are calculate separately for Associate and Full Professors, with reference to the each specific Italian Scientific Sector (SSD, i.e. Sector Specific Scientific Sector)

25

The National Habilitation Program: calculation of indicators

•On the basis of the archive of Italian Professors, on July 15, 2012, the number of publications in bibliometric sectors was equal to 1,316,000 research products

•The procedure of calculation of the aforementioned indicators can be divided in three major steps:1.Matching of authors listed in the Italian archive with the Scopus database, on the basis of the surname, initial of the name and affiliation. A specific algorithm was used in order to:a)Minimize errors due to mistakes in the Italian archiveb)Minimize errors due to mistakes in the Scopus database (in particular, with reference to the author’ surname)c)Minimize errors due to ambiguity in surnames (especially for composite surnames)d)Maximize identification of authors not recognised in the Scopus database2.Extracting bibliometric information from the matched dataset

26

The National Habilitation Program: calculation of indicators

3. Calculating the three bibliometric indicators:a)On the period 2002-2012, given that it has been impossible to correctly distinguish the papers published after July 2002, as it was originally requested from the Decreeb)For each Scientific Sector, separately for Associate and Full Professorsc)For the citation indicators, it has been chosen the max number of citations available for each article on the basis of the two available datasets

•At the end of the procedure, the percentage of articles contained in the Italian archive correctly identified in the bibliometric datasets was equal to 96%•Correct identification is lower for conference proceedings (32%, with higher levels where proceedings are particularly relevant, such as mathematics – 52% - and engineering – 45%)

27

The National Habilitation Program: selection of the Commissions

•ANVUR has also been responsible for ascertain the scientific qualification of Italian and Foreign Full Professors aiming to become part of the Commissions granting the National Habilitation•ANVUR activities in this field may be distinguished according to the affiliation (National or International) of the candidates•For National candidates ANVUR took care of:1.Calculation of individual indicators on the basis of the information contained in the dataset on July 21st, 20122.Communication to the candidate of a first assessment (“green” or “red” light) on the basis of this information, leaving the possibility to withdraw the candidature before August 28, 20123.New calculation of the indicators on the basis of the information available on August 28, 20124.In case of a negative assessment, candidates have been notified of the decision and informed about the calculations. Candidates had the possibility of appealing to the decision and ANVUR has re-calculated the indicators for those providing new evidence about their publication activities

28

The National Habilitation Program: selection of the Commissions

•For Foreign candidates ANVUR activities were the following:

•Creation of a list of candidates for the position, following the publication of an invitation to apply on Nature

•Manual calculation of the bibliometric indicators: only those with a value of two out of three indicators above the median of the sector have been admitted to a random draw selecting the Foreign Member of the Commission

29

Classification of journals

In Humanities and Social sciences the indicators required the identification of:- scientific journals (as opposed to popular, professional, cultural or other journals)- A-rated journals

Classification of journals has been done in several waves:- startup (based on lists of journals on the personal website of professors and researchers)- appr. 20,000 titles, of which 15,000 considered scientific (titles were evaluated separately in each of scientific disciplines in which the authors were active, summing up > 42,000 occurrences) (July-September 2012)- new journals included by candidates to the Habilitation in their publication list- appr. 9,000, of which 3,000 considered scientific (January-March 2013)- revision of rating- a procedure aimed at allowing journals to apply for upgrade- 550 applications, for 2,200 positions overall. Evaluation largely based on peer review (procedure close in June 2013)- new journals included by candidates to the Habilitation in their publication list- appr. 3,000 new items (January-March 2014)

A case study. Number of journals registered in personal archives of researchers and professorsArea 10 (Humanities and Arts)

July 15, 2012

Number of journals (with repetition) 10.989

Number of journals (without repetition) 5.000

Number of articles registered in personal archives 42.157

● Distribution of journals by number of articles

Number of journals Number of articles

More than 100 articles 16 3.109

10-99 articles 909 19.059

Less than 10 articles 10.064 19.989

Total 10.989 42.157

0

100

200

300

400

500

600

122

645

167

690

111

2613

5115

7618

0120

2622

5124

7627

0129

2631

5133

7636

0138

2640

5142

7645

0147

2649

5151

7654

0156

2658

5160

7663

0165

2667

5169

7672

0174

2676

5178

7681

0183

2685

5187

7690

0192

2694

5196

7699

0110

126

1035

110

576

1080

1

Numero di articoli

Distribuzione delle riviste per numero di articoli per SSD

Disciplinary field

Number of journals

Number of articles

Average number of articles per journal

10/A1 370 1943 5,2510/B1 178 824 4,6310/C1 192 1005 5,2310/D1 168 914 5,4410/D2 175 1063 6,0710/D3 145 831 5,7310/D4 176 936 5,3210/E1 217 875 4,0310/F1 324 1279 3,9510/F2 147 592 4,0310/F3 222 880 3,9610/G1 309 1279 4,1410/H1 154 676 4,3910/I1 162 569 3,5110/L1 206 760 3,6910/M1 124 371 2,9910/M2 61 205 3,3610/N1 184 788 4,2810/N3 70 214 3,06

Totale 3584 16004 4,47

34


Procedure

Creation of an Expert Group- Areas covered: Architecture (4 experts), Literature and Art (8), Philosophy and History (4), Law

(4), Economics and Business (4), Social sciences (4)- 2-year appointment- Part time appointment (only reimbursement of expenses)

Triangulation approach- Collect data on institutional repositories of Italian academics on journal lists (June 2012)- Ask opinion of scientific societies (July 2012)- Compare opinion of scientific societies with referees (July-August 2012)- Suggest consensu opinion by experts (September 2012)

35


Revision of ratingOpening of a procedure for the revision of rating• Annual procedure• Platform open to editors of journals (April-June 2013)• Questionnaire to be filled• Expert group in charge of revision- more intense use of anonymous referees• Text of referee report made available to editors

Second revision (2014)• Complete renewal of the Expert group• Better coverage of disciplines• Revision platform available since September 24, 2014

36


Robustness of the classification as scientific journal

Criteria for scientific nature (not defined in legislation)- Existence of a scientific committee- Editor (academic vs non academic)- Existence of an international scientific committee- Audience (academic, professional, popular or general public)- Formato of articles (lenght, references, footnotes)

Peer review generally considered a sufficient condition for scientific nature

Questionnaire addressed to editors in the revision procedure (April- June 2013)

Issues- Errors in initial classification (due to time pressure) in September 2012- several clearly non-

scientific journals (e.g, newspapers or popular science journals) wrongly included in the list- Media campaign (up to front page covering by Corriere della Sera)- Ex-officio correction of errors (publication in March 2013)- Few complaints about cancelation from the list of scientific journals (to date, one journal in

business, one in public law)- No legal complaints about the scientific nature of journals- No media attention or complaints after corrections of errors

37


Robustness of the A-classification

Criteria for A-rating (loosely defined in legislation)- Scientific nature- Peer review- Impact on the scientific community- Reputation

Questionnaire addressed to editors in the revision procedure (April- June 2013)

Operationalization- Nature of the peer review procedure (double-blind vs single-blind; referees only within the

scientific committee or in the larger scientific community; communication to authors; publication of the list of referees; rate of acceptance of manuscripts)

- Diffusion in libraries (national and international)- Reference by scientific associations- Questionnaire to referees (typically 3 per journal)

38


Controversies about the A-classification

Legal controversies- Some complaints to the Administrative Tribunal (TAR) tried to cast doubt on the overall

procedure, on grounds of excess power, irrationality, violation of principles of equity and trasparence in administrative action

- The Tribunal rejected the opposition (March 2014)- Other complaints addressed, on the contrary, specific evaluations, claiming there was lack of

justification. These complaints mostly come from journals in Law and History (mostly Contemporary History)

- In some cases the Tribunal forced ANVUR to repeat the evaluation. ANVUR nominated a different group of referees (typically, 3 professors different from those who proposed the initial classification). In most cases the classification was confirmed, in a few cases it was changed

Political controversies- Some scientific societies (mainly in part of the Humanities) have argued against the overall

journal classification system - After the revision procedure (2013) intense correspondence with editors of non-A journals

39

Special projects for Humanities and Social sciences

Humanities and Social Sciences do not have bibliometric indicators.

Special projects at ANVUR:- support HSS Italian journals wishing to apply for admission into WoS or Scopus- create a Database of Italian journals aimed at extracting citational information

Technical feasibility for a Database:- IT platform- delivery of full text- current issues + backlog (10 years)- automatic extraction of citations from the text of articles, via sophisticated computational linguistics system of rules- indicators: (a) production (b) use (c) citations- Altmetrics under experimentation- test to be carried out on A-rated journals

Workshop January 20, 2014Consultation with scientific societies and editors of A-rated journals

40

Research in Humanities and Social sciencesJune 2014

Creation of High Level Research Groups with a mission of• debating hot issues• listening to opinions (including scientific societies)• examining the international state of the art• producing recommendations

Set of research questions to be addressedStart June 2014Areas:- Humanities- Law- Management (to be decided)

Research Call- Research projects to be carried out by individual researchers or teams- First call (September 2014) on evaluation in HSS- 12 projecte received- selection by end of October 2014- Future call on evaluation of teaching + advancements in bibliometrics

41

The ANVUR Report on the Italian Research System

•ANVUR delivered in March 2014 its first Report on the Italian Research System

•The Report contains detailed analysis of the situation of the Italian Research system, in terms of demand and supply for higher education, academic staff, labor market outcomes and other indicators•The Report also contains a chapter devoted to the analysis of the position of the Italian research system in the international scenario•The benchmarking analysis is based on standard bibliometric information stemming from the major international databases in the field

•In this respect, choices have been made regarding:•The level of aggregation of the analysis•The countries involved in the comparison•The indicators used in the analysis•The period covered in the report

42

The ANVUR Report on the Italian Research System: level of aggregation

The report analyses the position of Italian research at 3 different levels of aggregation:•At the level of the Scopus Subject Areas•At the level of the Italian “Aree CUN”•For all the scientific sectors considered together

•In order to be able to gather relevant information at the level of the Italian «Aree CUN», ANVUR has realised a mapping of the Scopus subject areas into the Italian classification•The mapping has been based on information provided within the VQR 2004-2010:•We exclude from the calculation the subject areas in which less than 10 products have been submitted for evaluation•We calculate the number of publications presented in each Scopus subject area from each Italian “Area CUN”•We assign each Scopus subject area to the “Area CUN” with the highest share in that Scopus s.a.•In case the highest share is below the 75% threshold, we considered the possibility of multiple assignment if the second highest share is equal or superior to 20% (N.B: a subject area is assigned at most to 2 Aree CUN)

43

The ANVUR Report on the Italian Research System: countries involved in the comparison

The analysis referred to the following countries and aggregation of countries:

European countries and aggregatesEu 15 aggregateoItalyoFranceoGermanyoNetherlandsoSpainoUnited KingdomoSwedenEuropean Union (EU 27) aggregateOther European: Switzerland

44

The ANVUR Report on the Italian Research System: countries involved in the comparison

The analysis referred to the following countries and aggregation of countries:

OECDOECD aggregateoAustraliaoCanadaoJapanoSouth KoreaoUS Non-OECDBRIC aggregateoBrasiloChinaoIndiaoRussiaWorld total

45

The ANVUR Report on the Italian Research System: indicators

The analysis will be based on the following indicators:

Indicators measuring research productivity•Publications (whole)•Publications (fractional)Indicators measuring the impact of published research•Citations (whole)•Citations (fractional)•Citations per paper•Field weighted citation impactIndicators measuring the level of national and international collaboration:•Collaborations – single authorship•Collaborations – domestic collaborations•Collaborations – international collaborations•Collaborations – cross sector collaborationsIndicators measuring the quality of publication activities in terms of excellence:•The full percentile distribution of the number of publications with respect to the most cited publications.

•All the information will be available for the period 1996-2012

46


The Report addresses the following questions:

•What is the position of the Italian research system vis-à-vis its main competitors in terms of:•Scientific production (number of products)•Scientific impact (number of citations)

•How Italy compares with its main competitors in terms of publications in top rated journals?

•How Italy compares in terms of collaborating activity?

•What has been the evolution of relative scientific specialisation in Italy in the last 20 years

47


In order to answer those questions, bibliometric indicators will be combined with information stemming from official sources in order to calculate appropriate productivity indicators

More specifically, research production and impact will be assessed in terms of:•Production and impact per capita, dividing by (active) population•Production and impact per researcher, dividing by the number of (public and private) researchers•Production and impact per euro invested in R&D activity, dividing by R&D (Private, Governement, Higher Education) expenditures

Indicators of relative specialisation will also be calculated in order to investigate on the pattern of specialisation of the Italian research system

The report has been published in March, 2014.

Evaluating research on a massive scale in the ANVUR experience. Approaches and controversies Andrea...

Documents

Transcript of Evaluating research on a massive scale in the ANVUR experience. Approaches and controversies Andrea...