Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of...

Italian Research Evaluation Exercise “VQR 2004‐2010”

and the

University of Milano‐BicoccaBonaria Biancu

University of Milano‐Bicocca

University of Milano‐Bicocca

• Young

(1998),

dynamic

and

multidisciplinary, offering Bachelors, Masters, PhDs

• 16 Departments

• +32,000 students• +900 scholars

and researchers

• 23rd in the Times

Higher

Education

World Ranking

of

universities

under 50 years

old

ICT Area ‐

University of Milano‐Bicocca 2

http://www.unimib.it/english

My

appointment:ICT Area ‐

Research

Information Systems

Office

http://www.timeshighereducation.co.uk/world-university-rankings/2013/one-hundred-under-fifty

http://www.timeshighereducation.co.uk/world-university-rankings/2013/one-hundred-under-fifty

http://www.unimib.it/english

CAVEAT

• Data published

on ANVUR website on July 31st, 2013

but…

• Still

unstable

/ missing

• Slides based on ANVUR final report (07.31.2013 version)

• Focus on rating, not ranking

• Slides based on ANVUR final report (07.31.2013 version)

• Focus on rating, not ranking

ICT Area ‐


VQR 2004‐2010

• Evaluation of research quality of Italianuniversities and departments (and otherresearch institutions)

• Starting date: November 7th, 2011

• Ending date: July 31st, 2012– June 2nd, 2013

• Publication of VQR results report: July 16th,2013

ICT Area ‐ University of Milano‐Bicocca 4

http://www.anvur.org/rapporto/

VQR 2004‐2010 Data Requests• Departments (pre‐ and post‐ University Reform Law and Bicocca

bylaws changes)• ‘Soggetti valutati’ i.e. academic staff in post on the date of the

census (11.7.2011)• 3 scientific products for every researcher (2 / 1 depending on the

scientific age or leaves; 6 for researchers with a joint appointmentboth with Bicocca and a research institution)

• Researchers/academics recruited or advanced• PhD, Specialization, Post‐doc students• Research and financial data:

– Income from competitive grants and contracts with private firms– University expenditure for research projects– Patents, spin‐offs, consortia– Other 3rd stream activities• Self‐evaluation Report


Subject Area indicatorsand weights

w = mean of 4 indices (research

products and costs)

Subject Area wj (%)1 Mathematics 4,00%2 Physics 5,67%3 Chemistry 7,80%4 Earth Science 2,10%5 Biology 10,75%6 Medicine 19,47%9 Computer Engineering 10,73%

10 Literature 7,66%11a Philosophy, Pedagogy 4,94%11b Psychology 1,97%12 Law 5,07%13 Economics 5,12%14 Sociology 2,72%

7, 8a, 8b

Agriculture; Civil Engineering, Architecture ‐12%

Tot.: 0,88

ICT Area ‐ University of Milano‐Bicocca6

• IRAS1: Publications and patents: 0.5• IRAS2: Income from competitive

grants: 0.1• IRAS3: Publications and patents of

researchers recruited/advanced: 0.1• IRAS4: Internationalization: 0.1• IRAS5: PhD/Spec/Post‐doc students:

0.1• IRAS6: Expenditure of the university

for research projects: 0.05• IRAS7: Improvement on VTR 2001‐

2003: 0.05

Internal procedure steps• Working group of technical and administrativestaff– ICT, Research, HR, Accounting&Budget,representative for Departments

• University and Departments’ Faculty Committeesto evaluate research products– Avoid submitting the same publication more thanonce

• All steps of publication selection and submissionmanaged via an internal CRIS software (SURplus)


Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials

Internal procedure steps

• Working group of technical and administrativestaff– ICT, Research, HR, Account&Financial, deptsrepresentative

• University and departments’ Committees to evaluate research products– Avoid presenting same publication twice

• All steps of publications selection and submissionmanaged via an internal CRIS software (SURplus)

Focus on publications

ICT Area ‐ University of Milano‐Bicocca

Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials

Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials 8

Selection and submission of publications

The Scholar

selects, using an

internal procedure

(CRIS module),3 to 20

publications

previously

registered in the

institutional

repository

The Rector + the

University Committee

verify that VQR

criteria are met, no

duplicates have been

selected

(following

negotiation) and give

the OK to the

submission

February to June 2012

ICT Area ‐

University of Milano‐Bicocca9

The Department

Committees verify

that every scholar has

completed the

procedure, VQR

criteria are met and

no duplicates have

been selected

(following

negotiation); finally

check the overall

quality of publications

Publication selection

ICT Area ‐ University of Milano‐Bicocca

10

+ setting thepriority i.e.ranking 3 up to20 publications

Additional information onselected publications


Additional information on selected publications

ICT Area ‐


Here InCites would have

helped! ‐> Having

normalized field citation

score ( / expected), IF

quartile, university

publications ranking

within Subect Areas etc.

Subject Categories

ISI/SCOPUS were used by

ANVUR to assign

publications to the correct

Panel and to evaluate

their impact factor and

citation score in context

Hopefully for the next VQRs we will save

time and perform even better…

PDF upload or send request to the publisher

ICT Area ‐


University/Dept Committees ViewChecking completeness and correctness


University/Dept Committee View Confirming or rejecting the proposed ranking

ICT Area ‐


Package of publications XML record

i.e. NO manualintervention by

academics other than inSURplus…


+ one ormore PDF

files…

…

Sent to ANVUR FTP server

… Hence major control over the procedure by the ICT staff

Overall FTE / publications

• 916 academic staff

• 21 departments– 16 after bylaws changes

• 2,309 products (over 2,339 expected)– Small number of low‐

or non‐productive staff

ICT Area ‐


Area# prodotti attesi (n)

% sul totale di Area

# prodotti M % su attesi# prodotti conferiti

# Prodotti P% su

conferiti

1 Mathematics 236 2,71 12 5,08 224 0 0,002 Physics 166 2,75 2 1,20 164 0 0,003 Chemistry 117 1,49 0 0 117 0 0,004 Earth Science 75 2,58 0 0 75 0 0,005 Biology 227 1,73 0 0 227 0 0,006 Medicine 292 1,06 1 0,34 291 4 1,37

missing Subject Areas 7 and 8a/8b (less than 10 products)9 Computer

Engin.33 0,24 2 6,06 31 0 0,00

10 Literature 55 0,39 0 0 55 0 0,0011.a Philosophy,

Pedagogy193 2,09 0 0 193 0 0,00

11.b Psychology 184 5,40 0 0 184 0 0,0012 Law 260 2,07 7 2,69 253 1 0,4013 Economics 360 2,94 6 1,67 354 0 0,0014 Sociology 130 3,09 0 0 130 0 0,00

TOTALE 2328 1,52 30 1,29 2298 5 0,22

Number of publications per Subject Area and % of missing / not suitable


Area# prodotti

attesi (n)

% sul totale

di Area

# prodotti M % su attesi# prodotti

conferiti

# Prodotti P% su

conferiti

1 Mathematics 236 2,71 12 5,08 224 0 0,002 Physics 166 2,75 2 1,20 164 0 0,003 Chemistry 117 1,49 0 0 117 0 0,004 Earth Science 75 2,58 0 0 75 0 0,005 Biology 227 1,73 0 0 227 0 0,006 Medicine 292 1,06 1 0,34 291 4 1,37

missing Subject Areas 7 and 8a/8b (less than 10 products)9 Computer

Engin.

33 0,24 2 6,06 31 0 0,00

10 Literature 55 0,39 0 0 55 0 0,0011.a Philosophy,

Pedagogy

193 2,09 0 0 193 0 0,00

11.b Psychology 184 5,40 0 0 184 0 0,0012 Law 260 2,07 7 2,69 253 1 0,4013 Economics 360 2,94 6 1,67 354 0 0,0014 Sociology 130 3,09 0 0 130 0 0,00

TOTALE 2328 1,52 30 1,29 2298 5 0,22

Number of publications per Subject Area and % of missing / not suitable

ICT Area ‐


«…

The percentage of missing products over expected ones (1,32%) is lower than the overall universities mean (4,68%), as is

the percentage of non evaluable products over the submitted ones (0,22% against an overall mean of 0,89%), thus denoting a particular care in selecting and submitting products»

ANVUR Final Report – Univ. of Milano Bicocca, page 4th

Quality of research publicationsScore of the publications

Area I Quartile1 0,7 Q22 0,82 Q23 0,89 Q14 0,63 Q25 0,76 Q16 0,65 Q19 0,53 Q4

10 0,75 Q111a 0,69 Q111b 0,66 Q112 0,54 Q213 0,4 Q114 0,6 Q1

7, 8a, 8b < 10 publications ‐

Subject Areas(size‐

independent):1st Quartile: 82nd Quartile: 44th Quartile: 1


Quality of research publicationsExcellent publications

Area X Quartile1 1,12 Q22 1,12 Q23 1,17 Q14 1,57 Q15 1,12 Q26 1,34 Q29 0,84 Q4

10 1,63 Q111a 1 Q211b 1,3 Q112 0,92 Q213 1,02 Q214 1,84 Q1

7, 8a, 8b < 10 publications ‐


Ratio ≥ 1for all but two Subject Areas

Quality of research publications Ratio of quantity and quality and

IRAS 1

Area R Quartile IRAS1 IRAS1*1001 1,17 Q2 0,031687403 3,162 1,06 Q2 0,029066606 2,93 1,12 Q1 0,016720134 1,674 1,11 Q2 0,028623661 2,865 1,24 Q1 0,021431242 2,146 1,39 Q1 0,014727496 1,479 0,73 Q4 0,001779182 0,18

10 1,14 Q1 0,004477044 0,4511a 1,19 Q1 0,02477823 2,4811b 1,2 Q1 0,064709183 6,4812 1,09 Q2 0,022529619 2,2513 1,24 Q1 0,036191855 3,6114 1,35 Q1 0,041398247 4,14

7, 8a, 8b < 10 publications ‐ ‐ ‐

Subject Areas(size‐

independent):1st Quartile: 82nd Quartile: 44th Quartile: 1

ICT Area ‐


ANVUR ranking(size‐dependent):

1st in the segment: 11st Quartile: 6

2nd‐3rd Quartile: 6

IRAS 3

• Mean score for recruited and advanced researchers within the Subject Area; mean score within the Subject Area /

BicoccaArea # Soggetti assunti o

promossi nel settennioCategoria Voto medio assunti o

promossi nel settennio /

Voto medio assunti o

promossi nel

settennio Italia (Rmobil.

Italia)

Posizione nel

segmento

dimensionale

Voto medio assunti o

promossi nel settennio /

Voto medio Area‐Bicocca

(Rmobil. Area)

1 47 Medie 1,14 5°

su 22 1,48

2 25 Medie 1,08 14°

su 20 1,16

3 18 Medie 1,07 9°

su 22 1,16

4 16 Medie 1,26 4°

su 17 1,51

5 50 Medie 1,11 6°

su 20 1,30

6 57 Piccole 0,99 11°

su 17 1,33

8 1 Piccole ‐ ‐ ‐

9 8 Piccole 0,58 14°

su 16 0,66

10 15 Piccole 1,07 6°

su 18 1,16

11 98 Grandi 1,05 2°

su 12 1,14

12 62 Medie 1,00 23°

su 30 1,15

13 87 Grandi 1,11 11°

su 36 1,41

14 31 Grandi 1,26 1°

su 15 1,38

Totale 515

Ratio ≥

1

for all but one

Subject Areas

ICT Area ‐


IRAS 1‐7 IRFS 1 and IRFS 2

Università Prodotti attesi

% Prodotti attesi sul totale

IRFS1 x100

16 aree

IRFS2 x100

16 aree

IRFS1 x100 14 aree

IRFS2 x100 14 aree

Milano Bicocca 2.339 1,51995 1,95428 1,94394 1,94902 1,93740

Università

% Prodotti attesi sul

totale delle Università

IRAS1 x w x 100

IRAS2 x w x 100

IRAS3 x w x 100

IRAS4 x w x 100

IRAS5 x w x 100

IRAS6 x w x 100

IRAS7 x w x 100

Milano Bicocca 1,51995 1,71124 1,83654 2,03698 3,04352 1,65572 2,45750 2,37015

IRAS 1‐7

IRFS 1 and IRFS 2

Actually Bicocca governmental budget is underestimated

Here lies the importance of evaluation!


This talk is in loving

memory ofLuisella Sironi,THEBIGBOSS

of VQR Working Group


Thank you!

Bonaria BiancuICT Area – University of Milano‐Bicocca

Via Cozzi, 53 – Ed. U520125 Milan (Italy)

Website: http://www.unimib.itEmail: [email protected] phone: +39.0264486236Mobile phone: +39.3386940012

Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of...

Documents

Transcript of Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of...