Quality assurance practices and University evaluation systems

Stefano FantoniPresident, ANVUR

Euro-Mediterranean University Network TETHIS meetingUniversity of Palermo 10°-11° of April 2014

2

THE ITALIAN NATIONAL AGENCY FOR THE EVALUATION OF UNIVERSITIES AND RESEARCH

INSTITUTES

ANVUR is the Italian National Agency for the Evaluation of Universities and Research

Institutes. It started operating in May 2011.

ANVUR is in charge of

• Evaluating the quality of processes, results, and products of all the activities concerning

research, teaching and technological transfer

• Defining methods and criteria to be used for the periodical accreditation and evaluation

of Universities (undergraduate, postgraduate and Ph.D. courses)

• Providing general advices to internal evaluation bodies

• Evaluating public funding programs aimed at research and innovation activities

• Every two years, ANVUR will edit a general Report on the italian Higher Education status

3

THE ITALIAN NATIONAL AGENCY FOR THE EVALUATION OF UNIVERSITIES AND RESEARCH

INSTITUTES

ANVUR, in its evaluation process, refers to the Quality Assurance system as described in

European Standard Guidelines ESG/ENQA 2005/2009.

• ANVUR has performed the research evaluation exercise for the period 2004-2010 (VQR

2004 - 2010, Valutazione della Qualità della Ricerca), whose results have been

presented in July 2013

• ANVUR is currently performing the accreditation and assessment program for courses

and institution (AVA, Accreditamento, Valutazione e Autovalutazione)

• ANVUR is currently developing the accreditation and the evaluation models for the PhD

courses.

4

THE CONTEST OF THE EVALUATION

PUBLIC FUNDING

COURSES EVALUATION

MANAGEMENT EVALUATION

Evaluation of Research

INCOMING RESEARCHER

S EVALUATION

Evaluation of Universities

PLACEMENT & TRANSITION TO

WORK

5

Each university professor or researcher has been asked to submit three research products (six instead of

three to the researchers of the Research Institutes) published in the 2004-2010 period. Each research

product (for a total of about 180 thousands) has been evaluated by bibliometric or/and peer review to

evaluate

NOT THE INDIVIDUALS

but

RESEARCH AREAS, DEPARTMENTS, UNIVERSITIES

International character of the italian production: 62,1% of products is in English; mainly in scientific and

technical areas (maximum value 88,6% in informatics and communication engineering). Humanities, law

and in general social areas have the majority of works written in italian ( 5,7% of not italian in Law;

13.2% for Humanities and historical sciences).

THE EVALUATION OF RESEARCH VQR 2004-2010

6

THE EVALUATION OF RESEARCH

2004–2010 (VQR 2004-2010)Products are classified in 14 scientific areas, as established by the National University Council (CUN).

For eight areas (mathematics, physics, chemistry, earth Sciences, biology, medicine, agricultural and Veterinarian Sciences, part of Architecture and Civic Engineering, Industrial and communication Engineering) the research quality has been assessed mainly on the basis of bibliometric indicators ( impact factor and number of citations of the article in a mixed way).

For six areas (Humanities and Social Sciences) the products have been assessed through peer review.

Bibliometric indicators were used also in part of Architecture and Civic Engineering, Psychology Sciences, Economics and Statistics.

• Some areas were “importing” or “exporting” products, giving an image of the processes of crossed fertilization or of boundaries processes in Italian research

7

SOME NUMBERS

95 Universities, 67 public and 28 private, 11 of which are at distance institutions 12 Research Institutes under the control of the Ministry of Education and University 26 Research Institutes which have voluntary submitted an application to be evaluated by

ANVUR

450 national and international Scientific experts from all the 14 Areas of Research 14.770 reviewers, 10.150 Italian and 4.620 working in non Italian institutions

18 months the full time-length of the evaluation

• 184,878 research products have been submitted for evaluation; 5.1% of “missing products” from -not active, or partially active- researchers.

• The typology of products varies in different scientific areas: book and book chapters (average value 19.9%) are more common in humanities and social sciences, articles (average value 73.5%) in scientific and technical areas.

8

EXPECTED VS. SUBMITTED PRODUCTS

0

5.000

10.000

15.000

20.000

25.000

30.000

01 02 03 04 05 06 07 08 09 10 11 12 13 14

# Pr

odotti

GEV

# Prodotti attesi (Area del soggetto valutato)

# prodotti conferiti (Area del soggetto valutato)

# prodotti conferiti (Area del prodotto)

9

SUBMITTED PRODUCTS BY

AREA OF THE PRODUCT

1 2 3 4 5 6 7 8 9 10 11 12 13 140%

10%20%30%40%50%60%70%80%90%

100%

Articles Books, chapters Proceedings Others

10

THE EVALUATION RESULTS

A research product has been classified as follows:• Excellent, when it falls in the top 20% in terms of quality according to

the international scientific community in the field• Good, when it falls in the 60-80% • Adequate, when it falls in the top 50-60% • Limited, when it falls in the bottom 50% • Penalized (not pertinent or copied): 2076 products (1,1%)

TOTAL** 181878 100,0

Excellent 66973 36,8

Good 47925 26,4

Acceptable 22542 12,4

Limited 42362 23,3

• Comparisons only within the same area, NOT between different areas

• The products evaluation results have been used to elaborate the final assessment of the structures. Its weight was 50%

• The others indicators for the final assessment were

ability to attract funds in a number of competitive calls (10%) mobility index, calculating the results of the new coming researches in the examined period (10%) Internationalization index (incoming and outcoming researchers, products with non Italian co-

authors), 10% High qualification index (ph.d. students, post graduation grants…), 10% Funds for research coming from the institution, 5% Improvement index, as a difference from the former evaluation process (5%)

NOTE on the Evaluation Results

11

12

AVA AND THE QUALITY ASSESSMENT

COURSES & INSTITUTIONS ASSESSMENT

ACCREDITATION: CRITERIA FOR QUALITY

ASSESSMENT OUTCOMES ASSESSMENTINTERNATIONAL

COMPARISON

“ON-SITE” VISITS

QUALITY ASSESSMENT GROUP

13

SUA: THE ANNUAL SUMMARIZING CHART

Each dean or responsible of the course of study is required to fill

• the names of three professors and their teaching subjects yearly in charge of the course

• the aims and methods of the course• the professional profile for the graduates• The labor market needs• the curriculum (characterizing compulsory or optional subjects,

laboratories, stages, international programs…).

SUA chart provides information for the students enrolling the following year

14

THE ANNUAL REVIEW REPORT

Each University has a committee for the audit of the Quality Assessment (Nucleo di Valutazione). In November, the Nucleo

• Analize • the SUAs of the former year • the results of the questionnaires filled by students and graduates

• evaluates Quality Assurance system• writes a rewiev report • Suggestes the changes to improve the quality

15

ANVUR will also perform the evaluation of the system as a whole

• Universities have an external evaluation every five years • Courses of study have an external evaluation every three years

The external assessment is performed by a group of experts appointed by ANVUR (CEV, Experts of Evaluation Commission).

The CEVs work starting from the review of the Nucleo and from other documents produced by the course/university Then they do an on site visit of three to five days, depending from the dimension of the university. Their report is submitted to the University governance for comments . The final report gives an A, B or C classification.

THE EXTERNAL EVALUATION

16

Stakeholders and useers

CEV commission of experts

Teaching service

Self-assessmentStudent questionnaire

Review report

Project to improve quality

Final assessment report

Provisional report discussion

On site visit

AVA PROCESS

17

THANK YOU FOR YOUR ATTENTION!