Quality assurance practices and University evaluation systems
Stefano FantoniPresident, ANVUR
Euro-Mediterranean University Network TETHIS meetingUniversity of Palermo 10°-11° of April 2014
2
THE ITALIAN NATIONAL AGENCY FOR THE EVALUATION OF UNIVERSITIES AND RESEARCH
INSTITUTES
ANVUR is the Italian National Agency for the Evaluation of Universities and Research
Institutes. It started operating in May 2011.
ANVUR is in charge of
• Evaluating the quality of processes, results, and products of all the activities concerning
research, teaching and technological transfer
• Defining methods and criteria to be used for the periodical accreditation and evaluation
of Universities (undergraduate, postgraduate and Ph.D. courses)
• Providing general advices to internal evaluation bodies
• Evaluating public funding programs aimed at research and innovation activities
• Every two years, ANVUR will edit a general Report on the italian Higher Education status
3
THE ITALIAN NATIONAL AGENCY FOR THE EVALUATION OF UNIVERSITIES AND RESEARCH
INSTITUTES
ANVUR, in its evaluation process, refers to the Quality Assurance system as described in
European Standard Guidelines ESG/ENQA 2005/2009.
• ANVUR has performed the research evaluation exercise for the period 2004-2010 (VQR
2004 - 2010, Valutazione della Qualità della Ricerca), whose results have been
presented in July 2013
• ANVUR is currently performing the accreditation and assessment program for courses
and institution (AVA, Accreditamento, Valutazione e Autovalutazione)
• ANVUR is currently developing the accreditation and the evaluation models for the PhD
courses.
4
THE CONTEST OF THE EVALUATION
PUBLIC FUNDING
COURSES EVALUATION
MANAGEMENT EVALUATION
Evaluation of Research
INCOMING RESEARCHER
S EVALUATION
Evaluation of Universities
PLACEMENT & TRANSITION TO
WORK
5
Each university professor or researcher has been asked to submit three research products (six instead of
three to the researchers of the Research Institutes) published in the 2004-2010 period. Each research
product (for a total of about 180 thousands) has been evaluated by bibliometric or/and peer review to
evaluate
NOT THE INDIVIDUALS
but
RESEARCH AREAS, DEPARTMENTS, UNIVERSITIES
International character of the italian production: 62,1% of products is in English; mainly in scientific and
technical areas (maximum value 88,6% in informatics and communication engineering). Humanities, law
and in general social areas have the majority of works written in italian ( 5,7% of not italian in Law;
13.2% for Humanities and historical sciences).
THE EVALUATION OF RESEARCH VQR 2004-2010
6
THE EVALUATION OF RESEARCH
2004–2010 (VQR 2004-2010)Products are classified in 14 scientific areas, as established by the National University Council (CUN).
For eight areas (mathematics, physics, chemistry, earth Sciences, biology, medicine, agricultural and Veterinarian Sciences, part of Architecture and Civic Engineering, Industrial and communication Engineering) the research quality has been assessed mainly on the basis of bibliometric indicators ( impact factor and number of citations of the article in a mixed way).
For six areas (Humanities and Social Sciences) the products have been assessed through peer review.
Bibliometric indicators were used also in part of Architecture and Civic Engineering, Psychology Sciences, Economics and Statistics.
• Some areas were “importing” or “exporting” products, giving an image of the processes of crossed fertilization or of boundaries processes in Italian research
7
SOME NUMBERS
95 Universities, 67 public and 28 private, 11 of which are at distance institutions 12 Research Institutes under the control of the Ministry of Education and University 26 Research Institutes which have voluntary submitted an application to be evaluated by
ANVUR
450 national and international Scientific experts from all the 14 Areas of Research 14.770 reviewers, 10.150 Italian and 4.620 working in non Italian institutions
18 months the full time-length of the evaluation
• 184,878 research products have been submitted for evaluation; 5.1% of “missing products” from -not active, or partially active- researchers.
• The typology of products varies in different scientific areas: book and book chapters (average value 19.9%) are more common in humanities and social sciences, articles (average value 73.5%) in scientific and technical areas.
8
EXPECTED VS. SUBMITTED PRODUCTS
0
5.000
10.000
15.000
20.000
25.000
30.000
01 02 03 04 05 06 07 08 09 10 11 12 13 14
# Pr
odotti
GEV
# Prodotti attesi (Area del soggetto valutato)
# prodotti conferiti (Area del soggetto valutato)
# prodotti conferiti (Area del prodotto)
9
SUBMITTED PRODUCTS BY
AREA OF THE PRODUCT
1 2 3 4 5 6 7 8 9 10 11 12 13 140%
10%20%30%40%50%60%70%80%90%
100%
Articles Books, chapters Proceedings Others
10
THE EVALUATION RESULTS
A research product has been classified as follows:• Excellent, when it falls in the top 20% in terms of quality according to
the international scientific community in the field• Good, when it falls in the 60-80% • Adequate, when it falls in the top 50-60% • Limited, when it falls in the bottom 50% • Penalized (not pertinent or copied): 2076 products (1,1%)
TOTAL** 181878 100,0
Excellent 66973 36,8
Good 47925 26,4
Acceptable 22542 12,4
Limited 42362 23,3
• Comparisons only within the same area, NOT between different areas
• The products evaluation results have been used to elaborate the final assessment of the structures. Its weight was 50%
• The others indicators for the final assessment were
ability to attract funds in a number of competitive calls (10%) mobility index, calculating the results of the new coming researches in the examined period (10%) Internationalization index (incoming and outcoming researchers, products with non Italian co-
authors), 10% High qualification index (ph.d. students, post graduation grants…), 10% Funds for research coming from the institution, 5% Improvement index, as a difference from the former evaluation process (5%)
NOTE on the Evaluation Results
11
12
AVA AND THE QUALITY ASSESSMENT
COURSES & INSTITUTIONS ASSESSMENT
ACCREDITATION: CRITERIA FOR QUALITY
ASSESSMENT OUTCOMES ASSESSMENTINTERNATIONAL
COMPARISON
“ON-SITE” VISITS
QUALITY ASSESSMENT GROUP
13
SUA: THE ANNUAL SUMMARIZING CHART
Each dean or responsible of the course of study is required to fill
• the names of three professors and their teaching subjects yearly in charge of the course
• the aims and methods of the course• the professional profile for the graduates• The labor market needs• the curriculum (characterizing compulsory or optional subjects,
laboratories, stages, international programs…).
SUA chart provides information for the students enrolling the following year
14
THE ANNUAL REVIEW REPORT
Each University has a committee for the audit of the Quality Assessment (Nucleo di Valutazione). In November, the Nucleo
• Analize • the SUAs of the former year • the results of the questionnaires filled by students and graduates
• evaluates Quality Assurance system• writes a rewiev report • Suggestes the changes to improve the quality
15
ANVUR will also perform the evaluation of the system as a whole
• Universities have an external evaluation every five years • Courses of study have an external evaluation every three years
The external assessment is performed by a group of experts appointed by ANVUR (CEV, Experts of Evaluation Commission).
The CEVs work starting from the review of the Nucleo and from other documents produced by the course/university Then they do an on site visit of three to five days, depending from the dimension of the university. Their report is submitted to the University governance for comments . The final report gives an A, B or C classification.
THE EXTERNAL EVALUATION
16
Stakeholders and useers
CEV commission of experts
Teaching service
Self-assessmentStudent questionnaire
Review report
Project to improve quality
Final assessment report
Provisional report discussion
On site visit
AVA PROCESS
17
THANK YOU FOR YOUR ATTENTION!
Top Related