Electronic WorkingPaper Series

Paper No. 71

University Research Evaluation and Funding:An International Comparison

Aldo Geuna* and Ben R. Martin

First Draft: October 2000This Draft: August 2001

The authors are extremely grateful to Dudi Hidayat who provided researchassistance to the project for the Higher Education Funding Council of England on

which this paper is based - see Geuna et al. (1999).

SPRUScience and Technology Policy Research

Mantell BuildingUniversity of Sussex

Falmer, BrightonBN1 9RF, UK

Tel: +44 (0) 1273 877139Fax: +44 (0) 1273 685865

E-mail: a.geuna@sussex.ac.ukhttp://www.sussex.ac.uk/spru/

1

Abstract

Many countries have introduced (or are considering introducing) procedures for theevaluation of university research. This reflects the substantial sums now invested in suchresearch as well as political demands for greater public accountability. In some countries, theallocation of research funds to universities now depends wholly or partly on the results ofevaluation of their previous research. This paper analyses and compares the approaches to thefunding and evaluation of university research in 14 countries in Europe and the Asia-Pacificregion. It identifies the advantages and disadvantages of university research funding systemsbased on performance assessment, contrasting them with those where funds are distributed onthe basis of the volume of teaching activities. The paper points to the higher costs associatedwith performance-based funding systems. While the benefits of such systems may for a periodbe greater than those costs, over time diminishing returns are likely to set in, raising questionsabout the continued use of comprehensive research assessment exercises.

JEL Subject Classification: O3, L3, H4.

Keywords: University research, Research assessment, Public funding, Higher educationpolicy.

2

1. Introduction

Research evaluation has emerged as a key science policy issue in most industrialisedcountries (OECD, 1997). This has been driven by the increasing demand for accountability inthe context of growing constraints on public funding at a time when the number of researchactivities vying for funds is escalating. Consequently, governments have started to implementmechanisms for allocating resources that relate funding to some measure of universityresearch output or performance.

This paper analyses different approaches to the evaluation of research performance inuniversities and how they relate to the allocation of government research funds. It focuses ona number of European and Asian-Pacific countries.1 In many of these, a ‘dual support’ systemis in operation in which general infrastructural research funds are provided on an institutionalbasis while specific research costs are met through individual project grants. Two mainapproaches to general university funding have been identified. In the first, university researchfunds are allocated, at least in part, on the basis of some form of research evaluation.Countries such as the UK, Hong Kong, Australia and Poland are applying performance-basedallocations with differing degrees of assessment. In the UK and Hong Kong, the approach isbased on informed peer review, while in Australia and Poland output indicators are used.

The second approach also uses a formula but one based on volume of teaching. Funds forboth research and teaching are allocated to universities in the form of general institutionalfunding. Countries such as Germany, Italy, Sweden and Norway follow this approach.Finland and Denmark allocate the largest share of general funds on the basis of teachingvolume, but a small portion is allocated on the basis of teaching and research performance. InThe Netherlands, funds are also allocated using a formula based on student numbers.Although research assessments are carried out, these are not linked with funding decisions;instead, the assessments are used to create reputational incentives that may spur theimprovement of research quality within universities.

While only a few countries are currently using performance-based approaches to universityresearch funding, most of those considered here are either in the process of implementingsome form of performance-based allocation, or are considering doing so. The remainder ofthis paper is organised as follows. Section 2 discusses the analytical framework used toexamine university research assessment and funding allocation in the 14 countries considered.University research evaluation and funding of 11 European countries – including four Nordicand three Eastern European countries – is described in Section 3. Section 4 is devoted to theanalysis of university research evaluation and funding in Australia, New Zealand and HongKong. Section 5 provides a critical analysis of the advantages and disadvantages of the

1 Information pertaining to the 14 countries considered covers the period ending in 1999.

3

different approaches to university research funding. Finally, Section 6 considers the questionof how the benefits of a performance-based system for allocating university researchresources compare with its costs and how they evolve over time.

2. Research Evaluation

Over the last two decades, society’s attitudes towards universities, and the demands placedupon them, have undergone great change (Gellert, 1993; Geuna, 1999; Noll, 1998; OECD,1987 and 1998). In many countries, a growing concern has emerged “about the increasingcost of funding university-based research … and the need to obtain ‘value for money’ forpublic expenditure on higher education” (OECD, 1987, p.19). Universities have beenexpected to become more efficient in their use of public resources and more accountable(Massy, 1996). These pressures have made research evaluation a central issue. However,evaluation is certainly not new. Indeed, since the institutionalisation of science in the late18th and early 19th centuries, evaluation has been central to research. Under the broadcategory of peer review, various forms of research evaluation take place – for instance, whena paper is submitted for publication in a journal, when a new faculty member is appointed orpromoted, or when a funding council allocates a grant. However, in such evaluations, the unitof assessment is generally a project or an individual rather than an entire universitydepartment.

Before we embark on further discussion, it is useful to define several terms. Although somewriters have distinguished ‘evaluation’ from ‘assessment’ (e.g. Hills and Dale, 1995), for ourpurposes both terms are used to describe efforts to analyse qualitatively or quantitatively thenature and extent of the outputs from research by a given unit (e.g. a university department)taking into account the resources available to that unit. Evaluation can be distinguished intoex-ante and ex-post forms, and whether it performs a summative or formative function(Kogan, 1989; Massy, 1996; Suter, 1997). Ex-ante evaluation is evaluation conducted prior tothe execution of a research project to assess its potential importance and likelihood ofsuccess. Ex-post evaluation is conducted after the project has been completed to assess itsoutput or impact. Summative evaluation involves judgements about the particularperformance of a unit based on accumulated evidence in comparison with that of similarunits. In the case of formative evaluation, the aim is to assist the unit in meeting its goals.

The results of evaluations are increasingly used as inputs for decision-making in researchmanagement. Decisions are made at various levels – for example, the project or programme,the research organisation, or at a national level (e.g. an evaluation by a research council).Among the decisions taken based on evaluation results, two types are particularly important:strategy formulation and funding allocation (Steen and Eijffinger, 1998). Evaluation for

4

strategy formulation is often conducted at the institutional level. Institutions need to evaluatetheir research strengths and weaknesses and the environment in which they operate in order toprioritise research areas and to formulate a strategy for future work. Quality assessmentsystems within universities are an example of such evaluation. However, evaluation forstrategy formulation may also be conducted at the national level as part of the effort tostrengthen the national research system. In addition, evaluation is used as a tool to helpdetermine funding allocations, the research performance of individual researchers, projects,programmes and centres being evaluated and research funds allocated to those with betterperformance.2 The assumption is that funds allocated in this way will yield greater returns.

This paper describes current practices in research assessment at the national level for thepurpose of informing the funding allocation process for universities in 14 countries in Europeand the Asia-Pacific region (see Table 1). These countries use various types of assessment foruniversity research. Here we propose a general scheme to analyse them. The assessmentmethods of each country have been examined from the following aspects.

{TABLE 1 about here}

1. Evaluation performer(i) National level:

a. Higher education funding agency (providing general institutional funding)b. Research Council (providing funds for specific projects)

(ii) Institutional level:a. Board of Universityb. Department

2. Evaluation purposea. Funding allocation and/or accountability (summative)b. Strategy formulation (formative)

3. Evaluation criteria and methods(i) Criteria:

a. Quantityb. Qualityc. Impactd. Utility

(ii) Methods:a. Peer reviewb. Bibliometricsc. Peer review supplemented with bibliometrics

2 In some cases, however, a political decision may be taken to invest funds in building up weaker groups rather

than concentrating resources on more successful ones.

5

There has been much debate on the advantages and disadvantages of research evaluation as atool to assess and allocate funds to university research - a debate that is still ongoing.3

2.1 Evaluation Performer and Evaluation Purpose

Among countries, different agencies are responsible for conducting research evaluation. In theUK it is the Higher Education Funding Councils (HEFCs)4 while in The Netherlands anothertype of agency is responsible for evaluations (apart from internal ones by universitiesthemselves) – namely the Association of Netherlands Universities (VSNU) (Steen andEijffinger, 1998). The purpose of the evaluation tends to be related to who is conducting it.The UK HEFCs conduct assessments in order to decide how to distribute research funds,while VSNU carries out evaluations mainly to help university research management, althoughthey also provide ‘market information’.

2.2 Evaluation Criteria and Methods

Different evaluation mechanisms employ different criteria and methodologies depending onwhich aspects of performance are being measured. Evaluations have tended to focus on fourmain aspects: volume of research output, quality, impact (on other researchers or on theadvancement of knowledge), and utility in terms of generating technological, economic orsocial benefits. To measure these different aspects, various indicators have been developed.There is a large body of literature analysing the advantages and drawbacks of using indicatorsof research performance but little consensus as to which indicator (or set of indicators) is bestsuited to measuring each of these four aspects of performance (Geuna, 1999).5

As regards assessment methods, the literature on research performance shows thatbibliometric analysis and peer review are the main approaches to assessing the quality andimpact of research. However, given the time-consuming and costly character of bibliometricanalysis6 as well as other drawbacks, it is not very practical for macro-level evaluations suchas those focusing on all the universities within a nation. This leaves peer review, despite itsshortcomings, as the main assessment method for macro-level research evaluations.7 It is

3 See e.g. Cooper and Otley (1998) El-Khawase and Massey (1996), Geuna (2001), Kogan (1998), Kushner

(1996) and Whittington (1997).4 HEFCs are responsible for allocating teaching and general research funds to universities.5 For an early study on the difficulties involved in constructing research performance indicators, see Martin and

Irvine (1983). For further discussion, see e.g. Cave et al. (1997; Chapter 4), and van Raan (1988).6 The main problem is cleaning up and unifying the institutional addresses given on publications to the

necessary degree of accuracy, a task which can take person-years of effort (Martin and Skea, 1992)7 Surveys suggest that researchers favour peer review over other assessment methods (Martin and Skea, 1992).

6

sometimes supplemented with publication and citation data and other related information –this then being referred to as ‘informed peer review’.

3. University Research Evaluation and Funding in Europe

The pressures for effective and selective allocation of funds, although perhaps mostprominent in the UK, are also present in other countries. In what follows we describe theuniversity research evaluation and funding systems used in 11 European countries.

3.1 The UK

The UK has one of the most advanced research evaluation systems in Europe (Hills and Dale,1995) with evaluation not only at the level of the individual researcher or project but also atthe institutional or national level. Since the mid-1980s, there have been periodic researchassessments of all universities. The first Research Assessment Exercises (RAE) was carriedout in 1986 with others in 1989, 1992, 1996 and 2001. Since 1989, these exercises have had abroadly similar methodology, although wide consultation with the academic communitybetween one RAE and the next has brought methodological improvements.

Evaluation Performer and Evaluation Purposes

The 1996 RAE8 was carried out jointly by the four UK higher education funding bodies.9 Theaim of the RAE is to give university research10 a quality rating on which the distribution of agreat deal of public research funds is then based.11 (The RAE excludes teaching activities.)12)The definition of research used in the RAE is quite broad and includes:

original investigation undertaken in order to gain knowledge andunderstanding. It includes work of direct relevance to the needs of commerceand industry, as well as to the public and voluntary sectors; scholarship*; theinvention and generation of ideas, images, performances and artefactsincluding design, where these lead to new or substantially improved insights;and the use of existing knowledge in experimental development to produce

8 Although this section focuses on the 1996 RAE, the approach of the 2001 exercise was very similar.9 The four are the Higher Education Funding Council for England (HEFCE), the Scottish Higher Education

Funding Council (SHEFC), the Higher Education Funding Council for Wales (HEFCW) and the Departmentof Education for Northern Ireland (DENI).

10 In 1993, higher education in the UK moved to a unified system, in which polytechnics and some major furthereducation colleges were allowed to apply for the title of ‘University’ and given the right to award their owndegrees. All universities are eligible to participate in the RAE.

11 In the fiscal year 1995-96, in England, no less than 94.3% of the funding from HEFC for university researchwas allocated on the basis of the RAE results – see HEFCE (1997).

12 Another exercise, the ‘Teaching Quality Assessment (TQA)’, assesses teaching quality in British universities.

7

new or substantially improved materials, devices, products and processes,including design and construction. It excludes routine testing and analysis ofmaterials, components and processes, e.g. for the maintenance of nationalstandards, as distinct from the development of new analytical techniques.”(HEFCE, 1994)* Scholarship embraces a spectrum of activities including the development of teachingmaterial; the latter is excluded from the RAE.

There was no separate assessment of basic and applied research. There had been concern thatthe RAE was biased against applied research (e.g. Griffith, 1995) so panels in 1996 wereinstructed to give equal weight to all research, regardless of whether it was basic or applied(HEFCE, 1994), and to focus only on its quality. In accordance with Government sciencepolicy as set out in the 1993 White Paper, panels were instructed to give full recognition towork relevant to the needs of industry as well as academia. Furthermore, in response tofeedback from the 1992 RAE suggesting that interdisciplinary research had not been fairlyassessed (e.g. Royal Society of Chemistry, 1995), universities were encouraged to submitwork of an interdisciplinary nature to the most appropriate assessment panel but to suggest asecond, related panel that might also consider the submission.

Evaluation Criteria and Methods

The RAE is an ex post evaluation based on informed peer review. All research activitieswithin the university are categorised into a number of units of assessment (UoA) (e.g.biochemistry, physics). In 1996, 69 UoA were defined, these being broadly similar to thoseused in 1992. For each UoA, an panel of around six to ten experts was set up. Some 1,000outside bodies (e.g. subject associations, learned societies, professional bodies, organisationsrepresenting users) were asked to nominate candidates for panel membership. The panelChairs were selected by the Chief Executives of the funding bodies. Half of the Chairs in1996 had served as Chairs in the previous exercise; the remainder were appointed in the lightof recommendations from outgoing Chairs, almost all having previously been panellists.13

The Chairs recommended members for their panels drawn from the nominees of outsidebodies and taking into consideration such factors as the eminence of individuals, coverage ofthe subject, and sectoral and geographical balance.

Every department or group within a university is assigned to a UoA and hence a panel.14 Theinformation on the research performance in 1996 included the following (HEFCE, 1994):

13 For the 2001 RAE and beyond, in response to concern that ‘rolling-forward’ two thirds of panel members

from one RAE to the next does not constitute sufficient dynamism, panel members will be allowed to serveonly for two successive exercises (HEFCE, 1998).

14 Occasionally a university department may be assigned to more than one UoA, or two departments may becombined in a single UoA.

8

• an overall staff summary – this contained information on all academic staff and researchsupport staff, whether or not they were included as ‘research active’ staff;15

• details on research active staff whose work was to be evaluated;• publications and other public output – for each research active member of staff up to four

items could be submitted;16

• an overview of research students and research studentships;• details of external research income, including the volume and sources;• a statement of research plans.

Of all this information, the publications and other research outputs constitute the corematerial for the assessment. Unlike the 1992 RAE, which, in addition to the submission offour published/public outputs for each researcher, required statistics on the total number ofpublications from the department/UoA, the 1996 RAE required only up to four researchoutputs per research active staff to be evaluated. This change was introduced to try to focusthe assessment on the quality of the research output rather than the quantity. It also reducedthe problem of low quality ‘rush publications’ as a result of pressures within thedepartment/unit to produce as many publications as possible to increase the aggregate total.17

On the basis of all the information, panels judged the research quality of each universitydepartment and assigned a rating on a scale from 1 to 5*. The resulting rating is used byHEFCs in the formula applied to determine the research funding for each unit of assessment,with the total block grant received by a university being calculated by summing across allunits.

3.2 The Netherlands

University research is financed through a dual-support system: one component is institutionalcore funding provided by the Ministry of Education and Science through the so-called ‘first-flow’ of finance; the other consists of the ‘second flow’ and ‘third flow’ of finance – thesecond flow are project and programme grants from research councils and foundations andthe third flow are contracts from government departments, agencies and other organisations.

As in other countries in the late 1970s, growing concern about the quality and societalrelevance of university research led to demands for greater accountability. In 1979, a

15 A department is not obliged to submit the research outputs of all its staff; it can decide to submit only certain

‘research active’ staff, the implication being that it will receive funds only for those researchers.16 The work had to have been published during the period January 1990 to 31 March 1996 in the case of arts and

humanities and the period 1 January 1992 to 31 March 1996 for all other subjects.17 There is some evidence that the use of the total number of publications as a performance measure may have

led to ‘publishing inflation’ – i.e. maximisation of the number of articles produced either by repetition,lowering quality standards, or the ‘salami slicing’ of research into least publishable units (e.g. Cave et al.,1997).

9

government White Paper (Minister for Education and Science, 1979) recommended changesin the management of academic research and a system of ‘conditional funding’ wasintroduced in 1983 (see e.g. Irvine and Isard, 1990). In this, a clear distinction was drawnbetween teaching and research funds (termed, respectively, A-part and B-part funds). In theresearch part, academic staff posts were financed on the basis of proven quality of research. Inaddition, various national research objectives were identified with the intention that anincreasing proportion of institutional funding would be earmarked for research in fieldslinked to these. Universities were required to produce a medium-term research planidentifying areas in which had strengths and therefore deserved priority funding. This‘conditional funding’ scheme was replaced in 1993 by the HOBEK funding model.

The HOBEK model consists of three parts – teaching (23%), research (64%) and‘interweavement’ (13%). Funding is based on a formula with four components: number ofregistered students, degrees, doctorates and designers’ certificates. A large part of the researchbudget has been allocated incrementally on the basis of historical data rather than norms orquality (Koelman, 1998). In 1999, HOBEK was replaced by a new model called STABEK2,standing for stable funding (‘STAbiele BEKostiging’). Under this, the government approvesfunding to universities for a period covering several years, thereby providing universities withmore stability. However, this scheme is intended only to be temporary while the Ministry ofEducation, Culture and Science develops a new funding scheme with more emphasis onperformance (van der Meer, 1999).

Evaluation performer and evaluation purposes

While there have been no research evaluations for the purpose of allocating funds, there havebeen evaluations for the purpose of strategy formulation. Rip and van der Meulen (1995)argued that in the Dutch evaluation culture, informal bottom-up assessments are dominant,and that science policy agencies have been more interested in strategic changes in the researchsystem than in evaluating performance. As regards the former, the Association of TheNetherlands Universities (VSNU) has been very active. In 1992, 13 universities18 and theMinister of Education agreed that VSNU should develop a system of external researchevaluation of universities as a complement to internal efforts aimed at quality control. Thiswas first implemented in 1993-94, and a complete first round of evaluation covering all 27disciplines was completed in 1998. A second round started in 1999. The primary objective of

18 The Dutch university system consists of 14 universities − nine general, three technical, one agricultural and an

Open University. However, because of its different nature, the Open University has not been included in thescheme described here.

10

the ‘Quality Assessment of Research’ (as it is known) is to help university management todevelop strategies and to supply information for decision-making at various levels within theuniversity up to the executive board. This evaluation system replaced the system ofconditional funding, the objective of which was to make university research more efficient,productive and socially relevant (Steen and Eijffinger, 1998).

Evaluation criteria and methods

Research assessment covers all research conducted within universities, which is classifiedinto disciplines and then, in turn, into research programmes.19 Each discipline is evaluated bya committee. In total, there are 27 disciplines (e.g. Chemical Research). Unlike in the UKRAE where disciplines are evaluated simultaneously, in The Netherlands they are evaluated atdifferent times over a period of four to six years.20 In 1993, a first small group of disciplineswas evaluated, this initial evaluation being used as pilot for succeeding ones. Based on thisexperience, improved guidelines for evaluation were drawn up in 1994, which were set out inthe ‘Protocol 1994 for Quality Assessment of Research’. This Protocol contains importantprocedural details and describes the responsibilities of all the actors involved.

To evaluate each discipline, a Review Committee of five to seven experts with research andmanagement experience is set up by VSNU in consultation with faculty deans and the RoyalAcademy of Arts and Sciences (KNAW). The committee chair must be either a Dutch expertor a foreign expert thoroughly familiar with the Dutch situation. Committee members arepredominantly foreign experts, which ensures a high degree of impartiality. Given theinternational composition of the committees, the primary language of communication duringthe evaluation is English, and the results are also compiled and published in English.

The committee evaluates the performance of the research programmes based on the followinginformation on research activities conducted during the previous five years:• an overview of academic staff;• a summary of the programme mission and research plan;• the content of the programme and its main results;• a list of all publications;• a list of five selected key publications from the programme;

19 These represent the smallest unit of assessment – hierarchically, research programmes are organised under

university departments.20 Because the evaluation is not intended to be used for determining the allocation of funds, there is no need to

evaluate all disciplines at the same time.

11

• other indicators of quality and reputation (such as patents, invited lectures, etc.).

In addition, interviews with research programme leaders and site visits are often conducted tosupplement the written information. Whenever available, a comprehensive bibliometricanalysis is provided to the Committee,21 VSNU believing bibliometric analysis to be a usefultool that complements the written and oral information on which the assessment is primarilybased (VSNU, 1996). Committees assess each research programme in terms of four aspects:

1. Scientific quality – originality of ideas and methodology, importance of research outputfor the discipline, scientific impact and international prominence of the research group.

2. Scientific productivity – relates inputs to outputs of research; staff numbers and size ofresearch funds are used as input measures; output indicators include number and nature ofscientific publications, and number of dissertations, patents and invited lectures.

3. Scientific relevance – to the advancement of knowledge in the discipline and to science ingeneral; possible impact and application for future technology; and societal benefits.

4. Long-term viability – based on the submitted plans, ideas for future research, publicationpolicy, coherence of the programme and continuity of research lines.

The assessment of each of these four aspects is then translated into a five-point rating (1 =‘poor’, 5 = ‘excellent’). Although for the 1999-2002 research assessment the frameworkremains basically the same, there is greater leeway for assessing groups in relation to theirown stated (and differing) missions. Assessment committees can thus choose to evaluate agroup’s relevance and viability according to its stated mission. (Quality and productivitycontinue to be assessed according to one standard as previously.) This greater flexibilityreflects VSNU’s recognition that there are differences between institutions which need to bereflected in the evaluation. However, it remains to be seen how much effect this will havesince assessment committees are free to decide the extent to which they take into accountdiffering group missions (personal communication with Anne Klemperer).

3.3 Germany

Most university research in Germany is conducted in 90 ‘scientific universities’(wissenschaftliche Hochschulen) embodying the long-standing Humboldtian tradition ofintegrating teaching and research. However, some is also carried out in the polytechnics(Fachhochschulen); although these are mainly teaching institutions, they are undertaking an

21 For example, the Review Committee for Chemical Research was provided with a bibliometric analysis

produced by the Centre for Science and Technology Studies at Leiden University (CWTS). This assessed theimpact of scientific journal articles from all the research programmes assessed by the committee.

12

increasing amount of research. Whereas universities receive core funding for research fromgovernment, the core funding to polytechnics from government does not include any elementfor research. Research projects in polytechnics are largely funded by contracts from industryor specific project grants from government institutions.

There are three sources of public funding for university research. The first is institutional corefunding for teaching and research in the form of a block grant from the regional (Länder)government; this constitutes almost two-thirds of total university expenditure and coversbasic research infrastructure and staff resources. The second is a capital grant for buildingsand large-scale equipment and is provided by the central government (Bund) and Länderjointly. The third source is ‘third party funds’ (Drittmittel), which are grants and contractsprovided by public institutions for specific research projects. A large part of these areallocated by the German Research Foundation (DFG), which is financed jointly by the Bundand Länder. As in other countries, third-party funds are awarded on the basis of a peer reviewof applications using the criteria of scientific excellence and social relevance.

Institutional and capital funds are allocated based on a university’s educational profileincluding the number of students and scientific staff and current spending. In determining theshare of funds for research, an ‘R&D coefficient’ is used derived from surveys of workingtime spent on research and teaching of a sample of scientific staff. In general, no researchperformance measures are used to allocate research funds and there have been no evaluationsfor this purpose. In recent years, a few Länder have allocated additional resources on acompetitive or performance-related basis. In particular, Lower Saxony set up a scientificcommission in 1998 to assess research performance in the state’s 12 universities, with theinitial focus on biochemistry and history.22

There has been no research evaluation covering all universities. This is partly becauseuniversities are mainly financed at the regional level, but it is also due to university hostilitytowards the competition that would be created. Among German academics, “competition isnot seen as a principle for advancing and encouraging research (and teaching) quality”(Campbell and Felderer, 1997). Although there have been several evaluations of universityresearch, these have not influenced funding (Daniel and Fisch, 1990).

22 Research evaluations of the Leibniz and other research institutes have become more common since 1990.

13

In the late 1990s the Federal Government began to stress the increasing need for researchevaluation (Campbell and Felderer, 1997). However, one constraint on evaluations ofuniversity research stems from the fact that the German constitution grants universitiesconsiderable autonomy. Indeed, it has been argued that, with the exception of evaluations bystudents, systematic evaluations by government of the teaching of university professors areunconstitutional. Not surprisingly, the main actor in relation to research evaluation inGermany so far, has been the individual university. The Freie Universität in Berlin, forinstance, has implemented its own evaluation mechanism, the results of which are used forthe internal distribution of funding (Campbell and Felderer, 1997). However, manyuniversities – such as those in the Verbund Norddeutscher Universitäten (VNU) – perceivethe use of research evaluation for internal funding allocation as retrospective in that it rewardsthose who have shown good performance in the past, while what is needed is a mechanismthat helps universities to improve their performance.

In 1998, the HE sector underwent a major reform with the adoption by the Bundestag of anamendment to the HE Framework Act (BMBF, 1998). Its aims are twofold: to makecompetition possible through deregulation, performance orientation and the creation ofincentives; and to secure the competitiveness of the German HE sector in the 21st century. Itprovides a legal basis for important structural improvements including the introduction offinancing based on teaching and research achievements and the comprehensive evaluation ofeducation and research. It abolishes the previous ‘immunity’ of professors to externalevaluation. Hence, external research evaluation is more likely to take place in the future.

3.4 Italy

The HE sector in Italy consists of 65 institutions. The state owns 45 universities, threepolytechnics, three single-discipline universities, two universities for foreign students andthree Scuole superiori (comparable to the French Écoles Normales); the remaining nine areprivate (Boffo, 1996). Government finances university research through a dual-supportsystem. On the one hand, the Ministry of Universities and Scientific and TechnologicalResearch (MURST) provides basic institutional funding, which includes a small portion fornon-directed research; on the other, MURST allocates funds for individual research projectsthrough national university grants and grants distributed on a competitive basis by theConsiglio Nationale delle Ricerche (CNR). Core funding is allocated to each universityaccording to numbers of teaching and research staff, while funding for targeted research isallocated through peer review of applications. In addition, CNR allocates competitive fundsfor targeted research that are accessible not only to universities but also other research

14

institutions. With the exception of the peer review of applications, there has so far been nospecific evaluation of university research related to funding allocations.

However, the National University Evaluation Council (NUEC) has recently suggested thatuniversity funding should be linked to three variables: teaching demand in each subject area;results achieved in teaching; and results achieved in research (Biggeri and Scarpitti, 1998).NUEC is currently developing appropriate measures for each criterion. In particular, it isorganising a study on research outputs in the last two years for each university.

The formal government statement of the need for university evaluation can be traced back toAct 168 of 1989 which included provision for the establishment of a central office foruniversity evaluation (Boffo, 1996). However, the National University Evaluation Council(NUEC) was created only in 1996. The main function of the Council, which consists of fivemembers appointed for three years, is the evaluation of universities. It is an independent bodywith its own technical and administrative secretariat and its own budget, and it may appointteams of external experts to carry out specific studies.

Three years before NUEC was created, Act 537 of 1993 had already stipulated that in futureuniversities would need to support any request for additional funds with evidence of resultsachieved so far. From 1994, a growing portion of the resources allocated was meant to bebased on effectiveness and efficiency in teaching and research. However, this proviso hasbeen largely ignored; in practice, universities received additional resources over successiveyears regardless of results (Boffo, 1996). Nevertheless, research evaluation may become morecommon in future as a consequence of the NUEC proposal to relate funding to performance.

3.5 Nordic countries: Finland, Norway, Sweden and Denmark

As elsewhere in Europe, university research is financed through a dual-support systemcombining institutional funding with grants from research councils. The latter are based onpeer review of applications by international experts while the former is distributed as a lumpsum to each university which then has discretion as to its internal distribution. Only inFinland and Denmark do the institutional funds contain a performance-based component.However, in both cases the performance measures are mainly quantitative indicators such asthe amount of external funding – i.e. measures that do not require a special evaluation.

15

In Denmark, a new budgeting system was introduced in 1994 in which a clear distinction ismade between funds for teaching and for research. The budget contains five elements: a basicgrant, a performance-related grant for teaching, a research grant, a grant for other activities,and a capital grant. Until 1995, research funds were allocated on an incremental basis. Sincethen, the amount awarded has depended on the volume of teaching and external researchincome; no other research performance measures are used, although PhD student numbers areemployed as one of the measures to determine the performance-based grant for teaching.

In Norway, universities receive government block grants with no distinction between teachingand research. Until recently, there has been no serious attempt to adopt research performancemeasures in the funding allocation mechanism, apart from universities receiving a fixed sumper doctoral graduate. Similarly, in Sweden performance-based research funding has not beenimplemented. With the introduction of a new HE law in 1993, examination results are theonly performance indicator considered in deciding the allocation of funds. Given that Finlandhas implemented performance-based funding more extensively, we will discuss its fundingmechanisms and research field evaluations in greater detail. It should be noted, however, thatresearch field evaluations (which are not linked to funding allocation) are common in otherNordic countries (Helander, 1995; Luukkonen, 1995), so the discussion here in relation toFinland is broadly relevant to the region as a whole.

Finland23

Finland has 20 universities (ten multidisciplinary, six specialist, and four art academies) and anetwork of new polytechnics (AMK).24 University research is financed by the Ministry ofEducation and four research councils under the Academy of Finland; there is also contractresearch funded by industry or government institutions. Academy funds are allocated on thebasis of peer-reviewed project applications. General institutional funding (for teaching andresearch) is provided by the Ministry of Education under a contract arrangement betweenuniversities and the Ministry known as ‘Management by Result’. This arrangement, whichwas developed in the late 1980s, involves assessing part of the university budget on the basisof performance. Budgeting-by-result agreements were introduced in 1994 and were evaluatedby a group from the Ministry and the universities. On the basis of this assessment,management has been improved with the adoption in 1998 of a three-year agreement which

23 This section is based on Ministry of Education (nd).24 Polytechnic reform began in the late 1990s, with the upgrading of most vocational further education to the

polytechnic level (to create 30 polytechnics) and the dismantling of the previous vocational post-secondarysystem. Polytechnics do not engage in basic research, or offer postgraduate education, or have professorships.

16

covers the results that the university is expected to achieve and the level of funding. Thethree-year agreements are modified annually under the terms of an annex to the agreementallowing adjustments to the budget for the following year.

The operating expenditure agreed upon comprises basic funding (90%), project funding (7%)and performance-related funding (3% but expected to increase). Basic funding is intended tocover salaries and facilities. A formula-based model for basic funding has been used since1997, in which the connection between teaching and research is made explicit. The teachingcomponent is represented by target numbers of Masters degrees and the research element bytarget numbers of Doctoral degrees. Project funding is earmarked for national programmesdefined by government. Performance-related funding is awarded on the basis of a number ofagreed indicators focusing on the quality and impact of research and teaching. They includecentres of excellence25 in research and education; Academy funds; international funds andcollaboration; graduate placements; and a university-specific assessment of whether thetargets set have been met, and of the strategic planning in the university. All these data arecontained in the KOTA HE database, based on the reports submitted by universities eachyear, so no specific evaluation system is required to determine the funding allocation.

Evaluation performer and evaluation purpose

To assist universities and polytechnics with evaluation, the Finland Higher EducationEvaluation Council (FINHEEC) was established in 1995. Evaluations are of three main types:institutional evaluation; programme/thematic evaluation; and accreditation. None of these istargeted specifically at research. Institutional evaluations do not use a uniform model for alluniversities. In this way, the government recognises the differences between universities andemphasises the developmental role of evaluation. Nevertheless, most evaluations can becharacterised as broad institutional assessments of the basic preconditions for teaching andresearch and the capacity for change. These preconditions include a statement of theinstitution’s mission and goals, key processes, institutional arrangements, and resources andperformance, with less emphasis being given to the last two factors. The emphasis on each of

25 In 1993, a new policy to create centres of research excellence was introduced. A centre of excellence may

consist of research groups, research centres or a larger umbrella organisation and networks. The Academy ofFinland, FINHEEC and the Ministry of Education are responsible for the selection, which, of course, involvesevaluation. By 1997, 17 research units had been designated as centres of excellence and received extra funds.In judging whether to establish a centre of excellence, international peers consider six criteria: national/international standing of researchers; scientific significance, innovativeness and effectiveness of research;quality, quantity and focus of scientific production; patents; national and international mobility of researchers;and number and level of foreign researchers. Recently, the differing nature of disciplines has been explicitlytaken into account with these general criteria being adjusted to the specific discipline.

17

these also varies across universities; one might lay special emphasis on the evaluation ofteaching, while another might emphasise its strategy, regional role or administration.

The body that has been actively evaluating research since the early 1970s is the Academy ofFinland. However, it does not specifically evaluate university research; indeed, until recentlythere has been little systematic and nation-wide ex-post evaluation of university research(Felderer and Campbell, 1998). Traditionally, evaluations have focused on individualscientists, research projects and teams. More recently, evaluations of research fields thatinclude university groups have been conducted. So far, 18 fields have been evaluated; the firstwas inorganic chemistry in 1983 while others include automation technology (1986), peaceresearch (1990), legal science (1994), and molecular biology and biotechnology (1997). Thesefield evaluations have focused on research excellence in terms of international prestige andcontributions to the frontier of science. Such evaluations are driven more by the need toimprove the quality of science than by budgetary decisions.

Evaluation criteria and methods

FINHEEC evaluations generally consist of three phases: first, the university carries out a self-evaluation and prepares a report. Secondly, the report is assessed by an external team whichvisits the university. In the third phase, the external team produces a final report.

To illustrate the method adopted in research field evaluations by the Academy, we will focuson the evaluation of electronics research (Felderer and Campbell, 1998). In 1995, theAcademy of Finland commissioned the Research Council of Natural Science and Engineeringto evaluate Finnish electronics research. A committee was set up and two internationalexperts appointed to lead the evaluation. The scope of electronics research was limited tocertain pre-defined sub-areas based on which 28 research groups from universities andinstitutes were identified and evaluated with respect to six aspects:1. their missions, visions and goals;2. the supply of resources and efficiency of their use3. scientific competence and degree of innovativeness;4. technological competence and co-operation with other researchers, industry and users;5. the national and international importance of the groups and of their results for the

scientific community;6. the relevance of each group’s research and results for industry.

18

As in the FINHEEC case, evaluations were conducted in three phases. First, a questionnairewas distributed to the research groups to document their performance and use of resources.Second, having examined the questionnaire results, evaluators visited each group andinterviewed group members. In the third phase, drawing on all the written and oralinformation, evaluators summarised their findings in a report written in English. Researchgroups were given an opportunity to comment before the report was published. In addition torecommendations for individual groups, the report also analysed the current state ofelectronics research in Finland.

The Ministry of Education has suggested creating a performance-based funding mechanismsimilar to the RAE in the UK. The proposal was contained in the report ‘Management byResults’ produced by a working group set up by the Ministry to analyse future trends anddevelop the existing system. The report suggested that 35% of operational funds should beallocated on the basis of research performance. The performance of all groups withinuniversities should be evaluated by the Academy of Finland every three years using peerreview. Research units would be graded on a five-point scale which, in turn, would determinethe amount of funds they received (Kaukonen, 1997). The suggested evaluation mechanismwas heavily criticised by almost all universities and other relevant parties and the proposalwas ‘frozen’ by the Ministry. The main objections were that the mechanism would give theAcademy of Finland undue influence over policy, and that it would result in a normativestandard for resource allocation among universities (Kaukonen, 1997).

3.6 Eastern European Countries: Hungary, Poland and the Slovak Republic

Throughout Eastern Europe after 1945, national research systems were based on the Soviettripartite model in which universities focused on teaching, basic research was conducted inAcademy of Sciences institutes and applied research was carried out in institutes underministries (Frankel and Cave, 1997). Governments in the region also adopted the Sovietsystem for funding research largely through block grants to institutes, a system in which thescientist-administrator had great power over the internal distribution of funds. Favouritismand political connections often gave rise to poor quality research teams with the rightconnections being funded, while high-quality research teams were under-funded (ibid.).

This system has changed radically over the last ten years as Eastern European countries havebeen transformed from centrally-planned to more open, competitive market economies. Thishas had a major impact on science policy. The autonomy of science in terms of self-evaluation through peer review, an autonomy completely subordinated to the central plan

19

during the communist era, has been restored. In the early part of transition, economic crisessaw research evaluation emerge as an important tool mainly to examine where to cut budgetswithout completely destroying research activities. Peer review has now become the mainevaluation mechanism used to allocate funds (Hangos, 1997; Zilahy and Lang, 1997).

In the post-communist era, although research systems have experienced major changes, theAcademies of Sciences continue to play a major role and research evaluations have mainlyfocused on Academy institutes. There has been no comprehensive evaluation of universityresearch in Eastern European countries apart from the Slovak Republic where an evaluationof university departments is used to help determine the allocation of core funding (personalcommunication with Štefan Zajac). In the following paragraphs, the evaluation of Academyinstitutes in Hungary and the Slovak Republic is discussed, followed by a description of thefunding mechanism in Poland.

For several decades, the Hungarian Academy of Sciences has conducted a fairlycomprehensive evaluation of all its institutes almost every year. The evaluation in 1992 had aparticular impact as it coincided with the cutting of research funds as a result of the economiccrisis. The evaluation was conducted in two phases. In the first, each institute was evaluatedwith a view to restructuring the Academy’s research network and reformulating its tasks. Inthe second phase, each research unit or group within the institutes was evaluated to providethe basis for a more differentiated distribution of funds. The evaluation method was peerreview combined with quantitative indicators. The findings led to recommendations coveringthe function of the Academy’s research network, the management of human resources,financial conditions and organisational changes (Zilahy and Láng, 1997).

In the Slovak Republic in 1992 the Academy of Sciences and the universities set up anaccreditation committee to evaluate research in the institutes and university departments. Inthe evaluation of Academy institutes, the following indicators were used (Tino, 1997):• a full list of scientific publications covering the previous five years, classified by type,

together with ten representative publications;• number of citations (included in the SCI) during the previous five years;• membership of journal editorial boards;• participation in conferences, membership of international science organisations, relations

with international science community (co-operative agreements, joint projects etc.).

20

Based on these quantitative data, research institutes were evaluated and graded. For universitydepartments, the above indicators were used with the following additions:• titles and scientific qualifications of all teaching staff;• number of postgraduate students;• research activity of senior research staff;• post-university employment of graduates, number of applications from foreign students.

In addition to research criteria, faculties are also evaluated in terms of teaching andmanagement. Since 1992, such evaluations have been conducted every three years byuniversities under the supervision of the Accreditation Committee. Faculties are classifiedinto one of four grades (A = best, D = worst) which, when combined with student numbers,determines the amount of money each receives (personal communication with Štefan Zajac).

Poland26

In Poland, after the fall of Communism, a new system was established for managing researchbased on the 1991 Act, which set up the Committee for Scientific Research (CSR). The CSRchair is appointed by Parliament, and two-thirds of the committee members are researcherselected by the scientific community, the rest being ministers. CSR is responsible forformulating and executing science policy, including the distribution of funds through severalcompetitive channels. All institutions, including university faculties, have to compete forfunds.

Evaluation criteria and methods

Research funds administered by CSR are an important source for universities. In addition togeneral university funds allocated on the basis of education profile (e.g. student numbers),universities also compete for funds with other institutions through two CSR schemes. Thefirst is a grant system for individuals and research teams based on open competition;applications are peer-reviewed by panels of active researchers. The other is the so-calledstatutory funding that is distributed to faculties within universities (e.g. mathematics,sociology) on the basis of an ex-post evaluation. Each year, institutions submit applicationsthat include a record of the past year’s achievements and a research plan for the coming years.The assessment is conducted by expert panels of scientists who assign institutions to a

26 This discussion is based on Jablecka (1995 and personal communication).

21

category (A = best, C = poorest). The final funding allocation is decided by the CSRCommittee for Basic and Applied Research.27

In 1998 a new formula was introduced for statutory funding. Previously, the funding level forevaluated units was determined according to the categorisation of institutions (‘thealgorithm’) based on a combination of quantitative and qualitative methods (the latter ofwhich were criticised for their subjectivity). The new ‘parametric system’ is based almostentirely on quantitative methods. It consists of the number of points for performance, Rp, andfor so-called general results, Rg. The total number of points, R = Rp + Rg, is divided by thenumber of staff (N) to yield an indicator of effectiveness (E). This is the basis fordetermining, every three years, the classification of institutions (into one of five grades) andhence the level of funding. However, the new funding formula has been under discussion(including the weights ascribed to each indicator) and will probably be changed.

4. University Research Evaluation and Funding in the Asia Pacific Region

In the Asia Pacific region, Australia and Hong Kong are two countries that carry out nationalresearch evaluations and use the results to decide the allocation of funds (Atkinson andMassy, 1996; Bourke, 1997). While the approach in Hong Kong is similar to the RAE in theUK, in Australia the evaluation is concerned mainly with the quantity of published output.

4.1 Australia

Since the adoption in 1988 of the Unified National System in the HE sector, the number ofuniversities eligible for government research funding has increased. The CommonwealthGovernment provides the majority of such funds through a dual-support system28 consistingof an institutional operating grant and a separate, targeted grant scheme. Institutional fundsare allocated as block grants with universities having discretion as to how they distributethese internally. The targeted Commonwealth Competitive Grant scheme is managed byvarious research councils and government agencies, among which the Australian ResearchCouncil (ARC) is the largest provider. There are some 40 competitive grant schemes.

27 The relative proportion of funds received by units of classes A, B and C is, respectively, 1.151, 0.951 and

0.550 – see Committee for Scientific Research (KBN), http://www.kbn.gov.pl/en/general/reseval.html(accessed 31.08.01). Institutes of the Polish Academy of Sciences and other government researchestablishments are financed according to the same statutory formula and evaluation criteria as universityfaculties.

28 Over 90% of university research is funded by the Commonwealth Government. Other funders are state andlocal government, industry, non-profit organisations and overseas providers (Industry Commission, 1995).

22

In addition to institutional core funding, there are two schemes focusing on researchinfrastructure: the Research Infrastructure Equipment and Facilities (RIEF) scheme, and theResearch Infrastructure Block Grant (RIBG) scheme for universities. The former aims toencourage collaboration between institutions and is based on applications submitted byuniversities to the Committee for International and National Cooperation (CINC). The RIBGscheme is formula-based with allocations reflecting each university’s success in gainingcompetitive research funds, as calculated using the National Competitive Grants Index(NCGI) (based on data collected by DETYA through a procedure described below).

Whereas peer review of grant applications is the main method for allocating targeted grants(Industry Commission, 1995), core funding is distributed among universities based on aformula. Following consultation with the HE sector, the Relative Funding Model wasintroduced in 1990. In this, funding for teaching and research training is separated from thatfor research. The former is based on student numbers. The latter, the ‘Research Quantum’,29

was initially based on the success of universities in winning Commonwealth CompetitiveGrants. However, it was later recognised that success in obtaining such funds did not fullyrepresent research performance and the criteria were broadened to include other fundingsources. In addition, measures other than funding, such as publications and higher degreecompletion rates, were incorporated in the formula.

In 1993, the Education Minister announced that from 1995 the Research Quantum would beallocated on the basis of a new Composite Index. Since the restructuring of GovernmentDepartments in 1998, the Department of Education, Training and Youth Affairs (DETYA)has been responsible for managing the data for the Composite Index, for calculating fundingallocations, and for advising the Minister on the appropriate components in the Index and theweighting of elements within the Index. Both research input and output measures areincorporated in the Composite Index:

1 Research input measures (funding):a) the amount of each university’s funding from Commonwealth competitive grants;b) other public sector research funding;c) industry and other research funding.

29 The Research Quantum is used to support research separate from those activities directly linked to teaching

and research training.

23

2 Research output measures:

a) numbers of research and scholarly publications produced by staff and students;

b) numbers of higher degrees completed (Masters and PhD).

The various components of the Index have been weighted differently from year to year, as thebalance between them is still being refined. For 1999, the total Research Quantum fundingavailable was $221M. For instance, if university X’s shares of total national funding,publication and degree completion, averaged over the last two years are 4.5%, 3.6% and 5.3%respectively, then its Composite Index is obtained by first multiplying the share by thecorresponding weighting for each component (80%, 10% and 10% respectively), and thenadding them together. The Composite Index of University X represents its share in totalresearch activities for all universities, and its Research Quantum allocation is the CompositeIndex multiplied by the total Research Quantum available.

Compared to the RAE in the UK, the Research Quantum system is more mechanistic.Research performance is evaluated solely on the basis of quantitative measures – on thevolume of research within universities. It is “confined to counting annually the gross numberof undifferentiated entrants each institution can place in the classification categories whichare weighted for funding purposes” (Bourke, 1997, p. 25). In 1994 a sub-committee of theAustralian Vice-Chancellors’ Committee proposed that more qualitative aspects should beincorporated. However, this requires considerable development work, which has delayed itsrealisation. In addition, as funding generated through the Research Quantum system hasbecome more important to universities, any major change would need careful justification(ibid.).

Bourke (1997) suggests that, while the UK RAE fulfils three functions – a competitive sourceof discretionary income, a reward for the quality and/or volume of research output, and aninstrument of policy – the Research Quantum performs only one of these. It functions as areward for the volume of research output but it does not reward quality and is seldom used asan instrument of policy. Instead, Bourke has proposed a funding mechanism that combinesthe Research Quantum with the ‘professional judgement’ element of the RAE.

24

4.2 New Zealand30

There are 39 institutions within the HE sector – seven universities, twenty-five polytechnics,four education colleges and three wananga (education institutions devoted to perpetuating anddeveloping Maori knowledge). Funding for universities is provided by the Ministry ofEducation through Vote Education. About 8% of government funding to universitiesrepresents research core funding. Other sources of research funding are the Public GoodScience Fund (PGSF) and the Marsden Fund provided respectively by the Foundation forResearch, Science and Technology (FRST) and the Royal Society of New Zealand.

PGSF funds are allocated in a competitive bidding system and are open to applications fromany public or private organisation, including government departments and individuals with anability to conduct research. The fund is targeted at 17 priority areas determined bygovernment. The Marsden funds are also allocated on a competitive basis through peer reviewand are targeted at ‘blue sky’ research. Applications are assessed on three criteria: scientificmerit of the proposal, track record of the researcher, and the contribution to developingresearch skills. Whereas the PGSF and Marsden funds are allocated on the basis of peerreview, the core research funding is distributed to universities in the form of tuition subsidiesbased on student numbers. From 2000, however, core research funding is being separated intotwo parts; 80% of the total from Vote Education continues to be allocated based on studentnumbers, but the remaining 20%31 is allocated through a contestable pool after a peer reviewof applications using the following criteria:

1. Demonstrated quality and capacity of researchers – track record as reflected inrecognised publications and national/international significance of past research.

2. Quality of the proposed research portfolio – design and purpose of the research portfolio,feasibility, research methods, skills and technologies to be developed and used, andrelationship to other portfolios and programmes.

3. Strategic focus – whether and how the proposed research will increase New Zealand’sinnovativeness and human resource capabilities.

4. Cost-effectiveness – appropriateness of the proposed costs.

Operation of this scheme is being reviewed in 2001. If it proves to have been successful indelivering high quality research in a cost-effective manner, the portion of contestable fundswill be increased gradually up to 80% over a three to five-year period.

30 This section is mainly based on Ministry of Education (1998).31 These funds are targeted at advanced basic research with strategic importance. Strategically-focused research

projects of up to three years are preferred to short-term projects.

25

4.3 Hong Kong32

The HE sector in Hong Kong comprises three research universities, two polytechnics, twoliberal arts colleges, a teacher education institute, a performing arts conservatory and an openuniversity. All but one are degree-awarding institutions (and will be referred to asuniversities). They are publicly funded33 through the University Grants Committee (UGC), anon-statutory body that acts as advisor to government on the development, funding andquality assurance of HE. Research is supported through a dual funding system. The UGCprovides institutional funding covering recurrent teaching and research costs. In addition, theResearch Grants Council (RGC), which operates under the aegis of the UGC, allocatesresearch project grants based on rigorous peer review involving international experts in orderto concentrate funds on high quality research (RGC, 1998). With regard to the researchcomponent of the institutional core funding, this is allocated on the basis of researchperformance using (since 1993) a mechanism very similar to the British RAE. Hong Konghas even adopted the name - Research Assessment Exercise - for its evaluation process.

Three RAEs have been carried out by the UGC in 1993, 1996 and 1999. Their aim was “tomeasure the output and quality of research of the UGC-funded institutions by cost centre asthe basis for allocating some of the research portion of the institutional recurrent grant for thenext triennium in a publicly accountable way” (UGC, 1999). Methodologically, the HongKong RAE is similar to that of the UK. Indeed, in 1993 UK experts involved in the RAE inHong Kong provided extensive advice (French et al., 1998). In 1999, the unit of analysis wasa set of 58 ‘cost centres’ listed by UGC, and each university department or research unit wasrequired to map their research onto these. The 58 cost centres are evaluated by 12 panels (e.g.Biology, Physical Sciences).

5. Advantages and Disadvantages of Research Assessment-Based FundingSystems

This paper has analysed and compared current practices in national assessments of universityresearch and the relationship to the funding allocation process in a large number of countries.It has identified a continuum of approaches to university research funding with, at oneextreme, the pure performance-based ex-post evaluation of research and, at the other, the

32 This section is based on French et al. (1998) and UGC (1999).33 In addition, there are two publicly funded technical colleges that provide 3-year higher diploma courses.

26

allocation of resources for university research through general institutional funding basedsolely on educational size. Although few of the countries considered here have actuallyimplemented the first approach, we find clear evidence of a number moving in that direction.Most of the countries examined have either recently implemented some form of performance-based allocation or are considering doing so in the near future.

Current practice in university research evaluation and funding is the result of policy pressuresfor greater accountability for research funding. Among the countries analysed, the UK isusually considered to be leading this trend (Geuna, 1999). In the early 1980s, growingconstraints on public funding and the prevailing political ideology in the UK resulted inpolicies aimed at greater accountability and selectivity. In the 1990s, other Commonwealthand European countries’ university research funding mechanisms have been based, at least inpart, on some form of research output measurement. However, in many cases, the academiccommunity has resisted the introduction of specific evaluation criteria for university researchoutput or at least the linking of evaluation results directly to research funding decisions.

At present, with the possible exception of the UK, we lack the detailed data on universityresearch inputs and outputs over a sufficiently long period of time needed to identify withconfidence the results of different resource allocation systems. Nonetheless, some preliminarythoughts on the advantages and shortcomings of the two approaches to university researchfunding are presented here. Table 2 summarises these.

{Table 2 about here}

Keeping in mind the discussion in Section 2 surrounding the difficulties involved inconstructing research performance indicators for individuals and more especially fororganisations, probably the main reason for introducing a performance-based approach to thefunding of university research is the belief that it is meritocratic, providing a reward for thosewith a record of successful research, and thus constituting a strong incentive for improvingthe quality of research as measured by the selected performance indicators. Similarly, aperformance-based system can identify where resources have previously been spent ondepartments that have been ’less successful’ in their research, and can shift resources towhere they will be used with greater effect. Hence, a performance-based funding system canlead to increased efficiency in how research resources are used in the short term. It willcertainly encourage researchers to complete their research and to make available the results toothers through publication and other means of dissemination. Consequently, a performance-

27

based funding system is much more likely to provide the explicit public accountability forgovernment funds invested in university research, and this represents another major reasonwhy politicians have been keen to introduce such a system in recent years.

Where performance-based funding systems have been in operation for a while, theyundoubtedly encourage universities and university departments to formulate more explicit andcoherent research strategies than formerly. Indeed, a performance-based system provides amechanism for linking university research to government policy, shifting priorities acrossfields, for example, where this is deemed desirable. One policy that many governments havebeen pursuing over the last decade or more is that of increasing selectivity and concentration.As we have moved from relatively elitist higher education systems to an era of ‘mass’ highereducation, so the number of universities has tended to increase, with former polytechnics orfurther education colleges being awarded the status of universities. This brings the danger ofresearch resources being spread too thinly. A performance-based research funding system canbe used to concentrate resources on a smaller number of institutions, which, assuming thatthere are important scale and scope economies, could compete with the world leaders abroad.

A performance-based funding system inevitably suffers from certain disadvantages. First ofall, obtaining reliable and comparable information on research performance across institutionsis costly, particularly in terms of the time required on the part of those who compile thatinformation and those who must analyse it in order to draw conclusions about theperformance of institutions and departments. An assessment process based on peer review isespecially labour-intensive when all the nation’s universities and their constituentdepartments have to be judged. Nor do indicator-based approaches offer much of a shortcut;if the conclusions drawn from them are to be robust, considerable effort must be devoted toensuring that the data are accurate and reliable.

A second potential drawback with a performance-based funding system is that it will tend toencourage all universities to compete in the same arena, for example, in conducting researchthat yields articles acceptable to leading international academic journals. It may, therefore,result in a shift towards the ‘homogenisation’ of research and of universities, with feweracademics prepared to experiment with different ideas and new approaches. The resultingdecrease in the amount of diversity may be harmful in the longer term for both scientificknowledge and higher education. Similarly, a performance-based funding system maydiscourage some researchers from engaging in more innovative and risky research because the

28

risks of failure may be deemed too great; they will instead pursue ‘safe’ research that is morelikely to yield a demonstrable output.

A performance-based system in which publications output is a key criterion will also tend toencourage ‘publication inflation’, with researchers seeking to split the results of their workinto the ‘least publishable units’ (the phenomenon of ‘salami publication’). Indeed, whateverthe evaluation approach, whether it is based on indicators or not, some academics will almostcertainly respond with an element of ‘game playing’ - i.e. engaging in behaviour that resultsin their looking good (in terms of assessment criteria or indicators) without necessarilyimproving their research performance. In addition, in a system where journal articles or booksare central, researchers will perhaps be more inclined to pursue traditional ‘academic’research, irrespective of whether this is most beneficial to society.

Another danger accompanying performance-based funding for research is that it can result inwidening the gap within universities between research and teaching. If the rewards forsuccessful research are greater than those for high quality teaching, then academics willrespond to the incentive offered by the former at the expense of teaching.

While the intention of a performance-based funding system for research is to reward and toencourage, the assessment inevitably focuses on past rather than current performance, not tomention future potential. The consequence of such a system is that the existing research eliteis further reinforced and the status quo is maintained. Those institutions and departments thathave done well in the past will continue to attract a disproportionate share of the researchresources, depriving others of the resources they need to exploit their full potential andbecome leaders in their field. Finally, a performance-based system, by making it easier forgovernment to shift priorities, might be seen as leading to an excessive level of governmentinfluence or even ‘interference’ in university research. For many, such a decrease inautonomy would be seen as being potentially harmful to the long-term health of research andof higher education.

Despite criticisms of their inefficiency, the fact that university research-funding systems havefor many years been based on educational size is an indication that these systems must havecertain advantages. The first and most obvious is that such a system is simple and cheap tooperate; all that is required are comparable data on numbers of students and lecturers.Another undoubted benefit is that it provides institutions and departments with ‘seed corn’funds to invest in new or younger members of staff or in new areas. Likewise, becauseresearchers are not forever focusing activities towards the achievement of a high ranking in

29

the next RAE, they may be more inclined to engage in longer-term research and scholarshipthat may not produce a specific output for many years. They may also be more likely to trydifferent approaches and to work on more speculative ideas. Such diversity may be beneficialfor intellectual progress, especially in areas where there is no dominant paradigm or cleartrajectory. A funding system based on educational size also offers the opportunity foracademics at any institution, whatever its research reputation, to become involved in research,again adding to the diversity of ideas and approaches to the potential benefit of scientificknowledge.

Given that research funds depend on teaching, a funding system based on educational sizewill encourage the integration of teaching and research to the benefit of each. High-qualityteaching, because it is likely to bring in more students, thus increasing the educational size ofthe institution and with it the flow of research funds, will be highly regarded by colleaguesand perhaps rewarded by the institution. Lastly, a research funding system based oneducational size offers institutions and individuals maximum autonomy, a characteristic thatmany academics believe is essential for the long-term health of universities and of research.

As can be seen from Table 2, the disadvantages of a system of funding university researchbased on educational size are in many cases a mirror image of the advantages of aperformance-based system discussed above. In particular, a system that distributes researchresources to universities solely on the basis of the volume of teaching provides little directincentive to improve research performance, whether at the individual or the institutional level.Over time, this could bring stagnation in research activities. Such a situation can be seen in itsmost extreme form in the academic institutions of the former Soviet Union and other EastEuropean countries prior to transition. Another problem that such a system may produce isthat it gives excessive power to those officials who distribute core funding within theinstitution; funding can become a reflection of ‘political’ factors rather than research success.

A concern of politicians and others with a research-funding system based on educational sizeis that it offers little public accountability for the funds supposedly provided for researchpurposes. It may encourage ‘ivory tower’ research of little interest except to the researchersconcerned and of no relevance to society more broadly. This, in turn, may reinforce publicstereotypes of ‘lazy’ academics who have jobs for life, take long holidays and have thefreedom to do what they like. Another fundamental concern with a research funding systembased on educational size is that there may be little correlation between student numbers andthe level of research effort of a department or institution. An institution that chooses to devote

30

the greater proportion of its time to teaching will receive as much funding for research asanother of similar size that elects to devote more of its efforts to research.

With a research-funding system based on educational size there is also a strong chance thatthe overall distribution may bear little relationship to stated government policy for research.For example, a burgeoning in the number of students choosing to pursue media studies mayresult in a rapid increase in that field’s share of resources, irrespective of whether policy-makers believe it to be a research priority. Conversely, a sharp reduction in the number ofstudents choosing to study, say, physics may be translated into a substantial reduction in itsshare of the national research funding total, regardless of whether this is in line with currentpolicy. Lastly, if other source of research funding are marginal, the distribution of researchfunds solely on the basis of educational size is likely, in all but the largest or richest ofcountries, to result in those resources being spread too thinly, with the consequence that noinstitution is able to compete successfully with world-leading foreign institutions.

6. Conclusions - Do the Benefits of Performance-Based Funding SystemsOutweigh the Costs?

In the previous section, we analysed the advantages and disadvantages of performance-basedsystems for funding university research compared with systems based on educational size. Aswe have seen, there has been a trend over the last decade or so in the direction of the former.This raises the question: do the benefits of a performance-based system outweigh the costs?The lack of detailed data on university research inputs and outputs over a sufficiently longperiod of time restricts the possibilities of a proper cost-benefit analysis. However, aframework that addresses the possible evolution of the cost-benefit relationship is introducedhere. To make this move, we need to progress from the essentially static analysis of theprevious section to a more dynamic analysis. Figure 1 indicates how the benefits and costsmay vary over time.

{Figure 1 about here}

When research assessment is first introduced, it will often face considerable opposition fromacademics. Many will see it as a temporary aberration that will quickly disappear once thepoliticians or ‘bureaucrats’ who introduced it recognise its infelicities. Moreover, most ofbenefits will not accrue until several years from the first assessment. Hence, the benefit curvein Figure 1 is shown as initially rising gradually from zero then accelerating. In contrast, the

31

costs of introducing the assessment will be substantial. An appropriate assessment proceduremust be designed, negotiated with universities and agreed. Universities and departments mustput in place suitable procedures for compiling all the necessary information. A peer reviewsystem must be constructed and peers must spend considerable amounts of time judging theperformance of all the university departments in their field. Thus, for an initial period, thecosts will probably outweigh the benefits.

Over time, the benefits of a performance-based funding system outlined in Table 2 will grow.When it is carried out for the second time, most academics will accept that assessment is hereto stay and take it more seriously. Some will have begun to reap the rewards of the firstassessment in the form of additional funds, encouraging them as well as stimulating others toimprove their research. Some of the weaker groups will have been identified in the first roundof assessment and resources shifted to more productive groups, enhancing the overall‘efficiency’ of university research. Departments and universities will be developing clearerresearch strategies, although a few more years will be needed before the full benefits of theseemerge. At some point, the benefits may come to exceed the costs so the benefit curve inFigure 1 is shown as rising above the cost curve (we consider the case where this does nothappen below).

At a somewhat later stage, however, the increase in the benefits of a funding system based onresearch assessment will probably begin to level off. Although it is difficult to produceconclusive evidence, our impression based on the UK, where there has been the longestexperience of research assessment exercises, is that probably after three or so such exercisesthe level of benefits reaches a peak. It then begins to turn down as diminishing returns set in.The more obviously weak performers have already been identified and resources redirected sothe scope for further efficiency gains is diminished. The institutional research strategies put inplace at the time of the first or second assessment have taken effect and the scope for futuregains is less. For all these various reasons, the benefit curve in Figure 1 is shown as fallingaway from its peak although at an ever decreasing rate.

What of costs during this time? Figure 1 shows the cost curve as monotonically (although notnecessarily linearly) rising. What are the reasons for this? First, as more and more fundingcomes to depend on the outcome of the assessment, departments and universities will take theexercise more and more seriously and devote more and more effort to it – both in terms ofdeveloping institutional strategies and of implementing them to prepare all the materialneeded to succeed in the next assessment. Secondly, the early attempts at research assessment

32

will almost inevitably be subject to the criticisms of academics. Improvements to themethodology will therefore be introduced which will mean that more effort has to be devotedto preparing submissions to later assessments as well as to judging them.

Thirdly, as the numbers of universities and departments that take the research assessmentsseriously and attempt to improve their performance, an ‘arms race’ or ‘Red Queen effect’ willresult. As competitors become stronger (or learn to ‘play the game’ more effectively), so everincreasing organisational efforts must be devoted to just remaining at the same level, let aloneto improving relative positions. Lastly, some of the disadvantages listed in Table 2 and theirassociated ‘costs’, which initially would be relatively small, are likely to have come intoeffect (e.g. the gradual discouraging of risky or longer-term research).

Consequently, if the above assumptions about the trends in benefits and costs34 are valid, onewould expect that at some point the benefit curve would fall below the cost ‘curve’. Fromthen on, any subsequent assessment exercises will involve costs that are greater than thebenefits.

Until now, we have assumed that the peak of the benefit curve rises above the cost curve. Butwhat if this is not the case? Figure 2 suggests what the respective curves might then look like.The benefit curve has a similar shape but a lower peak so that it never rises above the costcurve. In this case, the cost ‘curve’ is shown as monotonically decreasing because in suchcircumstances research assessment would be widely seen as not worth the effort. Althoughperhaps required by politicians, academics would merely ‘go through the motions’. Ifassessments continued to be imposed, universities would find ways of doing less work tosatisfy the assessors. However, despite this hypothesised downward slope to the cost curve,once the benefit curve has peaked and begun to fall off, the gap between the benefits and thecosts is likely to grow, at least for some considerable period. In time, the weight of evidencewould presumably lead to the assessments being discontinued.35

{Figure 2 about here}

34 Summed over all the institutions participating in the assessment as well as taking into account the costs and

time of those engaged in carrying out the assessment.35 Some might suggest that this is the situation in relation to the Teaching Quality Assessments to which UK

universities are also subject.

33

Do the actual benefit and cost curves resemble those in Figure 1 or those in Figure 2? Theshort answer is that we cannot be sure. More research on the nature and extent of the benefitsand costs is needed to answer this question. However, our impression (albeit a subjective oneonly) from those countries where research assessments have been in operation for the longest(and, in particular, in the UK) is that the benefits listed in Table 2 are real and substantial andthat, for a period of time, they do indeed exceed the costs. Furthermore, it is perhapssignificant that more and more countries have begun to introduce performance-based systemsfor allocating at least part of their university research resources or are contemplating doing so.

Yet even if Figure 1 is a more accurate reflection of the relative sizes of the benefits and coststhan Figure 2, we should not forget that at a certain time the benefit curve is likely to againfall below the cost curve. Just as it is our view that for a while the benefits in the UK roseabove the costs, so we would argue that the benefit curve in the UK has already peaked anddiminishing returns have set in. One can only speculate as to whether the benefit curve has yetfallen below the cost curve or has still to do so. However, it is perhaps significant that thereare suggestions that the 2001 research assessment exercise will be the last one, or that theexercise is likely to be re-designed in such a way that the costs of conducting it aresubstantially reduced.36

Given the substantial (and perhaps increasing) costs of a full performance-based system forallocating research resources to universities, it is worth concluding by pointing to theadvantages of a hybrid system, that is, one based partially on performance (incentive-creating)and partially on educational size (cost-minimising). Among the countries analysed, TheNetherlands, Finland and Denmark have various forms of hybrid system. Of particularinterest is the Dutch system. In both the UK and The Netherlands, research assessment isbased mainly on informed peer review. However, unlike the UK, which focuses on a one-dimensional concept of quality and links the exercise directly to funding, in The Netherlandsthe quality of research performance is assessed in relation to four dimensions (scientificquality, scientific productivity, scientific relevance and long-term viability of the research).The resultant ranking is not directly linked to the funding received by the university. Thegreater breadth of the quality measurement encourages diversity in the research conducted byuniversities and in the strategies adopted to improving their research. The lack of directconnection with funding, results in significantly lower organisational and administrative costswhile, at the same time, the assessment creates a degree of competition based on reputationthat can generate most of the advantages identified in Table 2. For those countries

36 See e.g. Floud (2001).

34

contemplating a shift towards a performance-based funding system, such a hybrid system mayperhaps offer a better trade-off than the more expensive approach of the UK researchassessment exercise.

35

References

Atkinson, H. and W. F. Massy. 1996. Quantitative Funding Models. In W. F. Massy (Ed.)

Resource Allocation in Higher Education. Michigan: The University of Michigan

Press.

Biggeri, L. and L. Scarpitti. 1998. Evaluation in the Italian University System. Paper

presented at the International Conference on Evaluation: Profession, Business or

Politics? Rome, October 29-31.BMBF. 1998. Facts and Figures 1998. Bonn:

Bundesministerium für Bildung und Forschung.

Boffo, S. 1996. EVALUE, State of Play Report: Italy. Contract SOE2 - CT 95 - 2004.

European Commission, DG XII, TSER (Targeted Socio-Economic Research)

Programme.

Bourke, P. 1997. Evaluating University Research: The British Research Assessment Exercise

and Australian Practice. Commissioned Report No. 56. Canberra: National Board of

Employment, Education and Training.

Bourke, P. and B. Martin. 1992. Evaluating University Research Performance - What

Approach? What Unit of Analysis. ANU & SPRU.

Campbell, D. F. J. and B. Felderer. 1997. Evaluating Academic Research in Germany.

Patterns and Policies. Political Sciences Series No.48. Wien: Institute for Advanced

Studies.

Cave, M., S. Hanney, M. Henkel and M. Kogan. 1997. The Use of Performance Indicators in

Higher Education. Higher Education Policy Series. London: Jessica Kingsley

Publisher.

Cooper, C. and D. Otley. 1998. ‘The 1996 Research Assessment Exercise for Business and

Management’. British Journal of Management, Vol. 9: pp. 73-89.

36

Daniel, H. D. and R. Fisch. 1990. ‘Research Performance Evaluation in the German

University Sector’. Scientometrics, Vol. 19(5-6): pp. 349-361.

El-Khawasi, E. and W. F. Massy. 1996. ‘Britain’s “Performance-Based” System’. In W. F.

Massy (Ed.) Resource Allocation in Higher Education. Michigan: University of

Michigan Publishing.

Felderer, B. and D. F. J. Campbell. 1998. Evaluation der Akademischen Forschung. Vienna:

Institute for Advanced Studies.

Floud, R. 2001. ‘Universities Are Sinking Under Inspection Load’. The Times Higher

Education Supplement, No. 1479 (23 March): p.16.

Frankel, M. S. and J. Cave. 1997. ‘Introduction’. In M. S. Frankel and J. Cave (Eds)

Evaluating Science and Scientist: An East-West Dialogue on Research Evaluation in

Post-Communist Europe. Budapest: Central European University Press.

French, N. J., W. F. Massy, P. K. Ko, H. F. H. Siu, and K. Young. 1998. ‘Research

Assessment in Hong Kong’. Journal of International Education, Vol. 10(1)

Gellert, C., ed. 1993. Higher Education in Europe. London and Philadelphia: Jessica

Kingsley Publisher.

Geuna, A. 2001. ‘The Changing Rationale for European University Research Funding: Are

There Negative Unintended Consequences?’ Journal of Economic Issues, Vol. 35(3).

Geuna, A. 1999. The Economic of Knowledge Production: Funding and the Structure of

University Research. Cheltenham: Edward Elgar.

Geuna A, Hidayat D and Martin B. 1999. ‘Resource Allocation and Research Performance:

The Assessment of Research’. Report prepared for the Higher Education Funding

Council of England, Brighton: SPRU.

37

Hangos, K. 1997. ‘The Limits of Peer Review: The Case of Hungary’. In M. S. Frankel and J.

Cave (Eds) Evaluating Science and Scientist: An East-West Dialogue on Research

Evaluation in Post-Communist Europe.. Budapest: Central European University Press.

HEFCE. 1994. Circular Letter, RAE96 1/94, Bristol: Higher Education Funding Council for

England.

HEFCE. 1997. A Guide to Funding Higher Education in England. Accessed 23 September

1997. http://www.niss.ac.uk/education/hefce/funding/index.htm

HEFCE. 1998. Circular 3/98, Bristol: Higher Education Funding Council for England.

Helander, E. 1995. ‘Evaluation Activities in the Nordic Countries’. Scientometrics, Vol.

34(3): pp. 391-400.

Hills, P. V. and A. J. Dale. 1995. ‘Research and Technology Evaluation in the United

Kingdom’. Research Evaluation, Vol. 5(1): pp. 35-44.

Industry Commission. 1995. Research and Development. Report No. 44, Volume 1. Canberra:

Assistant Treasurer.

Irvine, J., B. R. Martin, and P. A. Isard. 1990. Investing in the Future, An International

Comparison of Government Funding of Academic and Related Research. Aldershot:

Edward Elgar.

Kaukonen, E. 1997. Evaluation of Scientific Research in Finland. In OECD (Ed.) The

Evaluation of Scientific Research: Selected Experiences. Paris: OECD.

Koelman, J. B. J. 1998. ‘The Funding of Universities in The Netherlands: Developments and

Trends’. Higher Education, Vol. 35: pp. 127-41.

Kogan, M. 1998. ‘The Treatment of Research’. Higher Education Quarterly, Vol. 52(1): pp.

48-63.

38

Kogan, M. 1989. ‘The Evaluation of Higher Education: An Introductory Note’. In M. Kogan

(Ed.) Evaluating Higher Educationed.. London: Jessica Kingsley Publisher.

Kushner, S. 1996. ‘The Research Assessment Exercise versus Development in Higher

Education: A Response to Richard Pring’. British Journal of Educational Studies,

Vol. 44(1): pp. 5-8.

Luukkonen, T. 1995. The Impacts of Research Field Evaluations on Research Practice.

Research Policy, Vol. 24: pp. 349-365.

Martin, B. R. and J. Irvine. 1983. Assessing Basic Research: Some Partial Indicators of

Scientific Progress in Radio Astronomy. Research Policy, Vol. 12: pp. 61-90.

Martin, B. R. and J. E. F. Skea. 1992. ‘Academic Research Performance Indicators: An

Assessment of the Possibilities’. Brighton: SPRU.

Massy, W. F. 1996. Resource Allocation in Higher Education. Michigan: The University of

Michigan Press.

Minister for Education and Science. 1979. Beleidsnota Universitair Onderwijs. The Hague:

Minister for Education and Science, Minister for Science Policy and Minister for

Agriculture and Fisheries.

Ministry of Education. 1998. Tertiary Education in New Zealand, Policy Direction for the

21st Century, Auckland NZ: Ministry of Education, New Zealand.

Ministry of Education. nd. Management by Result (see http://www.minedu.fi/eopm/hep).

Noll, R. G. (ed.) 1998. Challenges to Research Universities. Washington, D.C.: Brookings

Institution Press.

OECD. 1998. University Research in Transition. Paris: OECD.

OECD. 1997. The Evaluation of Scientific Research: Selected Experiences. Paris: OECD.

39

OECD. 1987. Universities Under Scrutiny. Paris: OECD.

Research Grants Committee. 1998. ‘Refutation of Article in "Next" Magazine of 9 October

1998’, http://ww.ug.edu.hk/RGC/documents/E_XNEXT.html, Accessed 20/7/99.

Rip, A. and B. J. R. van der Meulen. 1995. ‘The Patchwork of the Dutch Evaluation System’.

Research Evaluation, Vol. 5(1): pp. 45-53.

Royal Society of Chemistry. 1995. Chemistry in the UK - Will it Survive? Conclusions of the

Royal Society of Chemistry Workshop. London: Royal Society of Chemistry.

Steen, J. van and M. Eijffinger. 1998. ‘Evaluation Practices of Scientific Research in The

Netherlands’. Research Evaluation, Vol. 7(2): pp. 113-22.

Suter, L. E. 1997. ‘United States: The Experience of the NSF's Education and Human

Resources Directorate’. In OECD (Ed.) The Evaluation of Scientific Research:

Selected Experiences. Paris: OECD.

Tino, J. 1997. ‘Institutionalizing Evaluation of Research and Education in the Slovak

Republic’. In M. S. Frankel and J. Cave (Eds) Evaluating Science and Scientists: An

East-West Dialogue on Research Evaluation in Post-Communist Europe. Budapest:

Central European University Press.

UGC. 1999. Research Assessment Exercise 1999, Guidance Notes. Hong Kong: University

Grants Committee.

Meer, P. van der 1999. ‘Funding and Allocation of Resources in Higher Education: The

Dutch Case’. Paper prepared for the 3rd ALFA_BRACARA international conference

on 'Funding and Allocation of Resources in Higher Education' held 8-10 February at

Universidad Nacional Autonoman do Mexico.

van Raan, R. T. H. (Ed.) 1988. Handbook of Quantitative Studies of Science and Technology.

Amsterdam: North Holland.

40

VSNU. 1996. Quality Assessment of Research: Chemistry, Past Performance and Future

Perspectives. Utrecht: VSNU (Association of Netherlands Universities).

Whittington, G. 1997. The 1996 Research Assessment Exercise. British Accounting Review,

Vol. 19 section??: pp. 181-97.

Zilahy, P. and I. Láng. 1997. The Evaluation of Research Institutes in Hungary. In M. S.

Frankel and J. Cave (Eds) Evaluating Science and Scientist: An East-West Dialogue

on Research Evaluation in Post-Communist Europe. Budapest: Central European

University Press.

41

Table 1 Research Evaluation Practices in Different Countries

No Country University Research Evaluation forAllocating Core Research Funds

Other National Evaluationof University Research

1 UK Ex-post informed peer review (RAE)

2 Hong Kong Ex-post informed peer review (RAE)

3 Australia Ex-post quantitative evaluation(RQPC)

4 Poland Peer review + ex-post informed peerevaluation

5 Slovak Republic Student numbers corrected by ex-postinformed peer review

6 Denmark Education size/Performance based Research field evaluation

7 Finland Education size/Ex-post quantitativeevaluation

Research field evaluation

8 New Zealand Education size*

9 Germany Education size

10 Italy Education size

11 Norway Education size Research field evaluation

12 Sweden Education size Research field evaluation

13 Hungary Education size

14 The Netherlands Education size VSNU research evaluation

* It has been proposed that from 2000 a portion of the core research funding will be allocated based on an ex-ante evaluation using peer review.

42

Table 2: Advantages and Disadvantages of Alternative Approaches to University Research FundingAdvantages Disadvantages

Performance-based - ‘meritocratic’ in that it links resources to performance, may ‘reward’good research

- strong incentive to improve individual as well as institutionalperformance

- competition may ==> increased efficiency - ineffective researchidentified and cut

- encourages research to be properly completed and written up forwider dissemination

- provides public accountability for government funds invested inresearch

- encourages more explicit/coherent research strategy on part ofdepartment or institution

- provides mechanism for linking university research to governmentpolicy (e.g. to shift priorities)

- concentration of resources may enable best departments to competewith world leaders (e.g. in US)

- high cost and labour intensity (whether peer review or indicatorbased) for universities and/or evaluating agencies

- may ==> ‘homogeneisation’ of research and universities - i.e.decrease in diversity and experimentation

- may discourage more innovative and risky research- encourages ‘publication inflation’ (e.g. ‘salami publishing’) and

other ‘game playing’ (e.g. with indicators) i.e. ‘looking good’ ratherthan necessarily doing better

- may encourage traditional ‘academic’ research at expense of researchlinked to society’s needs

- tends to separate research from teaching ==> lower priority forteaching

- rewards past performance not current or future potential ==>reinforces research elite/status quo ==> over-concentration

- may lead to excessive government influence/‘interference’ inuniversity research

Educational size - low cost to administer- provides departments with ‘seed corn’ funds to invest in new

people/research areas- provides ‘space’ for long-term research and scholarship- encourages diversity in research- enables academic at any university (not just an elite few) to get

involved in research- encourages integration of teaching and research so can exploit

synergy- protects autonomy of institutions and individuals

- little direct incentive to improve research performance (whetherindividual or institutional) - may ==> stagnation

- may give excessive power to officials who distribute core fundingwithin institution

- little public accountability for funds (notionally) provided forresearch – may ==> ‘ivory tower’ research with no social or otherrelevance

- may reinforce public stereotype that some academics ‘lazy’- may be little or no correlation between student numbers and level of

research effort by department- distribution of resources/effort may bear little relationship to stated

government policy- spreading resources evenly but thinly may ==> unable to compete

with world-leading institutions

43

Figure 1

0

10

20

30

40

50

60

70

80

90

100

t0 t1 t2 t3 t4 t5 t6

Time

Costs,Benefits

BenefitsCosts

44

Figure 2

0

5

10

15

20

25

30

35

40

45

t0 t1 t2 t3 t4 t5 t6

Time

Costs,Benefits

BenefitsCosts