Effects of GRIs Evaluation System on the Evaluation Result Acceptability and Performance : Focusing...

Effects of GRIs Evaluation System on the Evaluation Result Acceptability and

Performance : Focusing on Economic and Human Society Research

Council in Korea

Soon Cheon ByeonByung Yong Hwang

KISTEP(Korea Institute of Science and Technology Evaluation and

Planning)

Seoul, South Korea

AEA 2009Orlando, Florida

2

Contents

I. IntroductionII. Backgrounds 2.1. Performance Evaluation of GRIs 2.2. Characteristics of GRIs and AcceptabilityIII. Research Model and Hypotheses 3.1. Research Model 3.2. General Model 3.3. Variables and Hypotheses IV. Methodology 4.1. Targets for Questionnaire 4.2. Sampling 4.3. Analysis Method V. Results 5.1. Analysis on Validity and Reliability 5.2. Results of Hypothesis Verification VI. Conclusions and Suggestions

3

I. Introduction

On 2005, the Economic Society Research Council and the Human Society Research Council were integrated into the Economic and Human Society Research Council(23 GRIs)

On 2008, Basic/Industrial/Public Technology Research Councils were reorganized as KRCF(Korea Research Council of Fundamental S&T, 13 GRIs) and ISTK(Korea Research Council for Industrial S&T, 13 GRIs)

23 GRIs in EHS Research Council are classified as follows according to their research areas

Economic Policy(5 Org.)

Resources and Infrastructure(6

Org.)

Human Resources(7 Org.)

Public Policy(5 Org.)

․ Korea Institute for Intl' Economic Policy ․ Korea Institute for Industrial Economics and Trade ․ Korea Development Institute․ Rural Economic Institute ․ Korea Institute of Public Finance

․ Korea Research Institute for Human Settlements․ Korea Energy Economics Institute ․ Korea Information Society Development Institute ․ Korea Transport Institute ․ Maritime Institute ․ Korea Environment Institute

․ Korean Educational Development Institute․Korea Institute for Curriculum & Evaluation․ Korea Labor Institute ․ Korea Institute for Health & Social Affairs․ Korean Women's Development Institute ․ Korea Research Institute for Vocational Education & Training․ Korea Institute for Youth Development

․ Science & Technology Policy Institute․ Korea Institute for National Unification․ Korean Legislation Research Institute․ Korea Institute of Public Administration․ Korea Institute of Criminology

1.1. Current Status of Research Councils and GRIs

4

I. Introduction

Governing Principles of the Research Council System Competition in the research and development securing of independency and autonomy of GRIs and their researches avoiding of redundant and overlapped investment

Purpose of the 23 GRIs under Economic and Human Society Research Council

to satisfy government's demands on policies which address social issues arising from Korea's fast economic development and science and technology advancement during 1970s and 80s, diversified international competition, and social conflicts and confrontation.

1.2. Purpose of Research Council and GRIs Purpose of the establishment of the EHS Research Council

to support national R&D policy to contribute toward the development of knowledge-based industry by support and promotion of research institutes in the area of Economic

and Human Society and by systematic management of GRIs

5

Functions & Roles

Details

Research planning by

areas

․ Establishing research direction conforming to national R&D policy․ Developing national agenda according to national vision․ Reviewing projects by the Planning Evaluation Committee and connecting to

evaluation

Planning development directions of

GRIs

․ Planning long-term development direction to expand GRIs' functions as main think-tank with world class policy research capability ․ Establishing basic guidelines which present desirable directions and visions

and instruct & manage GRIs

Coordinating & adjusting GRIs'

functions

․ Promoting specialization of GRIs through reasonable coordination between functions․ Establishing identity of government-invested research institutes․ Making GRIs' research function professional and specialized

Evaluating GRIs

․ Encouraging autonomous improvement for raising research productivity․ Facilitating active and continuous management innovation․ Spreading out research results and project performances․ Encouraging autonomy and accountable management of GRIs

Promoting joint researches

․ Maintaining close cooperation between the Research Council and GRIs․ Establishing joint networks among industry, academia, and research

institutes․ Conducting major national policy tasks & creating synergy effects through

joint researches

1.3. Functions and Roles of EHS Research Council Functions and Roles of the EHS Research Council

I. Introduction

6

I. Introduction1.4. Achievements and Problems of EHS Council

Major Achievements in … Establishment of autonomous and accountability management of

GRIs Facilitation of Industry-Academy-Research Institute joint researches Enhancement of key research capabilities of GRIs Increased reasonable use of budget through avoiding redundant and

overlapped projects

Still several Problems in … Establishment of identity of Research Council (too small joint

research fund) Coordination of GRIs functions through evaluation (low

implementation of evaluation result) Acquisition and distribution of research resources for GRIs (limited

activity in acquisition process) Development strategies of GRIs (including low activity in the

maintenance of high quality HR)

7

I. Introduction

1.5. Necessity of Research

Improvement of the functions and roles of the Research Council & GRIs Evaluation systemFocusing on improving Performance of GRIs

▪ Improvement of GRIs performance under the Research Council▪ Improvement of Pubic trust in government through reporting performances

National concerns on R&D performance

▪ Recent changes in the environment surrounding the Research Council and GRIs ▪ Reorganization of National Science and Technology Council (NSTC) by the new government and Selection of the policy to keep the existing Research Council system▪ Criticism and Demand for improvements on the roles

and management systems of the Research Council

8

I. Introduction

1.6. Orienting Questions Questions on the Functions and Roles of the Research

Council Have the functions and roles of the Economic and Human Research

Council been well established and institutionalized? If any, what are the problems of the Research Council system? What can be done to stabilize the system and enhance the

performance of GRIs?

Survey result on the satisfaction on the EHS Research CouncilTargets Marks

Government Agencies 70.1

Industry-Academy-Research Institute users 73.5

Members of Research Institutes 64.8

Average Point (of 100) 69.5

9

I. Introduction

1.7. Orienting Questions Questions on the GRIs Evaluation System of the Research

Council Dissatisfaction with GRIs evaluation system within the functions and

roles of the Research Council Has the evaluation system been fulfilled well to maximize research

performances? GRIs’ dissatisfaction than acceptance The satisfactions point on GRIs evaluation: 49.4 points What area should be improved to motivate GRIs and maximize

performances?

10

I. Introduction

1.8. Objectives of Research Measuring the relationship among GRIs evaluation system,

Evaluation result acceptability and GRIs performances Analysis on the status of GRIs evaluation system and the recognition &

acceptance attitude on GRIs evaluation system by institutes with high performances

Suggestion of enhancement directions of GRIs evaluation system to improve the performances of GRIs

Approach to problems of the Research Council and GRIs through integrated view

11

2.1. Performance Evaluation of GRIs Performance Evaluation by Government

a process which defines its mission and desirable outcomes, establishes standard on performance, connects performance to budget, reports results, and makes managing level officials take the accountability for the results. (Andris Vinelli, 2000)

- Act on R&D Performance Evaluation and Management (Dec. 2005) - National Evaluation System (NES)

Researches on the Performance Evaluation on GRIs in Korea studies focusing on the system design for GRIs evaluation studies suggesting the balanced scorecard (BSC) approach for GRIs

performance evaluation a possibility of adoption the intellectual capital model to performance

evaluation needs to encourage GRIs to raise their performance through

stabilizing the functions and roles of Research Councils Not enough study on the factors consisting of functions and roles of the

Research Council which is the basis of such R&D activities Insufficient efforts to analyze on how the functions and roles of the

Research Council affects the performance of GRIs

II. Research Backgrounds

12

2.1. Performance Evaluation of GRIs For non-profit organizations and public organizations,

performance related to their missions are regarded more important than financial performance. (Brinckerhoff, Peter C., 1994)

Recently, when it comes to GRIs' performance, more interests are placed in the operation of an organization as well as tangible results.

Models identifying performance of an organization focusing on the effectiveness of the organization of a research institute

These Models relate innovation, creativeness and excellence of science technology to structure, culture, characteristics, activities and processes of a research organization.

These Models made large contributions in that they identified factors affecting performances of research institutes and suggested such performances can be analyzed through the analysis on those factors. (Jordan and Streit, 2000)

However, the Models failed to include comprehensive performance related to mission and goals of the research organization, relationship between the organization and outside entities, and learning aspect. (Jang Jae Lee and others, 2003)


13

2.1. Performance Evaluation of GRIs

Models adopting BSC(balanced scorecard) to performance evaluation of the research institute

Min Hyeong Lee (2001, 2005) was the first to employ BSC approach into performance evaluation of GRIs.

Jang Jae Lee and others (2003) utilized general framework suggested by Kaplan & Norton (1996) to analyze performance based on four aspects including finance, customer, internal process and innovation and learning.

cases of May Institute, a non-profit research institute in Massachusetts, the U.S. Department of Transportation, Sandia National Lab, and museums

In this study, considering the characteristics and missions of GRIs, the GRIs’ performances are classified as follows based on the BSC, which includes comprehensive performances, relations and learning aspect.


14

2.1. Performance Evaluation of GRIs BSC(balanced scorecard)-based Performance Evaluation for

GRIs


BSC-basedPerformance

Details

Financial Performance

․ Appropriateness of funding (record of project winning, income from royalty)․ Soundness of money flow (labor cost, accumulation ratio of retirement reserve, excess or deficiency of operating costs, soundness of expenses of other income)․ Financial soundness (financial structure, amount of net income)

Customer Performance

․Improvement of project performance (No. of patent & paper, research reports, record of commercialization, support for researcher's establishment, promotion of ventures)․ Enhancement of customer satisfaction (customer evaluation & satisfaction with performance, institute's reputation)

Internal Process

Performance

․ Improvement of strategy establishment process (developing missions and strategic goals, level of implementation management of strategic goals)․ Improvement of project management process (appropriateness of project portfolio management and comprehensive project management)․ Improvement of budget management process (reasonableness of budget management process, internal control system and cost calculation system)

Learning & Growth

Performance

․ Growth of human resources (appropriateness of professional human resources management, soundness of composition of human resources, reasonableness of personal performance evaluation system, degree of education & training, turnover rate)․ Growth of org. culture (org. structure, employees’ satisfaction, cooperative labor- management relationship)

15

2.2. Characteristics of GRIs and Acceptability

Characteristics of GRIs should be considered to determine the effect of the functions and roles of Council on GRIs performances.

Research area, budget size, established year

The evaluation result acceptability could be considered to determine the effect of evaluation system on the GRIs performances by focusing on variables related to the responding attitude to the evaluation result.

Will to reflect evaluation recommendations, will to reflect negative evaluation result


16

3.1. Research Model Previous studies related to the operation of Research Council

insist that the functions and roles of the Council have significant impact on the performance of GRIs.

But no discussion on which factors of the Council's functions and roles have affected what kind of GRIs' performance

This study will examine the factors which comprise functions and roles of the Research Council first.

The effect of acceptability of evaluation result on the GRIs performances will be investigated.

This study will explore desirable future directions in the functions and roles of the Research Council consistent with the objectives of GRIs, by investigating the specific effect of the functions and roles of the Research Council on the performances of GRIs.

III. Model and Hypothesis

17

3.2. General Model


Functions and roles of Research Council

Research planning by areas

Planning GRIs’ Development Direction

Coordinating and adjusting GRIs’ functions

Evaluation on GRIs

GRIs performance

Financial performance

Customer performance

Internal process performance

Learning & growth performance

Characteristics of GRIs

Research areas

Size of GRIs

Established Year

Evaluation

Result

Acceptability

Promoting joint research

H1

H2

H3 H4

H5

18

3.3.1. Variables and Hypothesis(1)


Independent variables

control factors

Dependent variables

GRIs’ Characteristics

GRIs’ Performances

• Research Areas• Size• Established Year

• Financial Performance• Customer Performance• Internal Process Performance• Learning & Growth Performance

Functions and Roles of EHS Research Council

• Research Planning• Planning GRIs’ Development Direction• Coordination and Adjusting GRIs’ Functions• Evaluation on GRIs • Promoting Joint Researches

The objective of this study is to discuss the detail effects of the functions and roles of the Research Council on the specific performance of GRIs.

control factors are added to identify the effects of GRIs' characteristics including research areas, size and established year of GRIs.

19



Hypothesis 1. The functions and roles of Research Council will have significant impact on the performance of GRIs.

Hypothesis 2. The performance of GRIs will be significantly different according to the characteristics of GRIs and the functions and roles of the Research Council.

Hypothesis 2-1. The performance of GRIs will be significantly different according to the research area of GRIs and the functions and roles of Research Council.

Hypothesis 2-2. The performance of GRIs will be significantly different according to the size of GRIs and the functions and roles of Research Council.

Hypothesis 2-3. The performance of GRIs will be significantly different according to the established year of GRIs and the functions and roles of Research Council.

20



The other objective of this study is to discuss the detail effects of the evaluation system on the specific performance of GRIs.

Mediating variables are added to identify the effects of evaluation system on GRIs performances, such as ‘will to reflect recommendation’ and ‘will to reflect negative evaluation results’.

H3 H4

H5

GRIs evaluationsystem

GRIs evaluationsystem

Evaluation Result Acceptability

Evaluation Result Acceptability GRIs performanceGRIs performance

Mediating variablesIndependent variables Dependent variables

21



Hypothesis 3. The quality of GRIs evaluation system will have significant effect on the acceptability of GRIs evaluation results.

Hypothesis 4. The acceptability of evaluation results will have significant effect on the GRIs performances.

Hypothesis 5. The quality of GRIs evaluation system will have significant effect on the GRIs performances.

22

4.1. Methodology and Targets for Questionnaire

IV. Methodology

Empirical methodology A research model was established by collection and analysis on the

related laws and literatures, and then the hypotheses were verified by an empirical methodology which verifies hypotheses through statistical analysis based on a survey result.

Process The survey was carried out on researchers in 23 GRIs under the Economic

and Human Society Research Council. The samples consist of 551 persons including vice president,

administrative (planning) leader, research coordination (planning and budget) team leader, and president of labor unions.

Via e-mail Analyzed by using SPSS for Windows 15.0

23

4.2. Sampling

IV. Methodology

To survey for the empirical methodology, a qualitative research was performed on a group including staff, union members, researcher, administrative employee who are directly related to the functions and roles of the Research Council and the performance of GRIs.

the sample group was smaller than that in general studies, but had higher level of understanding for all questions.

Although the sample was small, it can be considered as appropriate and reasonable given that it was the qualitative research.

24

IV. Methodology Composition of samples

Characteristics Contents No. of samples Ratio (%)

Age

Under 30’s

40’s

50’s

60’s

193

252

95

35.0

45.7

17.2

Work type

researcher

management work

office work

others

309

191

31

12

56.0

34.7

5.6

2.2

Position

researcher

senior (research associate)

chief (research fellow, senior research fellow)

others

127

109

214

83

23.0

19.8

38.9

15.0

Duty

Person in charge

Position in charge

Others

258

198

76

46.8

35.9

13.8

Education career

bachelor

master

doctor

184

130

222

33.2

23.4

40.2

Labor unionmember

non-member

148

382

26.7

68.8

25

IV. Methodology Composition of samples

Characteristics Contents No. of samples Ratio (%)

Period of employment

1-5 years6-10years

11-15years16-20years21-25years

More than 25years

145

127

107

108

37

18

26.3

23.0

19.4

19.6

6.7

3.3

Research area

Economic policy

Resource & infrastructure

Human resources

Public policies

116

158

190

87

21.0

28.7

34.5

15.8

Size of GRIs

(budget)

Less than 10 bil.

10-15 bil.

15-20 bil.

More tha 20 bil.

130

69

235

117

23.6

12.5

42.7

21.2

Size of GRIs

(personnel)

Less than 50 persons

50-100 persons

100-150 persons

More than 150 persons

82

144

265

60

14.9

26.1

48.0

11

Experience of GRIs

(Based on the established year)

1970s

1980s

1990s

146

200

205

26.5

36.3

37.2

26

4.3. Analysis method

IV. Methodology

Verification Methodology

Sample Characteristi

cs

․ Technical analysis on samples․ Separation of factors of the functions and roles of the Council․ GRIs' performance type according to functions & roles of the Council․ Verification of reliability against factors of functions & roles of the Council․ Verification of reliability against GRIs' performance type according to factors of functions & roles of the Council

․ Frequency analysis․ Factor analysis

․ Factor analysis

․ Reliability analysis

․ Reliability analysis

Hypotheses

․ Effects of functions and roles of the Council on GRIs' performance (Hypothesis 1)․ Difference in GRIs' performances according to the characteristics of the GRIs and the functions & roles of Council (Hypothesis 2) ․ Effects of acceptability on GRIs‘ performance (Hypothesis 3,4,5)

․ Canonical Correlation analysis․ two way Anova

․ Structural equation model analysis

27

It is generally accepted that a measured value is considered as reliable when the Cronbach’s α value exceeds 0.6.

All the α values satisfy such requirement according to the following analyses.

enabling the analysis on all the factors in a concerned item with a single measure without undermining reliability.

5.1. Analysis on validity and reliability

V. Results

28

5.2. Results of Hypothesis Verification(1)

V. Results

Hypotheses Acceptance

<Hypothesis 1> The functions and roles of Research Council will have significant impact on the performance of GRIs.

Positive

<Hypothesis 2>The performance of GRIs will be significantly different according to the characteristics of GRIs and the functions and roles of the Research Council.

Partly positive

Hypothesis 2-1: The performance of GRIs will be significantly different according to the research area of GRIs and the functions and roles of Research Council.

Positive

Hypothesis 2-2: The performance of GRIs will be significantly different according to the size of GRIs and the functions and roles of Research Council.

Positive

Hypothesis 2-3:The performance of GRIs will be significantly different according to the established year of GRIs and the functions and roles of Research Council.

Negative

29

5.3. Analysis Results of Structural Equation Model(2)

V. Results

* All path coefficient values have significance at the level of p<.001

30

5.4. Cause-effect analysis among factors(2)

V. Results

Proposed pathPath

coefficientStandard deviation

t-value P-value hypotheses

GRIs Evaluation System → acceptability of evaluation results

0.826 0.047 16.646 .000 Hypo 3 adopted

Acceptability of evaluation results → GRIs’ performances

0.739 0.057 13.631 .000 Hypo 4 adopted

GRIs Evaluation System → GRIs’ performances

0.592 0.059 13.778 .000 Hypo 5 adopted

31

5.5. Analysis Results

V. Results

The GRIs Evaluation System has significant and positive influence on the acceptability of evaluation results and GRIs’ performances. Furthermore, the acceptability of evaluation results has relations with GRIs’ performances

Direct effect: 0.592, Indirect effect: 0.610

All the factors consisting of the GRIs Evaluation System serves as important deciding factors

‘Sufficient interactions’, ‘Fairness of evaluation criteria’, and ‘Systematic evaluation items and indicators’ receive more weights

‘Confidence of participants’ and ‘Appropriate incentives’ have relatively less weights

This might be interpreted as the importance of developing the evaluation criteria and indicators which would most reflect the characteristics of GRIs in the operation of GRIs Evaluation System, through effective interactions

This might be interpreted as the importance of developing the evaluation criteria and indicators which would most reflect the characteristics of GRIs in the operation of GRIs Evaluation System, through effective interactions

32

5.5. Analysis Results

V. Results

The factors for acceptability of evaluation results are also crucial determining factors

All the factors consisting of GRIs’ performances serve as important determining factors

‘Internal process performances’ takes up more importance than ‘Customer performances

It means that the evaluation results greatly influence on the improvement of GRIs’ management process, while they have less influence on the research performance or customer satisfaction

It means that the evaluation results greatly influence on the improvement of GRIs’ management process, while they have less influence on the research performance or customer satisfaction

33

V. Conclusions and Suggestions

First, this study conducts empirical analysis on the effects of functions and roles of the Economic and Human Society Research Council on the performance of GRIs.

As result of survey on 551 employees from 23 GRIs, functions and roles of the Economic and Human Society Research Council have significant impacts on the performance of GRIs.

In particular, 'research planning by each research area' has more impacts on the GRIs' performance, while functions and roles of the Council have the most impacts on GRIs' financial performance.

Furthermore, 'research area‘ and 'size‘ have control effects on customer performance, internal process performance and learning & growth performance, and internal process performance respectively.

In conclusion, the Research Council needs to play different functions and roles considering research area, size and work type of GRIs.

34


Second, the GRIs evaluation system has significant influence on GRIs performances, and the acceptability of evaluation results have a relation with GRIs performance.

Overall weakness of the GRIs Evaluation System results in the low acceptability, and also brings limitation to the active use of evaluation results.

In conclusion, the Research Council needs to improve the evaluation system and develop methods to enhance the acceptability of the evaluation results.

35


Based on the above results, the following development measures and improvements are suggested for the Research Council.

First, the Economic and Human Society Research Council should endeavor to establish and develop its identity. It should expand its functions to include 'planning on government R&Ds' through regular monitoring of current R&Ds, as well as planning and establishment of new government R&Ds.

Second, the Council's function of obtaining and distributing research resources for GRIs should be expanded. Considering that the Council's functions and roles have the most impacts on 'financial performance' of GRIs, the most desirable role for the Council should be 'assisting GRIs with obtaining research budget and excellent human resources.

Third, the Council should strengthen its role as function coordinator among GRIs. Therefore planning ability and expansion of expertness of the Research Council are required. The participation and discussion process of each actor is crucial to improve

the functions and roles of the Research Council and to establish development plans

The participation and discussion process of each actor is crucial to improve the functions and roles of the Research Council and to establish development plans

36


Also, GRIs evaluation system should be focused on the motivation of GRIs and maximization its performances.

First, the objectives of evaluation (and results use) should be specific and reasonable so that they serve as specific guidelines for the GRIs evaluation participants.

Second, Evaluation indicators should be employed and the weights be placed by most reflecting on the characteristics of GRIs. Furthermore, the indicators should be agreed among the government, Research Council, and GRIs.

The achievement should be evaluated by absolute evaluation method based on their own goals. And the efficacy should be enhanced through the connection to internal performance management.

Third, the government and the Research Council as evaluators need to strengthen the collection and accumulation of relevant information and knowledge including current states and issues of GRIs. They also make further endeavors to form an evaluation committee in consideration of expertise, integrity, and accountability, and to provide the committee with prior knowledge about GRIs. Moreover, face to-face contacts among evaluation participants should be continuously expanded to consolidate interactions. The volume of incentive and its distribution should also be improved.

Fourth, Enhancement of acceptability: The establishment of monitoring system

37


Research limitations Limitation of generalization. Measuring GRIs performance by subjective satisfaction of individual

members of GRIs rather than based on quantitative data.

Future research Suggest the indicators of GRIs performances reflecting the characteristics

of public research institutes in EHS sciences. Faithfully measure the concepts of the acceptability and effectiveness of

the GRIs evaluation system.

38

Thank you for your attention

For more questions :

[email protected]

[email protected]

Effects of GRIs Evaluation System on the Evaluation Result Acceptability and Performance : Focusing...

Documents

Transcript of Effects of GRIs Evaluation System on the Evaluation Result Acceptability and Performance : Focusing...