Evaluation of Research and Evaluation of Research and Innovation of University Academics Innovation of University Academics

Using Soft – Computing MethodologyUsing Soft – Computing Methodology

Faith-Michael UzokaFaith-Michael Uzoka

22

OutlineOutline

•IntroductionIntroduction

•Objectives/SignificanceObjectives/Significance

•LiteratureLiterature

•MethodologyMethodology

•Results and DiscussionResults and Discussion

•ConclusionConclusion

33

IntroductionIntroduction• There is a strong relationship There is a strong relationship

between academic research output between academic research output and the overall development of a and the overall development of a nation (Lootsma and Bots 1999nation (Lootsma and Bots 1999

• A number of stakeholders have A number of stakeholders have invested huge sums of money in the invested huge sums of money in the development of research in development of research in universitiesuniversities

• Universities place a very high Universities place a very high premium on research compared to premium on research compared to teaching and service (Uzoka 2003)teaching and service (Uzoka 2003)

44

IntroductionIntroduction• Research Measurement is a complex, Research Measurement is a complex,

multicriteria and multidimensional, multicriteria and multidimensional, context specific activity (context specific activity (Murphy 1994Murphy 1994 ) )

• Universities are different from other Universities are different from other corporate environments, so conventional corporate environments, so conventional evaluation models do not fit.evaluation models do not fit.

• Need to develop a model that attempts to Need to develop a model that attempts to capture research variables in a global capture research variables in a global perspective taking cognisance of fuzzy perspective taking cognisance of fuzzy nature of parameter assessment.nature of parameter assessment.

55

ObjectiveObjective• Develop a fuzzy-enhanced analytic hierarchy Develop a fuzzy-enhanced analytic hierarchy

process (FEAHP) model for the evaluation of process (FEAHP) model for the evaluation of university academic staff research productivityuniversity academic staff research productivity

SignificanceSignificance• Takes cognizance of the multi-criteria, multi-Takes cognizance of the multi-criteria, multi-

situation, and multi-dimensional nature of situation, and multi-dimensional nature of research output evaluationresearch output evaluation

• Utilizes confidence measures in obtaining the Utilizes confidence measures in obtaining the ratings of decision experts on various research ratings of decision experts on various research evaluation criteria evaluation criteria

66

LiteratureLiterature

• Universities adopt corporate evaluation Universities adopt corporate evaluation models [e.g. research output patents, models [e.g. research output patents, performance-based funding, performance-based funding, benchmarking, best practices, and total benchmarking, best practices, and total quality management]quality management] ( (Shafer and Caote Shafer and Caote 19921992 ) )

• Benchmarking seems to be most Benchmarking seems to be most prominent (prominent (Alt 2002Alt 2002 ). It is relevant in ). It is relevant in inter-institutional comparisonsinter-institutional comparisons

• Baughman and Goldman (1999) found a Baughman and Goldman (1999) found a strong positive relationship between strong positive relationship between scholarship and institutional rankingscholarship and institutional ranking

77

LiteratureLiterature• Performance indicators that fail to take Performance indicators that fail to take

account of input differences across account of input differences across institutions are inappropriateinstitutions are inappropriate ( (Johnes Johnes 1992, Neely 1992, Neely et al.et al. 2005) 2005)

• Various indices have been proposed in the Various indices have been proposed in the past:past:– Number of publications and presentations (E.g. Number of publications and presentations (E.g.

Lopes and Lanzer 2002Lopes and Lanzer 2002))– Publications including guidance and Publications including guidance and

supervision (E.g. Uzoka and Akinyokun 2005)supervision (E.g. Uzoka and Akinyokun 2005)– Patents and spin off companies (E.g. Patents and spin off companies (E.g. Wallmark Wallmark

et alet al. 1998. 1998 ) )– Citation index (E.g. Citation index (E.g. Hirsch 2005) Hirsch 2005)

88

LiteratureLiterature

• Popular Tools include:Popular Tools include:– Multi criteria analysis (E.g. Multi criteria analysis (E.g. Islam and Islam and

Rasad 2005Rasad 2005 ) )– Fuzzy Technology (Fuzzy Technology (Akinyokun 2002, Royes Akinyokun 2002, Royes

et alet al. 2003. 2003 ) )– Knowledge Based Methodologies (E.g. Knowledge Based Methodologies (E.g.

Vinsonhaler Vinsonhaler et alet al. 1996, Uzoka and . 1996, Uzoka and Akinyokun 2005)Akinyokun 2005)

• Dearth of Dearth of research on useful assessment research on useful assessment criteria and models for research of university criteria and models for research of university academic staff especially in the twenty first academic staff especially in the twenty first century.century.

99

Derivation of variables

Derivation of variables

Figure 1: FAHP Conceptual Framework

Determination of research productivity

Determination of research productivity

Decision makers’ rating of research

Aggregation of ratings using FEAHP

Decision experts

Domain experts

Linguistic PWC of variables

Aggregation of weights

Obtaining of research evaluation model

MethodologyMethodology

1010

Figure 2: Hierarchy of Criteria

1111

Age (x years)X < 30 30 ≤ x <

4040 ≤ x < 50 50 ≤ x <

60x ≥ 60

2.4% 23.8% 33.3% 33.3%% 7.1%

Experience ( y years)

y < 5 5 ≤ y < 10 10 ≤ y < 15 15 ≤ y < 20

y ≥ 20

16.7% 31.0% 21.4% 14.3% 16.7%Position Head of

DeptDean of

FacultyMember, Eva

Comm.HR

Manager

Others

23.8% 7.1% 31.0% 2.4% 35.7Overall rating

confidence (0≤ z ≤ 10)

z<2 2 ≤ z < 4 4 ≤ z < 6 6 ≤ z < 8 8 ≤ z <10

0% 4.8% 33.3% 30.9% 31.0%Avg.

standardized rating confidence (0≤ v ≤ 10)

v<2 2 ≤ v < 4 4 ≤ v < 6 6 ≤ v < 8 8 ≤ v ≤10

0% 0% 45.24% 54.76% 0%

Case Number

4037343128252219161310741

Va

lue

12

10

8

6

4

2

0

YEXP

OVERCONF

AVGCONF

Table 1: Distribution of ExpertsFigure 3: Experts’ Years of Experience, Average Confidence and Overall Confidence correlation between the years of experience and overall confidence = .764

1212

Aggregation of WeightsAggregation of Weights• Step 1: Standardization of Experts’ Step 1: Standardization of Experts’

Rating ConfidenceRating Confidence

• Step 2: Adjustment of Fuzzy Values Step 2: Adjustment of Fuzzy Values by Standardized Rating Confidenceby Standardized Rating Confidence

• Step3: Deriving Aggregate Fuzzy Step3: Deriving Aggregate Fuzzy WeightsWeights

• Step 4: DefuzzificationStep 4: Defuzzification

1313

Measure of Group ConsensusMeasure of Group Consensus• A high degree of group consensus increases the A high degree of group consensus increases the

validity of the evaluation (Chen and Liu 2005). validity of the evaluation (Chen and Liu 2005). • The group consensus measure presented in The group consensus measure presented in

(Shirland (Shirland et alet al. 2003) was adopted, but modified . 2003) was adopted, but modified to eliminate bias that results from the use of to eliminate bias that results from the use of sample, rather than population parameters. sample, rather than population parameters.

• For any expert For any expert i i to be included in the analysis, his to be included in the analysis, his overall PWC deviation overall PWC deviation δδii from the group from the group consensus consensus δ δ ≤ ≤ δδii

maxmax which evaluates to which evaluates to

)1p)(1p)(3/1()p(maxi

p is the number of pairwise comparisons and n is the number of DEs.

for i=1,2,…, n;

1414

Consensus ResultsConsensus Results

• δ = 6.575δ = 6.575 • δδii

max max = = 71.302 71.302• Percentage Percentage

consensus =81.63consensus =81.63

i iExpert δδii Expert δδii

1 5.47521 22 5.659618

2 7.251734 23 5.602077

3 7.246353 24 6.839391

4 6.816796 25 7.998329

5 5.779035 26 7.275522

6 7.054126 27 6.902183

7 6.349235 28 7.926907

8 4.524326 29 7.910003

9 6.424866 30 3.431349

10 5.123588 31 4.055824

11 5.437166 32 6.780548

12 7.056387 33 6.264001

13 6.972187 34 6.524648

14 7.215447 35 6.941077

15 6.608031 36 6.338408

16 6.920122 37 5.961696

17 7.037552 38 6.710627

18 6.376808 39 6.391779

19 6.990181 40 6.750813

20 6.132259 41 6.54038

21 5.807774 42 6.574896

Table 3: Consensus Values

1515

Obtaining the Final Obtaining the Final Evaluation ModelEvaluation Model • After defuzzification, the principles of AHP After defuzzification, the principles of AHP

were applied in deriving the final evaluation were applied in deriving the final evaluation modelmodel

• Let P(Let P(i,ji,j) be a pairwise comparison of two ) be a pairwise comparison of two elements elements i i and and jj;;

• Where {Where {i,ji,j} Є nk (nk = node k of the } Є nk (nk = node k of the FEAHP tree).FEAHP tree).

• The larger the value of P(The larger the value of P(i,ji,j), the more i is ), the more i is preferred topreferred to j j in the priority rating. The in the priority rating. The following rules govern the entries in the PWC.following rules govern the entries in the PWC.

• Rule 1Rule 1: P(: P(j,ij,i) = [p() = [p(i,ji,j)])]-1-1

• Rule 2Rule 2: If element : If element ii is judged to be of equal is judged to be of equal importance with element importance with element jj , then , then

• P(P(i,ji,j) = P() = P(j,ij,i) = 1; in particular, P() = 1; in particular, P(i,ii,i) = 1 ) = 1 for all for all ii..

• The Pairwise comparison (PWC) matrices were The Pairwise comparison (PWC) matrices were obtained by entering the individual ratings of obtained by entering the individual ratings of the experts into the Expert Choice 11 the experts into the Expert Choice 11

1616

RP PT AU QP SF PP

RP 1 1.9398 1.08412 1.85532 1.51493 2.00763

PT 1 3.34427 1.22032 1.63010 1.53252

AU 1 1.87424 1.19098 1.34899

QP 1 2.72616 2.29324

SF 1 1.67675

PP 1

Note: For shaded values, the top variable is more important than left hand side pair

Table 4: Research Output Evaluation PWC

1717

Synthesis - Goal (Level 1)Synthesis - Goal (Level 1)

1818

Synthesis - Publication Type Synthesis - Publication Type (Level 2)(Level 2)

1919

Synthesis - Reputation of Publisher Synthesis - Reputation of Publisher (Level 2)(Level 2)

Synthesis - Authorship (Level 2)Synthesis - Authorship (Level 2)

2020

Synthesis - Place of Publication (Level Synthesis - Place of Publication (Level 2)2)

Final Evaluation Factor IndexFinal Evaluation Factor IndexFEFIi = 0.133(0.425IB + 0.484PB + 0.091OT) + 0.258(0.201JA + 0.075CP + 0.053TR + 0.051OP + 0.051MO + 0.156PB + 0.116 BC+ 0.048PM + 0.054IO + 0.043SW + 0.047CM + 0.034SP + 0.033GS + 0.037RP) + 0.114(0.425SA + 0.326DA + 0.154TA + 0.095MA) + 0.250QP + 0.146SF + 0.099(0.200LP + 0.302RP + 0.498IP)

Research Evaluation Model

2121

Rating Rating • Authorship (AU), reputation of publisher Authorship (AU), reputation of publisher

(RP), publication type (PT), and place of (RP), publication type (PT), and place of publication (PP) are fairly structuredpublication (PP) are fairly structured

• Quality (QP) and specialization/ relevance Quality (QP) and specialization/ relevance (SF) of research are unstructured. (SF) of research are unstructured.

• This study proposes an ordinal rating of the This study proposes an ordinal rating of the unstructured attributes through a likert unstructured attributes through a likert type rating, which is normalized in order to type rating, which is normalized in order to fit the normalized structure of other ratings. fit the normalized structure of other ratings.

• The ratings and normalized equivalents are The ratings and normalized equivalents are as follows: as follows: excellent (EX) – 5 (0.333); very good (VG) – excellent (EX) – 5 (0.333); very good (VG) – 4 (0.267); good (GD) – 3 (0.200), fair (FR) – 2 4 (0.267); good (GD) – 3 (0.200), fair (FR) – 2 (0.133), poor (PR) – 1 (0.067). (0.133), poor (PR) – 1 (0.067).

2222

Numerical ExampleNumerical Example

• Model was tested using an expert’s Model was tested using an expert’s evaluation of thirty four publications evaluation of thirty four publications of three academic staff, two from the of three academic staff, two from the same department, and another from a same department, and another from a different department. different department.

• Figure 10 Figure 10 presents the AHP tree for presents the AHP tree for the evaluation of the research output the evaluation of the research output of the three academic staff. of the three academic staff.

• RO1, RO2,…, ROn represents various RO1, RO2,…, ROn represents various publications of academic staff S1, S2, publications of academic staff S1, S2, S3 respectively. S3 respectively.

• The raw evaluations obtained for the The raw evaluations obtained for the three academic staff are presented in three academic staff are presented in Table 5Table 5. .

2323

ResultsResults

• Table 6 Table 6 presents the final evaluation presents the final evaluation of research output after the of research output after the application of the fuzzy enhanced application of the fuzzy enhanced AHP systemAHP system

• A multidimensional analysis of A multidimensional analysis of research output is presented in research output is presented in Table 7Table 7

2424

DiscussionDiscussion

• publication typepublication type (PT) and (PT) and quality of quality of publicationpublication (QP) are considered the (QP) are considered the most important criteria in the most important criteria in the evaluation processevaluation process

• place of publication place of publication (PP) is considered (PP) is considered the least important factor.the least important factor.

• journal publications are considered journal publications are considered the most valued publication type the most valued publication type (20.1%), while guidance/supervision is (20.1%), while guidance/supervision is considered least valued (3.3%)considered least valued (3.3%)

2525

DiscussionDiscussion• Ordinal ranking of some publication types Ordinal ranking of some publication types

as follows: JA>PB>BC>CP>GC.as follows: JA>PB>BC>CP>GC.

• Shows slight variation from the results Shows slight variation from the results obtained in (Lootsma and Bots 1999), obtained in (Lootsma and Bots 1999), which ranked JA>PB>GC>BC>CP.which ranked JA>PB>GC>BC>CP.

• Reason:Reason:– books seem wider in circulation and usage books seem wider in circulation and usage

than postgraduate dissertations especially in than postgraduate dissertations especially in recent times. recent times.

– Also there is increased emphasis on teaching Also there is increased emphasis on teaching (Obasi 2008), which further increases the (Obasi 2008), which further increases the importance of books and book contributions in importance of books and book contributions in the evaluation process.the evaluation process.

2626

Conclusion and ImplicationsConclusion and Implications• This study provides a framework for This study provides a framework for

quantitative analysisquantitative analysis of university academic of university academic staff research productivity. staff research productivity.

• The model would assist international ranking The model would assist international ranking agencies in reconsidering what constitutes agencies in reconsidering what constitutes the critical elements of institutional ranking. the critical elements of institutional ranking.

• A computer program could be developed on A computer program could be developed on the basis of this model to assist in peer the basis of this model to assist in peer evaluations by academic heads and evaluations by academic heads and colleagues.colleagues.

• The model discussed in this study could be The model discussed in this study could be integrated with institutional databases of integrated with institutional databases of faculty research profiles for purposes of faculty research profiles for purposes of inter-faculty and inter institutional inter-faculty and inter institutional comparisonscomparisons

2727

EndEnd