Developing an Evaluation Approach for Software ... · FAHP method is used ... To avoid the...

Developing an Evaluation Approach for Software Trustworthiness Using Combination Weights and

TOPSIS

Li Shia,b*, Shan-lin Yanga,b,c , Kai Lia,c , Ben-gong Yua,b aSchool of Management, Hefei University of Technology, Hefei 230009, China

bInstitute of Computer Network System, Hefei University of Technology, Hefei 230009, China c Key Laboratory of Process Optimization and Intelligent Decision-Making, Ministry of Education,

Hefei 230009, China Email: {[email protected], [email protected], [email protected], [email protected]}

Abstract Software is everywhere. However, software is not always trustworthy. Confronting the demand of software trustworthiness evaluation, this paper proposes a novel software trustworthiness evaluation approach based on combination weights (CW) and improved TOPSIS methods. The determination of CW relies on exper gments and mathematical computation together. FAHP method is used to determine the importance of degree of criteria according

judgment. To avoid the subjective one-sidedness of weights, entropy weighting method is employed to calculate objective weights. Then, the paper adopts com-bination weighting method to obtain the CW of evaluation criterion. Next, the CW is applied to the improved TOPSIS method and the trustworthiness rating of software is calcu-lated. The result contributes to provide decision makers more decision information. Finally, the proposed approach is demonstrated with the evaluation of PLM software for an aircraft equipment manufacturer in China, which is fol-lowed by the sensitivity analysis and the results illustrate the robustness of the presented evaluation approach. Index Terms trustworthy software; FAHP; entropy weighting method; TOPSIS; evaluation model; PLM soft-ware

I. INTRODUCTION Modern society is irrevocably and virtually dependent

on information technology, and software systems or products in particular. Software has been a key technolo-gy in national security, government, finance, business, manufacturing, energy, and permeated services of all kinds. However, software is not always trustworthy. The failure of software can lead to serious consequences, some of which are extensive and even disastrous damage [1]. The problems of trustworthy software have become worldwide in scope, among which, how to evaluate Soft-ware Trustworthiness (ST) for providing decision makers more decision information is becoming a desiderated problem and challenger.

Up to now, much effort has been expended in methods for different aspects of trustworthy software [2,3,4,5,6,7,8,9,10], which primarily involve basic con-cepts, terminology, systems, research plan, measurement and software process trustworthiness evaluation, etc. Based on these studies, such criteria as functionality, co-

existence, reliability, safety, maintainability, and usability, etc., have been studied as major trustworthiness proper-ties that contribute to the trustworthiness of a software

t-standpoint that

ST is a holistic property encompassing a set of criteria has been an increasing consensus. Therefore, in this paper, software trustworthiness evaluation (STE) is regarded as a Multi-Criteria Decision Making (MCDM) problem that consists of both quantitative criteria and qualitative crite-ria.

The MCDM approach is suited to deal with two kinds of problems [11,12]. One is the classical MCDM prob-lems [13], among which the ratings and the weights of criteria are measured in crisp numbers. The other MCDM category is the Fuzzy Multi-Criteria Decision Making (FMCDM) problems [14], among which the ratings and the weights of criteria evaluated on imprecision, uncer-tainty and vagueness are usually expressed by linguistic variables and then set into fuzzy numbers [15].

The purpose of this paper is to formulate software trustworthiness evaluation as a FMCDM model, and then presents a novel approach based on FMCDM methods for evaluating ST. The evaluation results assist the decision makers to select a trustworthy software product from proposed alternatives. Among existing FMCDM methods, Fuzzy Analytic Hierarchy Process (FAHP) [16,17], en-tropy theory [18] and Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) [19] method have been widely used to deal with multiple criteria evaluation or selection problems in many fields [20,21,22,23,24,25,26,27,28,29,30,31,32]. However,

rarely used in the field of software trustworthiness evaluation. There-fore, different from the practical applications reported in the above literatures, this study adopts combination weighting method and improved TOPSIS together based on fuzzy set theory to evaluate the software trustworthi-ness.

The rest of this paper is structured as follows: Section II presents the evaluation framework. In Section III, the research methodology used in the proposed model is re-viewed. Section IV illustrates an application in the evalua-

532 JOURNAL OF SOFTWARE, VOL. 7, NO. 3, MARCH 2012

© 2012 ACADEMY PUBLISHERdoi:10.4304/jsw.7.3.532-543

tion of Product Lifecycle Management (PLM) software for an aircraft equipment manufacturer in China. The re-sults of evaluation are discussed and the model parameters are evaluated with the sensitivity analysis in Section V. The final section draws conclusions and makes sugges-tions for the future research.

II. EVALUATION FRAMEWORK The evaluation procedure proposed model consists of

three stages as shown in Fig. 1. Considering that main factors contribute to the trustworthiness of a software sys-tem, the criteria of STE are chosen in stage 1. Stage 2 de-termines the weights of criteria and includes two parts. Part one is to obtain subjective weights with FAHP, which is a kind of subjective weighting method widely used to express the judgment of evaluators in dealing with the multiple-criteria evaluation. Part two identifies objective

[18], which is well suited for measuring the related contrast intensities of attributes to represent the average intrinsic information transmitted to the DM [33]. Next, the comprehensive weights of criteria are determined by utilizing a combina-tion weighting method [34]. In stage 3, an improved TOPSIS based on vertical projection distance is used to obtain the final rating of software trustworthiness. The details of stepwise procedure can be found in each of the following sub-sections.

III. RESEARCH METHODOLOGY

A. Evaluation Criteria Since software trustworthiness does not appear in any

glos -m-

peting definitions r-ious literatures [1,2,3,7,35,36,37]. No matter which defi-nition the researchers gravitate to, the viewpoint that ST is a holistic property encompassing a set of criteria has reaching a consensus. Basic research on criteria is not green-field research. Stef-fen et al. consider that trustworthiness of software systems is determined by correctness, safety, quality of service (performance, reliability, availability), security, and priva-cy [6]. Tan et al. discuss the key properties that are men-tioned when talking about trustworthiness, which includes functionality, security, usability, safety, portability, main-tainability, and reliability [7]. According to ISO 9126-1, six major char-acteristics of software quality attributes such as functionality, reliability, efficiency, maintainabili-ty, usability, and portability are defined [38]. Fenton and Pfleeger provider rigorous and practical approaches to measure such criteria as maintenance, reliability, learnia-bility, and operability [39]. Zhao et al. consider the soft-ware trustworthiness consists of 5 criteria: availability, reliability, maintainability security and safety [37].

Based on research achievements in existing literatures, the criteria of trustworthiness can be summarized, and a criterion set denoted as = =1,2, ,jC C j n is yielded. For our research, the criterion set provides us a good

FAHP

ImprovedTOPSIS

Constructing theevaluation criteria

Determinesubjectiveweights

Approvecriteria

Y

N

Combinationweightingmethod

Original datamatrix

High Trustworthy0.00 =< Pi < 0.25

Result1

Trustworthy0.25 =< Pi< 0.50

Result2

Low Trustworthy0.50 =< Pi< 0.75

Result3

Determineobjectiveweights

Weightedevaluation matrix

DeterminePi value

Result3Untrustworthy

0.75 =< Pi< 1.00

Entropy

STAGE 3:

STAGE 1:

STAGE 2:

Combinationweights

Figure 1. Evaluation framework of ST

reference list of evaluation criteria of software trustworthi-ness. Since different organizations in all walks of life have the diverse requirements and expectations on software trustworthiness, in an actual evaluation, only important criteria are taken into consideration. The key trustworthy criteria for specific evaluation can be derived from com-prehensive investigation and consultation with different experts from diverse fields.

B. Fuzzy Set TheoryIn order to model the uncertainty or inherent impreci-

sion of human judgments and decision making process, Zadeh originally introduced fuzzy set theory [40]. In as-sessment procedure, such terms of

communality is that they are more or less tainted with ambiguity and vagueness, which makes it difficult to make an exact evaluation in numerical data. Fuzzy theory can play a significant role in dealing with this kind of evaluation situation. The preliminary notions are de-scribed as follows.

1) Fuzzy SetsA fuzzy set is an extension of a crisp set. Crisp sets

only allow full membership or no membership at all. Dif-ferent from the crisp set, fuzzy sets allow partial member-ship, that is, an element may partially belong to a fuzzy set [21]. One major contribution of fuzzy set theory is its ca-pability of modeling the vagueness of human recognition by using of vague data. Such a set is characterized by a membership function, which uses values ranging from 0 to 1 for showing the membership of the object in a fuzzy set. Complete non-membership is represented by 0, and com-plete membership as 1. Values between 0 and 1 represent intermediate degrees of membership [41].

2) Fuzzy Numbers

JOURNAL OF SOFTWARE, VOL. 7, NO. 3, MARCH 2012 533

© 2012 ACADEMY PUBLISHER

Fuzzy numbers are a fuzzy subset of real numbers, and they represent the expansion of the idea of confidence interval [29]. Among various types of fuzzy numbers, the use of triangular fuzzy numbers (TFNs) is fairly common in literatures [24,29,41,42] because of their computational simplicity and their usefulness in promoting representa-tion and information processing in a fuzzy situation. So TFNs are adopted in this study.

A triangular fuzzy number (TFN) denoted as Ã = (l, m, u) has the following type membership function:

- / - , ,

= - / - , , 0, otherwise.

A

x l m l l x m

u x u x u m m x u (1)

where the parameters l, m, and u, respectively, denote the smallest possible value, the most promising value, and the largest possible value that describe a fuzzy event.

Four important operations on TFN are put forward by Chen and Hwang [43]. The following is the operational laws of two TFNs Ã1 = (l1, m1, u1) and Ã2 = (l2, m2, u2). Addition of a fuzzy number

1 2 1 1 1 2 2 2

1 2 1 2 1 2+ , + , + .

A A l ,m ,u l ,m ,u

l l m m u u (2)

Multiplication of a fuzzy number

1 2 1 1 1 2 2 2 1 2 1 2 1 2, , . A A l ,m ,u l ,m ,u l l m m u u (3) Reciprocal of a fuzzy number

-1-11 1 1 1 1 1 1= = 1/ ,1/ 1/ . A l ,m ,u u m , l (4)

3) Linguistic Variables A linguistic variable is a variable values of which are

linguistic terms [44]. In this study, one example for the importance of t

Such five lingly impor-

and values for this variable. Each lingual expression values can be denoted by a TFN within the scale range of 0-1.

TABLE 1. shows the linguistic values and defines each membership function (scale of fuzzy number) [24]. Fur-

nguistic variable, for which the possible expression scales could

TABLE 1. FUZZY LINGUSITIC VARIABLE AND CORRESPONDING TFNS OF LINGUISTIC TERMS FOR CRITERIA

Relative importance of criteria TFNs Equally important (Eq) (1,1,3) Weakly important (Wk) (1,3,5)

Essentially important (Es) (3,5,7) Very strongly important (Vs) (5,7,9)

Absolutely important (Ab) (7,9,9)

TABLE 2. FUZZY LINGUISTIC VARIABLE AND CORRESPONDING TFNS OF LINGUISTIC TERMS FOR CANDIDATE SOFTWARE Software trustworthiness / performance TFNs

Very low (Vl) / Very bad (Vb) (0.0,0.0,3.0) Low(L) / Not good (Ng) (3.0,4.5,6.0)

Middle (Md) / Middle(M) (6.0,7.5,8.5) High(H) / Good(G) (8.5,9.0,9.5)

Very high(Vh) / Very good(Vg) (9.5,10.0,10.0)

lingual expression values can be defined and denoted by a TFN shown in TABLE 2.

C. Weighting Methods Each criterion has its own contribution or influence to

software trustworthiness evaluation. In MCDM methods, influence coefficient refers to the weights of each crite-rion. Up to now, there are two categories of weighting methods: subjective weighting methods and objective weighting methods.

In general, the subjective methods determine weights solely according to the preference or judgments of deci-sion makers, which include AHP, expert investigation methods, etc. The objective methods determine weights by solving mathematical models automatically without any consideration of the deciexample, the entropy method, maximizing deviation me-thod and multiple objective programming, etc.

However, the both weighting methods have their own strong points and defects. The subjective weighting me-thod can explain the evaluation clearly while the objectiv-ity one is relatively weak. The objective weighting me-thod expresses the evaluation in data, but sometimes the weight coefficients of some indexes disagree slightly on actual significance of these criteria, and it is more diffi-cult to explain intuitively than the subjective weighting method [45]. Therefore, in order to balance the advantag-es and disadvantages of subjective and objective weights, two-type weighting methods, specifically, the FAHP and entropy are both utilized in this study.

1) Objective Weighting Method Entropy In information theory, entropy is a criterion used for

measuring the amount of useful information with the pro-vided data. The larger of the difference of evaluating ob-jects under the same criterion, the smaller the entropy value is, which illustrates that this criterion provides more useful information and the weight of this criterion should be relatively large, and vice versa [32]. The entropy theory is based on mathematical computation and ignores

ference, which makes it to be a compa-ratively objective method. The entropy weighting method consists of the following steps:

Step 1: Constructing original data matrix Assure that the problem involves s alternatives eva-

luated against n criteria. The original data matrix can be modeled with quantitative data of each criterion and de-noted as:

1 2

1 11 12 1

2 21 22 2

1 2

X = ,

n

n

n

s s sns

C C CA x x xA x x x

x x xA

where ( =1,2, , )iA i s represents the candidate alternative (software product), ( =1,2, , )jC j n denotes the evalua-tion criterion, ( =1,2, , ; =1,2, , )i jx i s j n is the per-formance of alternative iA under criteria jC .

Step 2: Normalizing the data matrix



Since each criterion has its own dimension and unit, it is essential to transform the various criteria scales into a comparable scale limited between 0 and 1. The original data of evaluation criteria can be normalized as:

=1

= /s

ij i j i ji

v x x . (5)

Therefore, the normalized matrix can be obtained and represented by x= [ ]ij s nV v .

Step 3: Computing the entropy value for the jth crite-rion:

=1

[ ln ]s

j ij iji

N -k v v . (6)

where k is a constant and taken to be1/lns . If all i jx are identical, = 1/ijv s and 1jN .

Step 4: Calculating the objective weight vector (1)w based on the entropies and the element can be calculated as:

(1)

=1

= 1- / 1- = 1,2, , .n

j j kk

w N N j n (7)

where (1)jw is the objective weight of the jth criterion.

2) Subjective Weighting Method AHP is developed by Saaty [46] and addresses how

to determine the relative importance of a set of criteria in a multi-criteria decision problem. In conventional formu-lation of AHP, human judgment is described by crisp real numbers. However, in many real world applications, human pair-wise judgment is highly ambiguous and un-certain. In order to effectively handle the problems with imprecise and incomplete information, an approach combined the fuzzy set theory with AHP was first intro-duced by Van Laar-hoven and Pedrcyz [16], which lead to study on a novel decision analysis method Fuzzy Ana-lytic Hierarchy Process (FAHP). FAHP utilizes fuzzy numbers in place of exact numbers for the pair-wise comparisons.

In this study, the FAHP initiated by Buckley [17] is employed. Assume that an evaluation group includes m members. The procedure of determining the subjective weights of evaluation criteria through FAHP can be out-lined as follows:

Step 1: Build fuzzy pair-wise comparison matrices among all the criteria by using of (8),

12 1

21 2

1 2

11

= =

1

k kn

k knk k

ij

k kn n

a aa a

A a

a a

(8)

where-1

= , , =1, ,k kij jia a i j k m. kA represents the

fuzzy pair-wise comparison matrix of criteria given by kth evaluator, and k

ija denotes the relative importance of two criteria iC and jC .

Step 2: Aggregate the subjective judgment of m mem-bers.

This study employs the geometric mean method sug-gested by Buckley to compute the element of synthetic

pair-wise comparison matrix A of different evaluators, that is:

1/1 2=mm

ij ij ij ija a a a . (9) Step 3: Calculate fuzzy subjective weights of each cri-

terion. In this step, the method of normalization of the geo-

metric mean of the rows (NGM) [47] is adopted to obtain the fuzzy subjective weights vector (2)w . To multiply the n elements in each row and take the nth root, the fuzzy eigenvector from A matrix is obtained, and then norma-lizes the eigenvectors via (10):

(2) 1/n 1/n

11 1

= , , = 1,2, ,n nn

i ij iji=j= j=

w a a i j n. (10)

where (2)iw is the fuzzy subjective weight of the ith crite-

rion, expressed in a TFN (2)(1) (1) (1)=i w w wii ii ii

w l m u .

Step 4. Defuzzification Defuzzification is a technique to convert the fuzzy

number into crisp real numbers [29]. The above subjec-tive weights of criteria are indicated by triangular fuzzy numbers. In order to match the objective weights of crite-ria represented by exact numbers, one non-fuzzy method for TFNs is important and necessary. Various defuzzifica-tion methods are available and serve this purpose, the graded mean integration representation method adopted in this paper is derived from Chen and Hsieh [48]. The defuzzification value of a TFN Ã = (l, m, u) can be ob-tained by (11).

+ 4 +( ) = 6

l m uR A . (11)

Accordingly, the subjective weights vector (2)w can be obtained by using of (11).

Assume that ( )R A and ( )R B are the graded mean inte-

gration representations of TFNs A and B , respectively, and define that:

> ( ) > ( ),

= ( ) = ( ),

< ( ) < ( ).

A B R A R B

A B R A R B

A B R A R B

3) Combination Weighting Method In order to obtain the synthesized weights, the combi-

nation weighting method is used to combine the subjec-tive weighting and the objective entropy weighting in this study. The normalization (5) results in the normalized matrix with dimensionless and unit-free data, which re-flects the difference of raw data with respect to every criterion and is suitable for computing weights based on the entropy theory. However, some evaluating criteria are benefit criteria (i.e., the higher the better) whereas others are cost criteria (i.e., the smaller the better). In order to eliminate the anomalies resulting from the dif-ference of criterion types, the matrix x= [ ]ij s nV v should be further normalized by the following formula:

If jC is a cost criterion: = (max - ) / (max - min ).ij ij ij ij ijjj j

u v v v v (12)



If jC is a benefit criterion: = ( - min ) / (max - min ). ij ij ij ij ijj jj

u v v v v (13)

The generation equation for the combination weights vector has been proposed by Wang et al. and can be ex-pressed as follows [14]:

2( )

=1

= kk

k

W w . (14)

where (1) w and (2) w denotes the objective weights from entropy and the subjective weights through FAHP, respectively. k is the linear combination coefficient and is deducted based on the optimization method and Jaynes maximal entropy theory [45]. The value of k is given as:

( )

1 12

( )

=1 1 1

exp{-[1+ (1- )/ (1- )]}

exp{-[1+ (1- )/ (1- )]}=

s nk

j iji= j=

s nk

j ijk i= j=

q w u q

kq w u q

. (15)

where q is the balance coefficient and 0 < <1q . The

constraint condition is =1

0 1, and = 1l

k kk

.

Accordingly, the combination weights vector can be obtained and expressed as 1 2 = , , , , ,j nw w w wW .

D. Establishing the Rating of Candidate Software 1) Constructing Decision Matrix Given s alternatives (software products) Ai (i s),

n criteria Cj (j n) and m evaluators, the evaluators de trustworthiness rating with respect to each attribute by using TFNs shown in the Ta-ble 2. The typical structure of fuzzy decision matrix can be expressed as:

1 2

1 2

1 11 12 1

2 21 22 2

1 2

E =,

n

n

k k kk n

k k kn

k k ks s s sn

C C Cw w w

A e e eA e e e

A e e e

where kije is a TFN and represents the fuzzy judgment

rating of alternative iA concerning the criterion jC given by kth evaluator, and the weighting vector

1 2 = , , , nw w wW obtained in Section III - C is represented by real numbers.

Since each evaluator has different individual know-ledge and experience, the perception toward the software trustworthiness varies among evaluators. Assume each evaluator has the same importance, this study employs the method of average value to integrate the fuzzy judgment value

kije , that is,

1 2= 1/ mij ij ij ije m e e e . (16)

=1 =1 =1= ; = ; =

m m m

k k ke e eij iijj iijjk k ke e eij ij ij

l m ul m u

m m m. (17)

where ije denotes the average fuzzy scale of alternative Ai regarding the evaluation criteria Cj for m evaluators. After defuzzying the integrated fuzzy decision matrix

= ( )ij s nR r in (11), and the decision matrix is obtained and denoted as = ( )ij s nF f , where ijf is a real number.

2) Improved TOPSIS Method TOPSIS was originally proposed by Hwang and

Yoon [19], which is a practical and useful technique for evaluation of a number of possible alternatives through measuring Euclidean distances. The underlying logic of TOPSIS is that the optimal alternative should have the shortest distance from the positive idea solution (PIS), i.e., the solution that maximizes the benefit criteria and minimizes the cost criteria; and the longest distance from the negative ideal solution (NIS), i.e., the solution that maximizes the cost criteria and minimizes the benefit criteria [11].

However, traditional TOPSIS based on the Euclidean distance has an obvious defect, that is, the alternative that has the shortest Euclidean distance from the positive idea solution (PIS) may also have the closest Euclidean dis-tance from the negative ideal solution (NIS) [49,50,51]. Therefore, the ranking provided by traditional TOPSIS can not accurately reflect the difference of alternatives. In order to avoid it, this study proposes to use an improved TOPSIS [52] based on vertical projection distance which has an advantage that the alternative nearest to the PIS is farthest from the NIS.

The improved TOPSIS method is summarized as fol-lows.

Step1: Calculate weighted normalized decision matrix xG = [ ]ij ij s ng , and ijg is defined as:

= =1,2, , ; =1,2, , .ij j ijg w f i s j n (18)

Step 2: Determine positive idea solution (PIS) +jS de-

fined as follows: 1

1+

21

max{ } if = = 1,2, ,

min{ } if

ij ji m

jij j

i m

g C S j n.

g C (19)

where 1 and 2 indicate the benefit criteria set and cost criteria set, respectively.

Step 3: Transfer the origin of coordinates to the posi-tive ideal point for simplifying calculation. The elements of removed matrix T are given as:

+= - ,

= 1,2, , ; = 1,2, , .ij ij jt g S

i m j n (20)

After translation, the PIS becomes {0, ,0} and the NIS is denoted as - -= { 1,2, , }jH H j = n .

Step 4: Determine negative idea solution -jH as fol-

lows: - = , ,

1 , = 1,2, , ; = 1,2, , .j kj kj ijH t t t

k s i s j n (21)

Step 5: The vertical projection distance of each alter-native to PIS can be calculated by:



=1=

n¯ ¯

i i j ijj

P = H T H t . (22)

where Ti is the ith row vector of T matrix. The smaller the iP , the better the trustworthiness of the software is.

Step 6: The evaluation team defines the range for the linguistic variable of software trustworthiness within a scale of 0-1. Thus, according to the corresponding value of iP , the trustworthy rating of the ith candidate software can be determined.

IV. A CASE OF EVALUATION With the development of enterprise information, such large manufacturing industry as automobile, airplane and aerospace, have called for effective implementations of Product Lifecycle Management (PLM) software to en-hance enterprisecompetence. In this section, the proposed method is ap-plied to evaluate the candidate PLM software trustwor-thiness for an aircraft equipment manufacturer in China, and supports decision makers of the enterprise to select the trustworthy PLM software. This evaluation is sup-ported by results that are gained from a series of ques-tionnaires and widespread investigation. After the pre-liminary screening, three PLM software, A1, A2, A3, have remained in the candidate list. Four experts, D1, D2, D3, D4, selected from different subsystems of the organiza-tion, form an evaluator team (for each evaluator with the same importance).

In order to reach consensus on evaluation criteria of PLM software trustworthiness for the enterprise, we con-duct a widespread investigation and consultation with several experts, including three professors in information system fields, two evaluation experts from third part and

e-partment. After brainstorming sessions, six determinant criteria such as maintainability, functionality, portability, operability, learnability and co-existence are considered to be essential for evaluating the trustworthiness of PLM system. In this evaluation, functionality is regarded as

qualitative criteria, whereas the remaining criteria are regarded as quantitative criteria. Trustworthy criteria structure and their measurement are listed in Fig. 2. The following is the illustration of each quantitative criterion and their measurable objectives.

Portability means the behavior or ability of software during measurement of porting the software system to target the porting activity from one host environment to another, which can be measured by ET/ER , where ET is the resource environment, and ER denotes the resource measurement of creating this system in the target envi-ronment. The lower the ratio, the better the portability is.

Co-existence is the performance of software system harmonized with the other software in a common envi-ronment sharing common resources. In the evaluation, co-existence is measured by estimating of effort (person-months) in redeveloping candidate software in order to fulfill good integration with diverse independent soft-ware.

Maintainability means the software performance in-cluding both defect repairs and enhancements in re-sponse to new requirements. The maintenance activity can be regarded as the certain alteration, and the speed of finishing the alteration is a key to measure maintenance. Therefore, this evaluation adopts the Mean Time to Re-pair (MTTR) to measure the maintenance, which is the average time of finishing the alteration and returning to the normal functioning.

Training 20 new employees, learnability can be meas-ured by noting the speed of learning, concretely, the av-erage hours of cultivating before single operation, and operability is measured by recording the average num-bers of making mistakes during training. The method focuses on the workload of learning and operating the software system.

We gather the related measured data and information by taking a variety of ways, including introduction, building test platform in enterprise and try- outs for the candidate software, as well as consulting the

Objectivecriteria

Evaluating thetrustworthiness of

PLM Software

the average time of finishing thealternation and returning to the

normal functioning

the average hours of cultivatingbefore single operation

the average numbers of makingmistakes during training

the estimated effort (in person-months) in the redevelopment

ET/ER

deriving from the judgment ofexperts

Subjectivecriteria

A1

A2

A3

portability

co-existence

operability

functionality

learnability

maintainability

Figure 2. Trustworthy criteria hierarchy of evaluating PLM software



other aircraft equipment manufacturers that have applied the candidate PLM systems. The results construct the origi-nal matrix of criteria for each candidate PLM system and are shown in TABLE 3.

The trustworthiness level of PLM software can be as-sessed by aggregating the measurements for corresponding criteria and the implementing procedure of evaluation is summarized as follows.

Step 1: Computing the objective weights of criteria. TABLE 3 provides the original information collected

from inside and outside of the enterprise. Notably, the crite-rion C1 is qualitative criteria. The information under C1 is the experts perception described by linguistic terms with TFNS (shown in TABLE 2) after trying out the three can-didate software. Therefore, before computing the objective weights, the defuzzification of TFNs is performed by (11). After all these data are transferred to exam numbers, nor-malization of the values of matrix is made by using (5), as in TABLE 4. Then the objective weights can be calculated in (6) and (7), and the result is shown in TABLE 5.

Step 2: Calculating the subjective weights of criteria. First, the evaluators are asked to conduct their subjective

judgments on the relative importance of criteria based on their background of experience and knowledge, and estab-lish the fuzzy pair-wise comparison matrices. The related linguistic variable can be indicated by TFNs shown in TABLE 1. Next, to aggregate the fuzzy pair-wise compari-son matrix of criteria of four experts in (9), and the inte-grated results is presented in TABLE 6. Then, the subjec-tive weights can be obtained using (10) and (11), which are

shown in TABLE 7. The normalized subjective weights vector is computed as:

(2) = 0.4255 0.0815 0.1536 0.0746 0.2180 0.0468 . w Step 3: Determining the final combination weights

through combination weighting method.

the optimal linear combination weighting method is utilized by = 0.5q . According to (12) - (15), the combination weights are calculated as:

(1) (2) (1) (2)1 2= + = 0.4762 + 0.5238

= 0.2390,0.0799,0.1164,0.1997,0.2443,0.1206 .W w w w w

Fig.3 shows the difference of criteria weights through en-tropy, FAHP and combination weighting methods. It is

00.050.10.150.20.250.30.350.40.45

C1 C2 C3 C4 C5 C6

Subjective weights Objective weights

Combination weights

Figure 3. Comparison of different weights

TABLE 3. THE ORIGINAL MATRIX

Functionality (C1) Learnability(C2) (hours)

Operability(C3) (times)

Co-existence (C4) (person-months)

Maintainability (C5) (hours) Portability (C6)

A1 M(6.0,7.5,8.5) 46.35 h 19.10 t 6 p-m 2.97 h 0.2857 A2 G(8.5,9.0,9.5) 59.59 h 25.55 t 3 p-m 1.25 h 0.1667 A3 Vg(9.5,10,10) 38.52 h 29.65 t 8 p-m 2.14 h 0.3478

TABLE 4. NORMALIZED MATRIX

C1 C2 C3 C4 C5 C6 A1 0.2816 0.3209 0.2571 0.3529 0.4670 0.3571 A2 0.3418 0.4125 0.3439 0.1765 0.1965 0.2083 A3 0.3766 0.2666 0.3990 0.4706 0.3365 0.4346

TABLE 5. OBJECTIVE WEIGHTS OF CRITERIA

C1 C2 C3 C4 C5 C6 Nj 0.9936 0.9852 0.9857 0.9362 0.9483 0.9618

1- Nj 0.0064 0.0148 0.0143 0.0638 0.0517 0.0382 w(1)

j 0.0338 0.0782 0.0756 0.3372 0.2733 0.2019

TABLE 6. INTEGRATED PAIR- PINION

C1 C2 C3 C4 C5 C6 C1 1 (3.3437,5.6637,7.7700) (1.9679,4.2128,6.2997) (3.2010,5.5444,7.2968) (1.7320,2.2360,4.5825) (4.5825,5.9160,7.9372) C2 (0.1287, 0.1766, 0.2991 1 (0.3398, 0.5308, 1.0878) (0.8091, 1.2359, 2.1407) (0.2374, 0.4111, 0.7598) (0.8801, 1.3161, 2.1407) C3 (0.1587, 0.2374, 0.5081 (0.9193, 1.8841, 2.9428) 1 (0.8801, 1.9680, 3.6371) (0.3936, 0.6560, 1.6266) (2.5900, 4.7867, 6.8525) C4 (0.1370, 0.1804, 0.3124 (0.4671, 0.8091, 1.2359) (0.2582, 0.4671, 1.0000) 1 (0.1690, 0.2582, 0.5774) (0.8801, 1.9680, 3.6371) C5 (0.2182, 0.4472, 0.5774) (1.3161, 2.4323, 4.2129) (0.6148, 1.5244, 2.5407) (1.7321, 3.8730, 5.9161) 1 (2.9428, 5.2068, 7.2969) C6 (0.1260, 0.1491, 0.2182) (0.4671, 0.7598, 1.1362) (0.1459, 0.2089, 0.3861) (0.2749, 0.5081, 1.1362) (0.1370, 0.1921, 0.3398) 1

TABLE 7. FUZZY SUBJECTIVE WEIGHTS OF CRITERIA AND DEFUZZIFICATION

C1 C2 C3 C4 C5 C6 wi (0.1967,0.4332,0.9484) (0.0370,0.0787,0.1997) (0.0597,0.1479,0.3877) (0.0308,0.0713,0.1889) (0.0823,0.2240,0.4963) (0.0220,0.0454,0.1129)

R(wi) 0.4797 0.0919 0.1731 0.0841 0.2458 0.0527



TABLE 8. FUZZY DECISION MATRIX OF FOUR EVALUATORS C1 C2 C3 C4 C5 C6 D1 D2 D3 D4 D1 D2 D3 D4 D1 D2 D3 D4 D1 D2 D3 D4 D1 D2 D3 D4 D1 D2 D3 D4 A1 Md Md Md Md H Md Md H H Vh Vh H H Md H H H Md Md L Md Md Md H A2 H H H H Md Md L H Md H H Md H H Vh Vh Vh Vh H H H H H Vh A3 Vh Vh Vh Vh Vh H H Vh Md H H L Md L Md H H H Md Md Md L L H

TABLE 9. INTEGRATED FUZZY DECISION MATRIX

C1 C2 C3 C4 C5 C6 A1 (6.0000,7.5000,8.5000) (7.2500,8.2500,9.0000) (9.0000,9.2500,9.7500) (7.8750,8.6250,9.2500) (5.8750,7.1250,8.1250) (6.6250,7.8750,8.7500) A2 (8.5000,9.0000,9.5000) (5.8750,7.1250,8.1250 (7.2500,8.2500,9.0000) (9.0000,9.2500,9.7500) (9.0000,9.2500,9.7500) (8.7500,9.2500,9.6250)

A3 (9.5000,1.0000,1.0000) (9.0000,9.2500,9.6250) (6.5000,7.5000,8.3750) (5.8750,7.1250,8.1250) (7.2500,8.2500,9.0000) (5.1250,6.3750,7.5000)

TABLE 12. REMOVED DECISION MATRIX

C1 C2 C3 C4 C5 C6 A1 -0.5975 -0.0849 0 -0.1705 -0.5802 -0.1708 A2 -0.2191 -0.1747 -0.1455 0 0 0 A3 0 0 -0.2303 -0.4743 -0.3054 -0.3467

TABLE 13. THE TRUSTWORTHY RATING OF CANDIDATE SOFTWARE

0.00 0.25 0.25 0.50 0.50 0.75 0.75 1.00

High Trustworthy

Trustwor-thy

Low Trustworthy

Un-trustworthy

A1 0.8485 A2 0.1949A3 0.5754

clear that the combination weights balance the subjective of

ef-fectively avoid the subjective or objective one-sidedness of weights.

Step 4: Constructing decision matrix. Evaluators are asked to evaluate the trustworthiness of

software by using the linguistic terms in TABLE 2 based on the raw information of original matrix shown in TABLE 3. TABLE 8 shows the fuzzy judgment values of each evalua-tor. Based on the fuzzy numbers defined in TABLE 2, the linguistic scales can be transferred to the corresponding fuzzy numbers. To employ (16) and (17), the integrated decision matrix established using TFNs is obtained and presented in TABLE 9, for 15e as an example, the trustwor-

thy linguistic scales of the 1st candidate software A1 with respect to criterion C5 provided d, Md, L, respectively, then

4 4 4

15 e e e15 15 15=1 =1 =1

4, 4, 4

8.5 + 6.0 + 6.0 + 3.0 4, 9.0 + 7.5 + 7.5 + 4.5 4, =

9.5 +8.5 +8.5 + 6.0 4

= 5.875,7.125,8.125 .

k k kk k k

e = l m m

The other matrix elements can be obtained by the same calculation procedure. Next, to defuzzy the normalized fuzzy decision matrix using (11), and the non-fuzzy deci-sion matrix is established and shown in Table 10, where the elements of matrix are the real numbers between 0-10.

Step 5: To employ (18) to calculate the weighted decision matrix.

Table 11 gives the weighted result. In the decision matrix, the trustworthy scales of candidate software under each cri-terion can be regarded as benefit-related performance. In other words, the higher the trustworthiness rating of the ith alternative with respect to the jth criterion, the better the trustworthiness of this alternative is. Therefore, according to (19), the positive idea solution (PIS) +S can be determined and presented as:

+ = 2.3701,0.7407,1.1009,1. 8 ,2.3107,1.1130 .88 8S Step 6: According to (20), the removed matrix T is ob-tained and shown in TABLE 12, from which, the negative idea solution -H can be determined in (21) and shown as follows:

- = -0.5975,-0.1747,-0.2303,-0.4743,-0.5802,-0.3467 .H Step 7: The vertical projection distance of each alterna-

tive to the PIS can be calculated by using (22), and the ex-perts group provides the range for the linguistic variable of software trustworthiness within a scale of 0-1 based on their under-standing and background knowledge. The results are shown in TABLE 13. The PLM software A2 with the smal-lest Pi value (0.1949) has the highest trustworthiness among the three candidate PLM software and becomes the most preferred one for the aircraft equipment manufacturer.

TABLE 10. NON-FUZZY DECISION MATRIX C1 C2 C3 C4 C5 C6 A1 7.4167 8.2083 9.4583 8.6042 7.0833 7.8125 A2 9.0000 7.0833 8.2083 9.4583 9.4583 9.2292 A3 9.9167 9.2708 7.4792 7.0833 8.2083 6.3542 TABLE 11. WEIGHTED DECISION MATRIX C1 C2 C3 C4 C5 C6 A1 1.7726 0.6558 1.1009 1.7183 1.7305 0.9422 A2 2.1510 0.5660 0.9554 1.8888 2.3107 1.1130 A3 2.3701 0.7407 0.8706 1.4145 2.0053 0.7663



V. SENSITIVITY ANALYSIS AND DISCUSSIONS The sensitivity is measured by exchanging each crite-

. Hence, 15 dif-ferent interchanging and calculations are formed. We use Pij values for each calculation and give them different names, for example P14 means the weights of criterion 1 and criterion 4 have changed. TABLE 14 shows the 15 cal-culations results and Fig.4 summarizes 15 Pij values of the alternatives on graph.

As can be seen in these calculations, software A2 has the minimum Pij values in all 15 calculations and all the 15 values are less than 0.5, among which there are 10 Pij val-ues in the range (0.00 0.25) with high trustworthy. By contrast, software A1 has the highest Pij values in the first, third, fourth, sixth, seventh, eighth, ninth, 10th, 12th, 13th and 14th calculations. And all the 15 Pij values of software A1 are in the range of low trustworthy and untrustworthy, among which there are 11 Pij values in the range of un-trustworthy. In addition, except that P16 and P56 values are in the range of untrustworthy, software A3 has 13 Pij values belong to trustworthy and low trustworthy, among which P24 and P25 values are in the range of trustworthy. These results prove that the trustworthy PLM software chosen for the aircraft equipment manufacturer is the best one as we detected in the paper.

sensitivity analysis implies that the trust-worthy rating of alternative software is not obviously sensi-tive to the changes in the criterion weight, and illustrates the robustness of the evaluation model.

VI . CONCLUSIONS Nowadays, software trustworthiness, a holistic property

encompassing a set of trustworthiness criteria, arouses the increasing concerns in the academia and industry. In order to explore and build a new evaluation system for trustwor-

TABLE 14. THE Pij VALUES OF 15 INTERCHANGES

0.00 - 0.25 0.25 - 0.50 0.50 - 0.75 0.75 - 1.00

High Trustworthy

Trustwor-thy

Low Trustworthy

Un- trustworthy

P12 A1 0.6495 A2 0.3215 A3 0.5754 P13 A1 0.5763 A2 0.2029 A3 0.7461 P14 A1 0.7758 A2 0.1555 A3 0.6727 P15 A1 0.8502 A2 0.2008 A3 0.5678

(Continued Table) P16 A1 0.7559 A2 0.0973 A3 0.9273 P23 A1 0.8653 A2 0.2115 A3 0.5474 P24 A1 0.8585 A2 0.3553 A3 0.3865 P25 A1 0.6719 A2 0.4172 A3 0.4500 P26 A1 0.8344 A2 0.2340 A3 0.5080 P34 A1 0.7952 A2 0.2601 A3 0.5300 P35 A1 0.5884 A2 0.3091 A3 0.6191 P36 A1 0.8446 A2 0.1974 A3 0.5711 P45 A1 0.7771 A2 0.1950 A3 0.6283 P46 A1 0.9004 A2 0.1950 A3 0.6419 P56 A1 0.7779 A2 0.1950 A3 0.8145

0

0.2

0.4

0.6

0.8

1

A1 A2 A3

Figure 4. The new Pij values of the candidate software



thy software, we regard STE as a MCDM problem. Differ-ent from other studies in the literatures, this paper develops a new approach for evaluating software trustworthiness with fuzzy MCDM methodology.

The proposed approach is based on the combination weighting method and the improved TOPSIS method. In the evaluation procedure, both the vagueness of the evalua-

of information included in numerical data are taken into account. FAHP method is used to determine the importance

avoid the subjective one-sidedness of weights, entropy weighting method is employed to calculate objective weights by exploiting the useful information of raw data. Then the linear combination weighting method is used to synthesize the subjective weights and objective weights. Additionally, in order to calculate the trustworthy rating of the candidate software, an improved TOPSIS based on ver-tical projection distance is taken to replace the conventional TOPSIS based on Euclidean distances, which offers a way to overcome the shortcoming of conventional TOPSIS. In the application, the presented approach is adopted to eva-luate the trustworthiness of PLM software for an aircraft equipment manufacturer in China. Finally, by exchanging

nother criterion weight, the study conducts the sensitivity analysis to illustrate the ro-bustness of the evaluation model. The calculated results of sensitivity indicate that software A2 has the minimum Pij values in all 15 calculations and all the 15 values belong to the rating of trustworthy.

Although the case study is related to a specific software product and industry, the underlying concepts and methods of this evaluation model can be applied to evaluate other software systems. Furthermore, it is should be pointed out that this study focused primarily on static criteria as the basis of evaluation criteria, and neglected other dynamic influencing factors of ST in different stage of software life cycle. Future research can consider constructing a dynamic criteria structure and improving evaluation approach with other methods.

ACKNOWLEDGMENT The authors appreciate the support from the National

Natural Science Foundation of China (Nos.70871032, 71071045 and 71131002), Anhui Provincial Natural Science Foundation under fund No. 11040606Q27. Special thanks are given to Prof. Jing-biao Huang, Dr. Chun-feng Liu, and other anonymous reviewers for their constructive suggestions.

REFERENCES [1] J. B. Michael, J. M. Voas, and R. C. Linger, 2nd National Software

& Workshop series (NSS2) Report: Trustworthy Systems Workshop Report , <http://www.cnsoftware.org/nss2report/>, 2004.

[2] S. Becker, M. Boskovic et al, Trustworthy Software Systems: A Discussion of Basic Concepts and Terminology , ACM SIGSOFT

Software Engineering Notes, vol. 31, no. 6, pp. 1 18, November 2006.

[3] W. Hasselbring and R.Reussner, Toward trustworthy software systems , IEEE computer, vol. 39, no. 4, pp. 91 92, April 2006.

[4] K. Liu, Z. G. Shan, J. Wang et al, Overview on major research plan of trustworthy software , Bulletin of National Natural Science Foundation of China, vol. 22, no. 3, pp.145 151, 2008. (in Chinese)

[5] D. L. Parnas, J. Schouwen, and S. P. Kwan, Evaluation of safety-critical software , Communications of ACM, vol. 33, no. 6, pp. 636-648, June 1990.

[6] B. Steffen, H. Wilhelm, P. Alexandra et al, Trustworthy software systems: a discussion of basic concepts and terminology , ACM SIGSOFT Software Engineering Notes archive, vol. 31, no. 6, pp. 118, November 2006.

[7] T. Tan, M. He, Y. Yang, Q. Wang, and M. S. Li, An Analysis to Understand Software Trustworthiness , The 9th International Conference for Young Computer Scientists, pp. 2366 2371, 2008.

[8] Y. Y. Yuan, Q. Han, Data Mining Based Measurement Method for Software Trustworthiness , Proceedings-2010 International Symposium on Intelligence Information Processing and Trusted Computing(IPTC 2010), pp. 293 296, December 2010.

[9] H. P. Zhang, F. D. Shu, Y. Yang, X. Wang, and Q. Wang, A fuzzy-based method for evaluating the trustworthiness of software processes , Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 6195, pp. 297 308, 2010.

[10] L. Zhu, A. Aurum, I. Gorton, and R. Jeffery, Tradeoff and sensitivity analysis in software architecture evaluation using analytic hierarchy process , Software Quality Journal, vol. 13, no. 4, pp. 357 375, December 2005.

[11] T. C Wang and H. D. Lee, Developing a fuzzy TOPSIS approach based on subjective weights and objective weights , Expert Systems with Applications, vol. 36, no. 5, pp. 8980 8985, July 2009.

[12] Y. J. Wang and H. S. Lee, Generalizing TOPSIS for fuzzy multiple-criteria group decision-making , Computers and Mathematics with Applications, vol. 53, pp. 1762 1772, April 2007.

[13] C. M. Feng and R. T. Wang, Performance evaluation for airlines including the consideration of financial ratios , Journal of Air Transport Management, vol. 6, no. 3, pp. 133-142, July 2000.

[14] Y. J. Wang, H. S. Lee, and K. Lin, Fuzzy TOPSIS for multicriteria decision-making , International Mathematical Journal, vol. 3, pp. 367 379, 2003.

[15] R. E. Bellman and L. A. Zadeh, Decision-making in a fuzzy environment , Management Science, vol. 17, no. 4, pp. 141 164, December 1970.

[16] P. J. M. Van Laarhoven, and W. Pedrcyz, A fuzzy extension of , Fuzzy Sets and Systems, vol. 11, pp. 199

227, 1983. [17] J. J. Buckley, Fuzzy hierarchical analysis , Fuzzy Sets and Systems,

vol. 17, no. 3, pp. 233 247, December 1985. [18] C. E. Shannon and W. Weaver, The mathematical theory of

communication. Urbana: The University of lllinois Press, 1947. [19] C. L Hwang and K. Yoon, Multiple attribute decision making

Methods and application. Berlin: Springer, 1981. [20] S. W. Hsiao and J. R. Chou, A Gestalt-like perceptual measure for

home page design using a fuzzy entropy approach , International Journal of Human-Computer Studies, vol. 64, pp. 137 156, February 2006.

[21] Ì. and Performance evaluation of Turkish cement firms with fuzzy analytic hierarchy process and TOPSIS methods , Expert Systems with Applications, vol. 36, pp. 702 715, January 2009.

[22] C. Fausto, Fuzzy TOPSIS approach for assessing thermal-energy storage in concentrated solar power (CSP) systems , Applied Energy, vol. 87, no. 2, pp. 496 503, February 2010.

[23] H. S. Gao, J.X. Ran, Y.Q. Sun, Risk evaluation of communication network of electric power based on improved FAHP , Systems Engineering - Theory & Practice, vol. 28, no. 3, pp. 133 138, 2008. (In Chinese)



[24] T. Y. Hsieh, S. T. Lu, and G. H. Tzeng, Fuzzy MCDM approach for planning and design tenders selection in public office buildings , International Journal of Project Management, vol. 22, no. 7, pp. 573 584, October 2004.

[25] S. S. Kima, I. O. Yang, M. S. Yeo, and K. W. Kim, Development of a housing performance evaluation model for multi-family residential building in Korea , Building and environment, vol. 40, no. 8, pp. 1103 1116, August 2005.

[26] P. A. Morteza, Project selection for oil-fields development by using the AHP and fuzzy TOPSIS methods , Expert Systems with Applications, vol. 37, no. 9, pp. 6218 6224, September 2010.

[27] K. K. Pochampally, S. M. Gupta, and S. V. Kamarthi, Evaluation of production facilities in a closed-loop supply chain: a fuzzy TOPSIS approach . Proceedings of the SPIE International Conference on Environmentally Conscious ManufacturingIII, Providence, Rhode Island, pp. 125-138, October 2003.

[28] C. T. Tsao, Evaluating investment values of stocks using a fuzzy TOPSIS approach , Journal of Information & Optimization Science, vol. 24, no. 2, pp. 373 396, 2003.

[29] S. H. Tsaur, T. Y. Chang, and C. H. Yen, The evaluation of airline service quality by fuzzy MCDM , Tourism Management, vol. 23, pp. 107 115, April 2002.

[30] T. C. Wang and T. H. Chang, Application of TOPSIS in evaluating initial training aircraft under a fuzzy environment , Expert Systems with Applications, vol. 33, no. 4, pp. 870 880, November 2007.

[31] H. Y. Wu, G. H. Tzeng, and Y. H. Chen, A fuzzy MCDM approach for evaluating banking performance based on Balanced Scorecard , Expert Systems with Applications, vol. 36, pp. 10135 10147, August 2009.

[32] Z. H. Zou, Y. Yun, and J. N. Sun, Entropy method for determination of weight of evaluating in fuzzy synthetic evaluation for water quality assessment indicators, Journal of Environmental Sciences, vol. 18, no. 5, pp. 1020 1023, October 2006.

[33] M. Zeleny, Multiple criteria decision making, New York: Springer,1999.

[34] Z. Y. Wang, H. F. Gu, X. X. Yi, and S. R. Zhang, A Method of Determining the Linear Combination Weights Based on Entropy , Systems Engineering - Theory & Practice, vol. 23, no. 3, pp. 112116, 2003. (in Chinese)

[35] F. B. Schneider, Trust in Cyberspace. Washington. DC: National Academy Press, 1998.

[36] M. Li, X. Z. Zhou, J. C. Wang et al, A perspective of software trustworthiness based on distrustable factors , Proceedings of the 2009 IEEE International Conference on Networking, Sensing and Control( ICNSC 2009), pp. 873 878, 2009.

[37] X. X. Zhao, Y. Shi, Y. Liu, and L. L. Zhang, An empirical study of the influence of Software Trustworthy Attributes to Software Trustworthiness , 2nd International Conference on Software Engineering and Data Mining (SEDM 2010), pp. 603 606, June 2010.

[38] ISO 9126-1, Software engineering Product quality Part 1: Quality model. ISO/IEC, Published standard, 2001.

[39] N. E. Fenton and S. L. Pfleeger, Software Metrics: A Rigorous and Practical Approach, Second Edition. Beijing: China Machine Press, 2004. (in Chinese)

[40] L. A. Zadeh, Fuzzy sets , Information and Control, vol. 8, pp. 338353, 1965.

[41] M. Z. Naghadehi, R. Mikaeil, and M. Ataei, The application of fuzzy analytic hierarchy process (FAHP) approach to selection of optimum underground mining method for Jajarm Bauxite Mine, Iran , Expert Systems with Applications, vol. 36, no. 4, pp. 8218 8226, May 2009.

[42] and , Journal of Intelligent Manufacturing, vol.

17, no. 2, pp. 179 190, 2006. [43] S. J. Chen and C. L. Hwang, Fuzzy multiple attribute decision

making, methods and applications , In: Lecture notes in economics and mathematical systems, vol. 375. New York: Springer, 1993.

[44] L.A. Zadeh, The concept of a linguistic variable and its application to approximate reasoning , Information Sciences, vol. 8, pp. 199249(I), 301-357(II), 1975.

[45] J. J. Wang, C. F. Zhang, Y.Y. Jing, and G.Z. Zheng, Using the fuzzy multi-criteria model to select the optimal cool storage system for air conditioning , Energy and Buildings, vol. 40, no. 1 pp. 20592066, 2008.

[46] T. L. Saaty, The analytic hierarchy process. New York: McGraw-Hill, 1980.

[47] T. Y. Chou, C. L. Hsu, and M.C. Chen, A fuzzy multi-criteria decision model for international tourist location selection , International Journal of Hospitality Management. vol. 27, no. 2, pp. 293 301, June 2008.

[48] S. H. Chen, and C. H. Hsieh, Representation, ranking, distance, and similarity of L-R type fuzzy number and application , Australian Journal of Intelligent Processing Systems, vol. 6, no. 4, pp. 217 229, 2000.

[49] A. S. Milani, C. EI-Lahham, and J. A. Nemes, Different approaches in multiple-criteria optimization using the Taguchi Method: A case study in a cold heading process , Journal of Advanced Manufacturing Systems, vol. 3, pp. 53 68, 2004, DOI: 10.1142/S0219686704000399.

[50] A. S. Milani, A. Shanian, and C. EI-Lahham, A decision-based approach for measuring human behavioral resistance to organizational change in strategic planning , Mathematical and Computer Modelling, vol. 48, pp. 1765 1774, December 2008.

[51] T. L. Saaty, Fundamentals of Decision Making and Priority Theory with the Analytic Hierarchy Process. Pittsburgh: RWS Publications, 2000.

[52] X. Y. Hua, and J. X. Tan, Revised TOPSIS Method Based on Vertical Projection Distance Vertical Projection Method , Systems Engineering - Theory & Practice, vol. 24, no. 1, pp. 114 119, 2004. (in Chinese)

Li Shi received the B.Sc. degree in Electrical Engineering from Hefei University of Technology, Hefei, China, in 2002. She has been working as an engineering and researcher for an electronics company after studies. Since 2005, she is working for the school of Physical and Electronic Information at Huaibei Normal Univer-sity, Huaibei, China.

She is currently pursuing the Ph.D. degree in the group of Prof. Shan-Lin Yang. Her main research interests are in the areas of multiple criteria decision analysis, decision-making science and technology and trustworthy software evaluation. Shan-Lin Yang received the M.Sc. degree in Computer Science and Application from Hefei University of Technology, Hefei, Chi-na, in 1985. He had been a visiting scholar at Dresden University of Technology, Germany and the University of Melbourne, Aus-tralia in 1986 and 1988, respectively.

He is currently working as a Professor in the School of Man-agement, Hefei University of Technology, Hefei, China. His main research interests include decision-making science and technology, decision support system, operation research, and information management and information system. His work has been reported in many journals such as Computer & Operations Research, Euro-pean Journal Operations Research, International Journal of Pro-duction Economics, Systems Engineering Theory & Prac-tice(Chinese), Journal of Management Science (Chinese), etc.



Kai Li received the B.Sc. degree(1999) in Computer Science Education from Anhui Normal University, Wuhu, China, the M.Sc. degree(2005) in Computer Software and Theory and Ph.D. de-gree(2009) in Management Science and Engineering from Hefei University of Technology, Hefei, China.

He is currently working as an associate professor for School of Management at Hefei University of Technology. His research interests include operations research, optimization algorithm, pro-duction scheduling. He made the paper such as International Jour-nal of Systems Science, Applied Soft Computing, Mathematical and Computer Modelling, Applied Mathematical Modelling.

Ben-gong Yu received the B.Sc. degree in Computer Science, the M.Sc. and Ph.D. degrees in Management Science and Engineering from Hefei University of Technology, Hefei, China, in 1994, 2005, and 2011, respectively. His research interests include information management and information system.



Developing an Evaluation Approach for Software ... · FAHP method is used ... To avoid the...

Documents

Transcript of Developing an Evaluation Approach for Software ... · FAHP method is used ... To avoid the...