Journal of Purchasing & Supply Management 16 (2010) 206–215

Contents lists available at ScienceDirect

Journal of Purchasing & Supply Management

1478-40

doi:10.1

n Corr

Portuga

E-m

pt (J.A.

journal homepage: www.elsevier.com/locate/pursup

Full disclosure of tender evaluation models: Background and application inPortuguese public procurement

Ricardo Mateus a,n, J.A. Ferreira b, Jo~ao Carreira a

a CISED Consultores, Rua de Dona Estefania, 157, 5. %o

Esq., 1000-154 Lisbon, Portugalb CESUR-IST, Av. Rovisco Pais, 1049-001 Lisbon, Portugal

a r t i c l e i n f o

Article history:

Received 5 August 2009

Received in revised form

1 March 2010

Accepted 23 April 2010

Keywords:

Public procurement

Multiple criteria decision analysis

Tender evaluation

92/$ - see front matter & 2010 Elsevier Ltd. A

016/j.pursup.2010.04.001

esponding author. Rua Ferreira de Castro, 8

l. Tel.: +35 1936265287.

ail addresses: ricardo.mateus@sapo.pt (R. Mat

Ferreira), joao.carreira@cised.pt (J. Carreira).

a b s t r a c t

Public procurement rules in the European Union require that public contracting authorities must

publish all tender evaluation criteria and its weights in advance. In order to define sound weights, the

authors argue that the scoring rules for all evaluation criteria must be defined beforehand. Furthermore,

the authors further argue that those scoring rules should also be published, in order to provide much

more relevant and meaningful information for tenderers when preparing their tenders. Based on these

evidences, the Portuguese Public Contracts Code compels public contracting authorities in Portugal to

publish the comprehensive tender evaluation model in advance. This paper presents the Code, outlines

the feasibility of establishing such a model in advance, addresses detailed instructions on how to go

about in developing and applying it, and submits that existing European provisions on public

procurement could yet go further in ensuring effective transparency, competition and best value for

money tenders.

& 2010 Elsevier Ltd. All rights reserved.

1. The european directive on public procurement

The award of public contracts in European Union (EU) memberstates is subject to the respect of the principles of the Treatyestablishing the EU, in particular to the principles of freedom ofmovement of goods, establishment, and to provide services aswell as the principles deriving therefrom, such as equal treatment,non-discrimination, mutual recognition, proportionality, andtransparency.

For the purpose of upholding these principles, the EuropeanParliament and the Council have defined, through Directive 2004/18/EC (European Union, 2004), the provisions to be followed bypublic contracting authorities (CAs) for the award of publiccontracts above certain threshold amounts.

The text of the Directive is based on Court of Justice case-law(WGPP, 2007), in particular case-law on award criteria, whichclarifies the possibilities for the CAs to meet the needs of thepublic concerned, provided that such criteria are linked to thesubject-matter of the contract, guarantee that tenders areassessed in conditions of effective competition, do not confer anunrestricted freedom of choice on the CA, are expressly men-tioned and comply with the fundamental principles mentionedearlier.

ll rights reserved.

, 3.o Dto., 2805-140 Almada,

eus), antunes.ferreira@cised.

On the one hand, ‘‘in order to guarantee equal treatment, thecriteria for the award of the contract should enable tenders to becompared and assessed objectively’’; on the other hand, ‘‘it isappropriate to lay down an obligation to ensure the necessarytransparency to enable all tenderers to be reasonably informed ofthe criteria and arrangements which will be applied to identifythe most economically advantageous tender. It is therefore theresponsibility of CAs to indicate the criteria for the award of thecontract and the relative weighting given to each of those criteriain sufficient time for tenderers to be aware of them whenpreparing their tenders’’ (idem). Therefore, article 53 stipulatesthat, should the award criterion be that of the most economicallyadvantageous tender, ‘‘the CA shall specify in the contract noticeor in the contract documents, the relative weighting which itgives to each of the criteria’’ (ibidem).

But do these two requirements effectively ensure by them-selves that public procurement in EU member states is compliantwith the fundamental principles mentioned earlier? In particular,do they ensure effective transparency and real openness tocompetition? Do they indeed prevent CAs from having (virtually)unrestricted freedom of choice? Or is the disclosure in advance ofthese requirements by themselves really objective or evenrelevant for tenderers when preparing their tenders?

The authors will try to answer these questions. For thatpurpose, this paper will present at first the most commonweighting procedure used by CAs and show why it is wrong.Based on that, the following Sections 3 and 5 will demonstratehow to build a tender evaluation model, which is bothmethodologically accurate (in representing CA’s values and

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215 207

preferences) and meaningful for tenderers when preparing theirtenders.

Meanwhile, it will be presented in Section 4 how existingEuropean provisions on public procurement could yet go furtherin ensuring effective transparency, competition and best value formoney tenders, as showcased by the Portuguese Public ContractsCode which transposes into force the Directive while imposingfurther requirements on tender evaluation. Concluding remarksare summarised in Section 6.

2. The most common procedure for weighting evaluationcriteria

It is almost intuitive to think that the weight of eachevaluation criterion reflects its relative importance: if criterionA is more important than criterion B, then the weight of criterionA should be greater than the weight of criterion B.

But consider, for example, the case of an open procedureregarding reparation works on a bridge where, for the sake ofsimplicity, only two criteria were deemed relevant for tenderevaluation, price (A) and time (B). When defining the weights forthese criteria, the CA usually begins by picking the criterion which itconsiders the most important one, for instance, ‘‘price is moreimportant than time’’. Based on these concepts of relative impor-tance, the CA develops an idea of what the relative weights shouldbe, that is, ‘‘the weight of price criterion should be greater than time’’.

But if the CA had to choose between a tender priced ath500 000 and a completion time of 12 months, and another onepriced at h510 000 and a completion time of 8 months, which ofthese two tenders should it choose? The tender with a smallerprice, because price is the criterion with the greater ‘‘weight’’ inthis procedure?; or the one with shorter completion time (butgreater price), since in this case it seems natural that the CAprobably prefers to pay just an extra h10 000 (2% more) to havethese works completed in just 8 months (instead of 12)?

So after all what exactly is the meaning of the relativeimportance between price and time as stated initially by theCA? It means nothing at all, as it is impossible for the CA todetermine, in a clear and objective manner, the relative weights ofthe criteria without taking into account the level of performanceoffered by each tender on each criterion, that is, the rules bywhich tenders will be technically analysed and scored on eachcriterion (see Section 3).

Returning to the previous example, the CA usually goes onquantifying weights for the criteria directly based on its concep-tion of ‘‘how much’’ more important should price be vis-�a-vistime. For instance, if the CA considers that ‘‘price is about threetimes more important than time’’, then the weights to be assignedto price and time criteria should be, respectively, 75% and 25%.

This weighting procedure follows a rationale which to mostpeople would at first glance seem logical and commonsensical.Probably for that reason, and based on author’s experience, thisweighting procedure is likely to be the most common one; however,unfortunately, it is also the most common mistake in publicprocurement procedures. As a matter of fact, definition of weightsin a way such as described in this example is completely arbitraryand inconsistent with the real preferences of the CA. See also Tavareset al. (2008), Boer et al. (2006), and Keeney (2002) for other commonmistakes in public procurement and weighting procedures.

3. The tender evaluation model

Determining which tender is the most economically advanta-geous involves the use of an aggregation method like the

weighted sum. This method is implicitly referred to in Europeancase-law on public procurement and is certainly the most used byfar (Pictet and Bollinger, 2003). Thus, a tender evaluation modelmay be represented by the following mathematical expression:

VðpÞ ¼ kAvAðpAÞþkBvBðpBÞþ . . .þkiviðpiÞ ð1Þ

where V(p) is the overall score of each tender p; ki is the weight ofeach evaluation criterion i (i¼A, B, y); vi(pi) is the partial score ofeach tender p on criterion i; and pi is the level of performance ofeach tender p on the performance measure defined for criterion i.

Naturally, CAs should award the contract to the tender withthe highest overall score V(p), as determined by expression (1).

Let us now consider the standpoint of a tenderer willing tosubmit a tender in the procurement procedure mentioned inSection 2. If he chooses to use his own resources solely, his besttender (p1) will have a price of h400 000 and a completion time of12 months. Alternatively, should he employ subcontractors forpart of the works, his best tender (p2) would instead have a priceof h450 000 and a completion time of 9 months. In this situation,which of these alternative tenders should the tenderer submit?

Since the price criterion has a higher weight (75%) than thetime criterion (25%), should this tenderer submit the p1 tender?Would the CA be willing to pay an extra h50 000 in exchange for acompletion time which is 3 months shorter? It is impossible toknow. In other words, the disclosure in advance of just theevaluation criteria and its relative weights does not inform thetenderers in an objective or even relevant manner.

Suppose now that the CA defines the following functions forscoring each tender on price criterion: vAðpAÞ ¼ 200�0:4pA (withpA measured in thousands of euros), and on time criterion:vBðpBÞ ¼ 300�25pB (with pB measured in months). Replacing thesefunctions in expression (1), and considering the weights assignedto each criterion within this procurement procedure (75% and25%, respectively) would result in

VðpÞ ¼ 75%ð200�0:4pAÞþ25%ð300�25pBÞ ð2Þ

Using this particular tender evaluation model, the p1 tenderwould have an overall score of 30.00 points and the p2 tenderwould have an overall score of 33.75 points. Therefore, thistenderer should submit the p2 tender, that is, the one with thehigher price (despite the 75% weight of the price criterion), sinceit is the one that the CA considers the most economicallyadvantageous.

Notice that the CA might choose to define any other functionsfor scoring each tender on price and time criteria. For instance, itmight define the following alternative function for the timecriterion: vBðpBÞ ¼ 180�10pB. In this case, it is straightforward tosee that the p1 tender would now be the most economicallyadvantageous one for the CA and therefore the tenderer shoulddecide to submit p1 instead.

In short, whenever the CA publishes the tender evaluationmodel, i.e., an expression such as (2), the tenderer is able toconstruct and submit a tender which offers the best value formoney considering the CA’s ‘‘values’’ and preferences on a givencontract.

This exercise clearly shows that the provision, imposed byDirective 2004/18/EC, to publish the weights of all criteria in thecontract documents, does not in fact prevent, by itself, CAs fromhaving (virtually) unrestricted freedom of choice, nor does itprovide potential tenderers with objective or relevant informationon how they may best tailor their tenders to the CA’s ‘‘values’’ andpreferences.

If, on the other hand, the CA does publish the exact way bywhich it will evaluate each tender according to each criterion(that is, the way it will transform the tenders’ features andperformance levels into partial scores on each elementary

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215208

criterion), as well as the evaluation criteria and associatedweights required by Directive 2004/18/EC, this information wouldbe considerably more meaningful for tenderers when preparingtheir tenders, as well as to evaluation committees whenevaluating submitted tenders.

Tender Evaluation Model Tender PreparationIdentification and

structuring of evaluationcriteria Tender features

5.1.1

4. Portuguese legislation on public procurement

In light of the above, the Portuguese Public Contracts Code(Portugal, 2008), which regulates public procurement in Portugal,states that contract documents for any public procurementprocedure shall clearly indicate the complete tender evaluationmodel to be used (al. o), n.o 1, article 132.o), that is, besides thecontract award criteria and its relative weights, as required byDirective 2004/18/EC, they shall also stipulate the scoring rule bywhich tenders will be evaluated on each elementary criterion.These scoring rules shall be defined: either through a mathema-tical expression, or; based upon an ordered set of plausibleperformance levels (n.o 3, article 139.o). Also, it shall not dependon the features submitted in others tenders (n.o 4, article 139.o), aswould be the case with relative scoring functions based uponeither the lowest or the average price (Chen, 2008; Dini et al.,2006).

This law further states that the overall score of each tender isthe sum of all partial scores on each elementary criterionmultiplied by their relative weights (n.o 2, article 139.o), andtender evaluation is carried out by an evaluation committee whoshall assign partial scores to tenders on each elementary criterionby applying the appropriate mathematical expression or bymaking a comparison between the tenders features and thepublished ordered set of plausible performance levels (n.o 5,article 139.o).

These provisions were based on an initial draft prepared by theauthors in March 2006 and submitted to a peer review by otherexperts and specialists comprising the working group appointedby the Portuguese Government to prepare the Code. Amendmentsintroduced by this group consisted mainly in two suppressions,namely the option to award a contract based upon Benefit-Costratio, and the obligation to publicize at least two particularintrinsic performance levels (the ‘‘wished’’ and ‘‘required’’performance levels) on each elementary criterion.

The text was then presented to several public hearings, namelyin May 2006 (MNE, 2006), September 2006 (TContas, 2006) andJune 2007 (Autarnet, 2007). Main objections to these provisionsreceived during this phase concerned (Cabral, 2007, 2008):

Definition of aspects of Tenderer

a) the contract submitted to

the use of a rather technical language, concerning theawareness of a typical user;

Tendercompetition5.1.2

b)

Evaluation

its mandatory application whatever the value or complexity ofthe contract;

c)

Definition of performance

measuresTender performance

analysis

the risk of building too closed evaluation models which leaveno room to value innovative tenders by evaluation commit-tees;

5.1.2 5.2.1

d)

Definition of scoringfunctions Tender partial evaluation

Definition of weights Tender overall evaluation

Evaluation CommitteeContracting Authority

5.1.4

5.1.3 5.2.2

5.2.3

Fig. 1. Building blocks of the tender evaluation methodology.

the loss of power from Portuguese CAs (PCAs) to tenderers.

The first objection was addressed by the working group byreplacing the existing provisions by simpler and clearer ones, e.g.replacing mathematical expression (1) by its semantic form (n.o 2,article 139.o), or simply suppressing some technical terms like‘‘value tree’’, ‘‘plausible impact levels’’, or ‘‘monotonically increas-ing/decreasing’’. In our opinion, we think this sometimes led tonon-ideal compromise solutions.

The remaining objections were not considered really signifi-cant for different reasons. Firstly, one must keep in mind that themandatory application of the provisions is restricted to contracts

above the EU-Directives thresholds. Also, the innovative nature ofa tender solution can be openly evaluated considering either an‘‘innovation’’ criterion or verbal descriptions on various perfor-mance levels. Finally, although some power may actually betransferred from PCAs to tenderers, one must not forget that PCAsalso enforce its real preferences vis-�a-vis the evaluation commit-tee and tenderers, and that power is first and foremost transferredto the public interest through the achievement of better value formoney tenders (see also Section 6).

Let us now examine in detail this tender evaluation model andthe process by which PCAs will be able to represent their realpreferences.

5. Tender evaluation methodology

The overall process of setting up a tender evaluation modeland subsequently evaluating the submitted tenders, as requiredby current public procurement legislation in Portugal, is depictedin Fig. 1. The methodological building blocks of this process, withan emphasis on the definition of the tender evaluation model, willnow be presented in detail.

5.1. Setting up a tender evaluation model

5.1.1. Identification and structuring of evaluation criteria

First of all, it is necessary to identify and structure the criteriato be used on the tender evaluation model. This task isfundamental, as all criteria shall be published together with thecontract documents and cannot be further altered or disaggre-gated afterwards. It involves the characterization of the decisioncontext for each procurement procedure by determining namelythe object of the contract, the available finite public resources(like time, money and effort) and other constraints (legal,environmental, administrative, etc.) applying to that particularprocedure.

The goal of each procurement procedure is the selection of thebest tender, and therefore, evaluation criteria must help clarify

Table 1

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215 209

and objectivate what constitutes the ‘‘best tender’’ from the CA’sstandpoint.

To begin with, all objectives to be achieved with the contractshould be listed (e.g., ‘‘to maximize the quality of the serviceprovided’’), based on a value-focused thinking approach (Keeney,1992) and the ‘‘Why Is That Important’’ test (Clemen and Reilly,2001). In this process, aspects other than objectives may beidentified, such as other concerns, restrictions or undesirableconsequences, which may be deemed worthy to be translated intoadditional objectives, such as ‘‘minimizing total cost’’, or ‘‘mini-mizing delivery time’’. See Keeney (1994) for a list of other criteriaidentification techniques.

However, not all aspects or objectives identified as relevantshould necessarily be submitted to competition. Some must bestated as fixed contract provisions (e.g., specifications, plans,drawings, designs, requirements concerning test methods, con-formity certification, or terminology) and others as minimumand/or maximum requirements (e.g., minimum quality standards,or maximum delivery time) which must be completely fulfilled byall tenders. These tender requirements are to be written down atthe outset on the contract documents and must not be confusedwith the aspects of the contract submitted to competition whoseparticular features are to be presented by each tenderer on itstender (see Section 5.1.2).

Evaluation criteria must be organized in a hierarchal structure.Criteria of higher rank represent more general objectives, such as‘‘price’’, ‘‘time’’ and ‘‘quality’’. The desegregation of each criterioninto subcriteria enables CAs to describe and make clearer themeaning of the criterion defined at the level immediately above.For example, ‘‘price’’ might entail ‘‘construction cost’’ and‘‘maintenance cost’’, to be accordingly considered as subcriteria.

Finally, for a given objective to be translated into an evaluationcriterion, a set of properties must be observed, in order to ensurethe intrinsic coherence of the tender evaluation model, as definedby expression (1), namely isolability (mutual independencepreference), non-redundancy, completeness, conciseness, opera-tionality and measurability, non-ambiguity (understandability)and consensuality (Keeney, 1992; Dogson et al., 2000).

Also, criteria should not represent too broad objectives suchas ‘‘maximizing public interest’’, for which the contractcontributes only in an indirect manner. They should representonly the contract objectives over which the tenders may, in fact,have direct influence, such as minimizing ‘‘delivery time’’ or‘‘cost’’.

Example of a qualitative performance measure for a work plan quality criterion.

Level Description

L1 Only generic tasks are presented; without clear durations.

L2 Only generic tasks are presented; durations are clearly assigned and; an

overall completion time of no more than 12 months is proposed.

L3 Most tasks are presented, divided by types of work; durations are

clearly assigned and consistent with allocated resources; principal task

relationships are presented and; an overall completion time of no more

than 10 months is proposed.

L4 Almost all tasks are presented, divided by types of work; durations are

clearly assigned and consistent with allocated resources; all task

relationships are presented and; an overall completion time of no more

than 9 months is proposed.

Table 2Example of an ill-defined performance measure.

Level Description

L1 Excellent

L2 Sufficient

L3 Insufficient

5.1.2. Definition of performance measures and aspects of the

contract submitted do competition

It has been said that you can only get what you can measure(Kaplan and Norton, 1992). Therefore, the meaning of eachcriterion must be described and clarified, as objectively aspossible, by means of a performance measure. For example, inone particular procurement procedure, the price criterion couldmean the total cost, in other, the present value, and in yet anotherone, the ‘‘total cost of ownership’’ (Baily et al., 2004). Likewise, thecompletion time criterion could be measured, for example, eitherin calendar days, working days or weeks.

A performance measure is an ordered set of plausibleperformance levels set on a quantitative or qualitative scale(Bana e Costa, 1992; Bana e Costa et al., 2000). Order allows themeasurement of the achievement on a given criterion by eachtender, i.e., the measurement of its performance. Plausibility ofthe performance levels is mandatory because the actual tender’sperformances are not yet known at the moment a performancemeasure is defined.

A performance measure may either be defined as a:

�

quantitative scale (numbers only), such as delivery time indays for a time criterion, or total cost in euros for a pricecriterion; � qualitative scale (verbal or even pictorial descriptions), such as

the four performance levels defined in Table 1 for a work planquality criterion.

Notice that a qualitative performance measure does not implythat it would necessarily be subjective, as would be the case withthe simple three-level scale depicted in Table 2. Indeed, aperformance measure should be as clear and objective aspossible, even if it will not always be possible to eliminate allsubjectivity. Also of relevance, it must always be defined by atleast two widely distinct performance levels (‘‘anchors’’).

There are three types of performance measures (Keeney,1992):

a)

natural: whenever its performance levels directly reflect theeffects, consequences or ends to be evaluated (e.g. a pricecriterion measured by total cost in euros);

b)

proxy: whenever its performance levels mostly reflect causesor means of which to be evaluated (e.g. a technical qualitycriterion measured by means of a warranty in years);

c)

constructed: whenever its performance levels are definedthrough a holistic combination of plausible tenders features tobe evaluated (e.g. a work plan quality criterion measured bythe qualitative performance measure presented in Table 1).

It is preferable, if possible, to assign a natural performancemeasure to each criterion, as this type allows for the most directlyand objectively defined performance levels. Since these propertiesmake criteria more intelligible and less ambiguous, the tenderevaluation model will be more easily accepted (less controversial)and effective, thus making the award decision easier to sustain, tounderstand and to conform with the requirement for non-arbitrariness in tender evaluation procedures (Bana e Costaet al., 2000).

If the definition of a natural performance measure isnot feasible, it may be conceived through an indirect wayusing: either a proxy performance measure based on a pertinentobjective which is considered a mean to achieve the objective-end

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215210

to be evaluated, or; a constructed one based on a combination ofthe preference dependent objectives to be evaluated (see Bana eCosta et al. (2008) for a systematic procedure for developingconstructed performance measures).

Whatever the chosen performance measure, it must bemeasurable, and therefore it is essential that the contractdocuments unequivocally and objectively define which aspectsare to be submitted to competition (i.e., which contents must beincluded on the tender documents) and how this informationshould be presented (i.e., what is the standardized model ortemplate to be observed) in order to ensure the necessarycomparability between the tenders and the performance levelsset for each performance measure.

Sometimes, one aspect submitted to competition is by itself aperformance measure, for instance, when the contract documentsask directly for the submission of the total cost (euros) or thecompletion time (days). On other instances, a performancemeasure may depend on the transformation of one or severalaspects of the contract submitted to competition, e.g., aperformance measure like the present value of bids, for the pricecriterion, is sometimes a function of two aspects submitted tocompetition (e.g., unit cost and delivery schedule). Notice alsothat one specific aspect submitted to competition, like deliveryschedule, may influence the performance measures for distinctcriteria, such as price and completion time.

Carrying out the process of establishing performance measuresmay sometimes lead to new insights and uncovering hiddencriteria (Keeney, 1992), in which cases, the original criteria haveto be redefined. For example, when operationalising a criterion,which evaluates methodology quality, it may be deemedadvantageous to disaggregate it in subcriteria, in order to makethe performance analysis of distinct aspects of that methodologyfeasible. Furthermore, it may other times lead to suppressions,changes or additions to the aspects of the contract submitted tocompetition.

5.1.3. Definition of scoring functions

Scoring functions allow the transformation of the performancelevels set on each performance measure, and defined in variousunits (euros, days, verbal descriptions), into partial scores set on anumeric scale representing local attractiveness (‘‘preference’’ or‘‘value’’) between each performance level on a given criterion(Belton and Stewart, 2001). A scoring function vi(pi) makes thecorrespondence between each performance level, set on a givenperformance measure (pi), and a unique score. Scoring functionsare used both to:

a)

represent the differences in attractiveness between the variousperformance levels set for a given performance measure(a task for PCAs);

b)

operationalise tender partial evaluation on each criterion(a task for the evaluation committee)—see Section 5.2.2.

The more attractive a performance level is, the higher should itsscore be. If L1 performance level corresponds to 10 points, L2 levelcorresponds to 15 points and L3 level corresponds to 20 points,then L3 performance level is the most attractive, L2 the secondmost attractive and L1 the least attractive.

The score difference between two performance levels repre-sents the strength (or intensity) of the attractiveness differencebetween these two performance levels. For example, being thedifference between the scores of the L1 and L2 performance levelsequal to 5 points and the score difference between L2 and L3 also5 points, this means that these two ‘‘value’’ differences areequivalent (or equally attractive) for the CA, that is, it considers

that going from L1 to L2 is an equivalent improvement as goingfrom L2 to L3.

These (partial) scores are therefore numerical values represent-ing the relative attractiveness of each performance level, and also ameasure of its effectiveness in achieving the objectives of the CA.

Scoring functions may be increasing or decreasing. In increas-ing functions, the greater the performance level, the higher thescore (for instance, the greater the warranty period offered,the higher the score). Contrariwise, in decreasing functions, thegreater the performance level, the lower the score (for instance,the higher the total cost to be paid, the lower the score).

Scoring functions may also be absolute or relative. Relativescoring functions define the score of a tender by comparing itsperformance with the ones of other tenders (e.g. the averageprice, the best quality, etc.), which imply that assigning a tender’sscore is impossible without having knowledge of all other tenders.As a result, the accountability for the definition of the relativeattractiveness of each performance level is transferred from theCA to tenderers (the market). Furthermore, it allows for theoverall ranking of two tenders to be dependent on a third tender(Chen, 2008), giving tenderers the possibility of manipulationthrough removal of tenders, extreme ‘‘fake’’ tenders, and so on(Boer et al., 2006). Therefore, as referred in Section 4, relativescoring functions should never be used.

The procedures for defining scoring functions reflecting theproperties presented previously will normally differ whether theyare to be based in: either natural or proxy performance measuresor constructed ones. Scoring functions for the latter are usuallydefined using continuous or discrete mathematical expressions,while for the former they are usually defined based on directrating of scores to each predefined performance level.

The literature describes various techniques for the definition ofa scoring function, among which:

�

the bisection method (Goodwin and Wright, 1997); � direct rating (von Winterfeldt and Edwards, 1986); � MACBETH (Bana e Costa and Vansnick, 1994); � MIDVAL (Tavares, 2008).

Scoring functions for continuous performance measures

The ‘‘bisection’’ method will be presented here as anenlightening example of the construction of a scoring function.This method is applied only to continuous performance measures,therefore resulting in continuous scoring functions.

First of all, it is necessary to choose a particular performance levelconsidered rather attractive, although not necessarily the mostattractive one, for example, a price of h100 000. This level is assigneda certain score, for example, 100 points. That is, it is defined thatv(h100 000)¼100. Next, another performance level is chosen, thistime one considered not much attractive, but not necessarily theleast attractive one, for example, a price of h300 000. This level isassigned a score lower than the previous one, for example, 0 points.Thus, v(h300 000)¼0. Any other scores might have been assigned,provided that the score assigned to the most attractive performancelevel was higher than the score of the least attractive one.

Afterwards, the performance level placed exactly midwaybetween the two performance levels previously chosen isdetermined, in the example, a price of h200 000. The CAsrepresentatives are then asked whether reducing the price fromh300 000 to h200 000 is considered equivalent (i.e., equallyattractive) to reducing from h200 000 to h100 000, and

a)

In case of an affirmative answer, the score to be assigned tothat performance level will be 50 points and the scoringfunction may come to be considered linear;

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215 211

b)

Otherwise, the score to be assigned to that performance levelwill be different from 50 points and the scoring function is tobe considered non-linear.

In the first case, the scoring function is only to be consideredlinear after confirmation that the repetition of the previouslydescribed process for a sufficiently broad set of performancelevels will result in the assignment of scores which are directlyproportional to its performance levels. After this validation, thislinear scoring function can be analytically defined by using thegeneral mathematical expression of a linear function intersectingtwo reference points (p1, v(p1)) and (p2, v(p2)), given by

vðpÞ�vðp1Þ

p�p1¼

vðp2Þ�vðp1Þ

p2�p13vðpÞ ¼

vðp2Þ�vðp1Þ

p2�p1

� �ðp�p1Þþvðp1Þ

ð3Þ

In the example (see Fig. 2), the linear scoring functionintersects the points (h300 000,0) and (h100 000,100), resulting,by replacing expression (3), in

vðpÞ ¼100�0

h100000�h300000

� �ðp�h300000Þþ03vðpÞ ¼ 150�

p

2000

where p is the level of performance of each tender on thecriterion; and v(p) is the partial score to assign for that level ofperformance on the criterion.

In the case of a non-linear function, the CA would consider, inthe example, that reducing the price from h300 000 to h200 000 isnot equivalent to reducing it from h200 000 to h100 000. If ,forexample, there were only h250 000 of own resources available forfinancing the contract (i.e., without borrowing), the CA mightconsider that reducing the price from h300 000 to h200 000 ispreferable (more attractive) than reducing it from h200 000 toh100 000. This preference implies that the score to be attributedto the performance level h200 000 should be greater than 50points, since v(h200000)�v(h300000)4v(h100000)�v(h200000)3v(h200000)�04100�v(h200000)32.v(h200000)41003-

v(h200000)450.As a result, a score of 50 points should be assigned to a

performance level between h200 000 and h300 000. To determinethat price is to find the performance level which the CA considersindifferent to choose between a reduction from h300 000 to thatperformance level and a reduction from that performance level toh100 000; for example, h250 000, that is, v(h250 000)¼50.

€100000 €300000

100

0

v(p)

p

(p1,v(p1))

(p2,v(p2))

v(p)=150 - p/2000

Fig. 2. Linear scoring function.

When only three points are defined, exponential functions areusually a good representation of the CAs preferences (see Fig. 3).

In other circumstances, where the representation of the CAspreferences assume a more irregular shape, a piecewise linearfunction can be used to represent them (see Fig. 4). For thatpurpose, it is sufficient to determine the scores to be assigned to asuitably broad set of performance levels. For example, besides thethree points defined earlier, one might also determine:v(h280 000)¼25; and v(h200 000)¼75. The scores to be givento the remaining performance levels could then be obtained bydefining linear segments between pairs of successive performancelevels. The scoring function in our example could thus be

vðpÞ ¼

100�0:25p�h100000

h1000

� �if prh200000

75�0:50p�h200000

h1000

� �if h200000oprh250000

50�0:83p�h250000

h1000

� �if h250000oprh280000

25�1:25p�h280000

h1000

� �if p4h280000

8>>>>>>>>>>>>><>>>>>>>>>>>>>:

Scoring functions for qualitative or discrete performance measures

However, it is not always possible to define a scoring functionencompassing all range of feasible performance levels, namely forqualitative or discrete performance measures.

In these cases, one of the most used weighting methods is‘‘direct rating’’ (another example would be the MACBETHmethodology). Direct rating requires all performance levels tobe previously ranked by decreasing attractiveness to the CA, i.e.,from the most to the least preferred level. For instance, thequalitative performance measure regarding the technical qualityof a work plan, exemplified in Section 5.1.2, set four distinctperformance levels which could thus be ordered by decreasingrank of attractiveness: L44L34L24L1.

Thereafter, one of these performance levels, considered ratherattractive, although not necessarily the most attractive one, ischosen, for instance L4, and assigned a certain score, for instance100 points; that is, v(L4)¼100. Next, another performance level ischosen, this time one considered not much attractive, but notnecessarily the least attractive one, e.g. L1, and assigned a lowerscore; that is, v(L1)¼0. Once more, any other scores might havebeen assigned, provided that the score assigned to the most

000003€000001€

100

0

v(p)

p€250000

50

v(p)=109.574 - 2.83xe0.000012187xp

Fig. 3. Exponential scoring function.

€300

000

100

0

v(p)

p

50

€280

000

€250

000

€200

000

€100

000

25

75

Fig. 4. Piecewise linear scoring function.

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215212

attractive performance level was higher than the score of the leastattractive one.

To the remaining performance levels, scores should beassigned such that its differences represent the differences inattractiveness between the performance levels. For example, thefollowing scores could be assigned:

�

L4¼100; � L3¼80; � L2¼40; � L1¼0.

The scores in this example mean that improving the technicalquality of the work plan from L1 to L2 is considered as attractiveas improving it from L2 to L3, or improving it from L1 to L3 isconsidered twice as attractive (preferable) as improving it from L1to L2. It is crucial to understand that it is the differences betweenscores which are compared. It is incorrect to state that L3 is twiceas attractive as L2, or that L4 is twice and a half as attractive as L2,since the assignment of 0 points to L1 and 100 points to L4 wasarbitrary; any other scores might have been chosen. In the scorescale defined above, there is no absolute zero. That score scale isan interval scale (Pictet and Bollinger, 2003), and therefore, it onlyallows the relative comparison between score intervals (differ-ences).

5.1.4. Definition of weights

First of all, it should be noticed that the ki weights for thevarious criteria are nothing more than coefficients which enablethe transformation of the partial (local) scores on the variouscriteria into an overall score (Belton and Stewart, 2001). Forexample, a weight of 75% set for criterion A will transform apartial score of 100 points, assigned to a tender on criterion A, intoan overall score of 75 points.

On the other hand, the tender evaluation model defined inexpression (1) is a compensation one, meaning it is assumed thata bad performance of a tender on a given criterion may always be‘‘compensated’’ by a good performance on other criteria, and vice-versa. Notice that expression (1) implies that the ki weightsrepresent replacement ratios (trade-offs) between the partialscores of the various criteria. These trade-offs enable one to

determine how much the CA is willing to compensate a scorereduction on a given criterion by the improvement on anotherone. Therefore, weights materialize the notion of reciprocalcompensation between criteria.

For example, to define a 75% weight for the price criterion (A)and a 25% weight for the time criterion (B) mean that, accordingto expression (1), losing 10 partial points on criterion A(75%� �10¼�7.5 overall points) is equivalent to gaining 30partial points on criterion B (25%� +30¼ +7.5 overall points).

As weights represent trade-offs between partial scores, theirdefinition will necessarily have to take into account the way thosescores were determined, that is, the performance levels set foreach criterion.

Non-compliance with this precondition is considered ‘‘themost common critical mistake’’ (Keeney, 1992) in weightingprocedures, resulting in completely arbitrary weights, wronglyinterpreted as conveying an intrinsic notion of the relativeimportance of the criteria (see Section 2), instead of reflectingthe CAs real preferences. Weights must be computed based ontrade-offs between criteria, established by the CA, not be based ona meaningless measure of relative importance.

For that reason, technically sound weighting procedurescompute weights by placing specific questions to the CAsrepresentatives, requiring from them a comparison (trade-off)between the distinct scoring units set on the various performancemeasure scales of the criteria. Obviously, one will not easily reachdirect conclusions such as, for example, reducing 10 partial pointson price criterion A is equivalent to increasing 30 partial points ontime criterion B. Therefore, these weighting procedures require acomparison between the levels of performance of fictitious (butplausible) reference tenders, constructed on basis of two distinctperformance levels on each one of the criteria, such as the bestand worst performance levels, or any other pair of sufficientlydistinct performance levels, for example, a ‘‘good’’ (unquestion-ably attractive) performance level and a ‘‘neutral’’ (neitherattractive nor unattractive) performance level. That implies thatthe CA must define from the beginning, for instance, how muchmore is it willing to pay (in terms of the price criterion A) for ashorter delivery time (criterion B). The answer to that sort ofquestions will determine the trade-offs between the criteria andultimately its weights.

From the scope of technically sound weighting procedures, thefollowing are highlighted:

�

trade-off procedure (Keeney and Raiffa, 1976); � swing weighting (von Winterfeldt and Edwards, 1986); � MACBETH (Bana e Costa and Vansnick, 1997, 1999); � Silent negotiation (Pictet and Bollinger, 2008).

The trade-off procedure is quite enlightening on the substantivemeaning of weights and therefore it will be illustrated here. Inorder to determine the weights for two criteria A and B, the CAwill only have to identify two fictitious tenders p1 and p2 which itconsiders equivalent. These being equally attractive, then theiroverall scores must be equal: V(p1)¼V(p2). Therefore, byreplacement in expression (1), an equation with two unknownskA and kB may be found, where kA � vAðp1AÞþkB � vBðp1BÞ ¼

kA � vAðp2AÞþkB � vBðp2BÞ.By also setting that the weights must total 1, one gets a second

equation, where: kA+kB¼1. By solving this linear equation system(two linear equations and two unknowns), it is possible tocompute the weights corresponding to given ‘‘value’’ judgements.

Let us go back to the A and B criteria mentioned earlier in thissection representing price and time, respectively. The priceperformance measure is defined in euros and the time perfor-

Table 4Two equivalent fictitious tenders p1 and p3.

Tender Price (A) Time (B)

p1 h20 000 50 days

p3 h22 500 30 days

€20000

€30000

p1

p330 days

50 days

100

0

€22500

100

75

pv (p ) p v (p )

Fig. 6. Trade-off between criteria A and B.

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215 213

mance measure in days. Suppose that the scoring function for theprice criterion was defined by the following mathematicalexpression: vAðpAÞ ¼ 300�0:01pA. The scoring function for thetime criterion was: vBðpBÞ ¼ 250�5pB. The CA should begin bydefining two fictitious tenders (as many as the number ofelementary criteria involved), such as each one of these has arather attractive performance level (although not necessarily thebest possible one) according to one of the criteria and a notparticularly attractive performance level (although not necessa-rily the worst possible one) on the remaining criteria. Forexample, see Table 3 and Fig. 5.

Considering these fictitious tenders, the CA should then orderthem by decreasing rank of overall preference (attractiveness). Inother words, if the CA could choose only one of these fictitioustenders, which one would it consider to be the most attractive?Suppose it would choose p1, so: V(p1)4V(p2). In that case, inorder to find two equivalent fictitious tenders, it would benecessary either to worsen the p1 tender or to improve upon p2.

In order to achieve that, the CA must select only one of thecriteria and adjust the level of performance from one of thetenders regarding that criterion, until it considers the changedtender equivalent to the other one. For example, if it chooses toadjust the price criterion, the CA might either increase the price ofp1 or decrease the price of p2, until both tenders are consideredequivalent. Suppose now that, after some analysis and discussion,the CA decides that the two fictitious tenders presented in Table 4(see also Fig. 6) would be equivalent, that is, V(p1)¼V(p3).

By using expression (1), one would now have kAvAðp1AÞþ

kBvBðp1BÞ ¼ kAvAðp3AÞþkBvBðp3BÞ, and by replacing the scoringfunctions, kA(300�0.01� p1A)+kB(250�5� p1B)¼kA(300�0.01�p3A)+kB(250�5� p3B)3kA�100+kB�0¼kA�75+kB�1003kA¼

4kB.In order to make the weights add up a total of 100%, one would

add another equation into the system: kA+kB¼1. Solving thisequation system would follow that kA¼80% and kB¼20%.

In this case, the trade-off defined by the CA means that itwould be willing to spend an extra h2500 (in relation to theh20 000) in order to reduce the delivery time by 20 days (from theinitial 50 days).

This completes the definition of the building blocks of a tenderevaluation model as defined by expression (1) and as required bycurrent public procurement legislation in Portugal.

Table 3Definition of the two fictitious tenders p1 and p2.

Tender Price (A) Time (B)

p1 h20 000 50 days

p2 h30 000 30 days

€20000

€30000

p1

p2

100

0

30 days

50 days

100

0

pv (p ) p v (p )

Fig. 5. Comparing p1 and p2.

5.2. Evaluating the submitted tenders

5.2.1. Tender performance analysis

Later, once tenders are submitted, all the specific features ofeach tender presented for each one of the aspects of the contractsubmitted to competition shall be analysed by an evaluationcommittee. Tender performance analysis on each elementarycriterion consists of using a scale (‘‘yardstick’’) of plausibleperformance levels to measure the specific features presentedby the various tenders (as defined in Section 5.1.2) and assigningthem a particular level of performance. This task is essentially anobjective one, based mainly on factual information, and thereforemay (and sometimes must) be performed solely by technicalexperts without loss of discretionary power by the evaluationcommittee.

5.2.2. Tender partial evaluation

Tender evaluation is itself a rather distinct task from tenderperformance analysis. It must be carried out exclusively by theevaluation committee, after tender performance analysis isconcluded, and is operationalised through predefined scoringfunctions on each elementary criterion (as defined in Section5.1.3).

The partial score of each tender on each elementary criterionshall be assigned:

�

by applying the appropriate mathematical expression whichoperationalises the criterion, or; � by making a comparison between its performance and the

published ordered set of plausible performance levels set forthe performance measure which operationalises the criterion,resulting either in one of the scores predefined for thoseperformance levels, or any other score in between, or evenanother one beyond the maximum and minimum scorespredefined for the performance measure (i.e., that is an openscale). This means that, unless stated otherwise, during tenderevaluation, the (partial) scores to be assigned to each tender,on each elementary criterion, are not limited to the scoresscale set for its performance measure. Obviously, the scoreassigned to a tender should reflect the ‘‘value’’ judgementselicited by the evaluation committee regarding the differencesin attractiveness between its level of performance and the

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215214

various performance levels set for the performance measurewhich operationalises the criterion.

5.2.3. Tender overall evaluation

The overall score of each submitted tender must be computedby summing all partial scores on each elementary criterion (asdefined in Section 5.2.2) multiplied by its weights (as defined inSection 5.1.4).

6. Conclusions

The publication of all contract award evaluation criteria and inparticular its weights at the beginning of public procurementprocedures, as currently required by Directive 2004/18/EC, ismeaningless for tenderers when preparing their tenders (aspresented in Section 3).

Furthermore, as discussed in this paper, that provision impliesthat, in order to enable the definition of accurate weights in ameaningful and technically sound manner, the scoring rules foreach one of the evaluation criteria shall be defined beforehand. Inthis context, it must be referred that case-law on publicprocurement in Portugal (and other EU member states) has longacknowledged the obligation for the definition of the scoring rulesbefore the tenders are known (albeit without imposing the needfor its publication before).

Based on the literature (Bana e Costa et al., 2002, 2008; Csaba,2006; Pictet and Bollinger, 2003; Tavares, 2008; Dini et al., 2006),this paper has outlined the feasibility of establishing, before theactual tenders features are known, a comprehensive, clear,objective and accurate tender evaluation model consistent withthe ‘‘values’’ and real preferences of a CA (see Section 5). Thisevidence along with the authors’ long and successful experiencein supporting Portuguese public bodies, with the application oftender evaluation models within their procurement procedures,culminated in their contribution in the preparation of the text ofthe Public Contracts Code which currently regulates publicprocurement in Portugal. Besides the evaluation criteria and itsrelative weights, this Code further requires that PCAs shallpublish, at the beginning of public procurement procedures, thecomplete tender evaluation model by which tenders must beevaluated (see Section 4).

In the same way, the UNCITRAL Model Law on Procurement ofGoods, Construction and Services (www.uncitral.org) which iscurrently being revised will incorporate similar provisions. Like-wise the literature on public procurement (Asker and Cantillon,2008, 2009) has already stressed the advantages of announcingthe CAs real preferences regarding performances and trade-offs.When these are revealed to the market in advance, a tenderer isable to better define his tendering strategy by comparing hismonetary cost of improving a ‘‘quality’’ point against its monetaryvalue (from the CAs point of view), and therefore he can optimallyallocate his budget among all aspects of the contract submitted tocompetition (Dini et al., 2006). Moreover, less ambiguity reducesthe premium risk which comes with uncertainty, and thereforethe tenders’ price. In the end, all players (PCAs and tenderers)have the opportunity to create better ‘‘value for money’’ fromtheir point of view. In other words, by exploring potentialsynergies and joint values, both sides stand to gain through thecreation of a ‘‘larger pie’’.

More transparency through a ‘‘Full, Open and TruthfulExchange’’ of values and preferences (Raiffa et al., 2007) not onlyincreases public accountability but also its level of interest andensures equal opportunities to compete for, promoting anenvironment of genuine market competition.

The obligation for a PCA to define and publish the tenderevaluation model, before the tenders are known, creates anopportunity to improve communication, awareness (know-how),and the building of consensus within the PCA organisationtowards a better understanding of the contract needs, justificationfor the award decision and, one hopes, satisfaction with theexecution. Simply stated, with a better understanding of wherethe PCA wishes to go, more tenderers can make better tenders tohelp the PCA get there. Moreover, not only tenders can be mademore effective, tenderers can make it with greater economy andefficiency.

Also of relevance, tender evaluation, by the evaluationcommittee, can also be made with greater economy, efficiencyand effectiveness, by means of the improved comparability andcoherence of the tenders as well as the greater ‘‘simplicity’’ andeasiness of evaluation. Furthermore, discretionary behaviour ofthe evaluation committee is duly restricted to the PCAs prefer-ences (the actual ‘‘decision-maker’’), therefore minimizing therisk of abuse. Taken as a whole, these conditions reduce thepotential risk for predictable irrationality (Ariely, 2008) andlitigancy with tenderers.

Finally, the advantages derived from implementing tenderevaluation models such as the ones presented on this paper arenot restricted neither to tender evaluation nor to public procure-ment, indeed, they have been applied successfully in bothtenderers assessment (e.g., on restricted procedures and pre-qualification systems) and private procurement.

Acknowledgements

The authors express their appreciation to the associate editorand two anonymous referees for their thorough and insightfulcomments on an earlier version of this paper. The first author alsowould like to thank Antonio Mateus for his assistance in writingand editing this paper.

References

Ariely, D., 2008. Predictably Irrational: the Hidden Forces that Shape our Decisions.Harper Collins.

Asker, J., Cantillon, E., 2008. Properties of scoring auctions. Rand Journal ofEconomics 39 (1), 69–85.

Asker, J., Cantillon, E., 2009. Procurement when price and quality matterforthcoming in RAND. Journal of Economics (available online in /http://pages.stern.nyu.edu/� jaskerS).

Autarnet, 2007. Projecto de Codigo dos Contratos Publicos—Junho de 2007/http://www.autarnet.com/files.php?file=ccp20070703.pdfS, last visited in30th January 2010.

Bana e Costa, C.A., 1992. Structuration, Construction et Exploitation d’un Mod�eleMulticrit�ere d’Aide �a la Decision. Technical University of Lisbon, Ph.D. Thesis.

Bana e Costa, C.A., Vansnick, J.C., 1994. MACBETH: an interactive path towards theconstruction of cardinal value functions. International Transactions inOperational Research 1, 489–500.

Bana e Costa, C.A., Vansnick, J.C., 1997. Applications of the MACBETH approach inthe framework of an additive aggregation model. Journal of Multi-CriteriaDecision Analysis 6 (2), 107–114.

Bana e Costa, C.A., Vansnick, J.C., 1999. The MACBETH approach: basic ideas,software and an application. In: Meskens, M., Roubens, M. (Eds.), Advances inDecision Analysis. Kluwer Academic Publishers, Dordrecht, pp. 131–157.

Bana e Costa, C.A., Ferreira, J.A., Correa, E.C., 2000. Metodologia multicriterio deapoio �a avaliac- ~ao de propostas em concursos publicos. In: Antunes, C.H.,Tavares, L.V. (Eds.), Casos de Aplicac- ~ao da Investigac- ~ao Operacional. McGraw-Hill, Amadora, pp. 336–363.

Bana e Costa, C.A., Correa, E.C., De Corte, J.M., Vansnick, J.C., 2002. Facilitating bidevaluation in public call for tenders: a socio-technical approach. Omega 30 (3),227–242.

Bana e Costa, C.A., Lourenc-o, J.C., Chagas, M.P., Bana e Costa, J.C., 2008.Development of reusable bid evaluation models for the Portuguese electrictransmission company. Decision Analysis 5 (1), 22–42.

Baily, P., Farmer, D., Jessop, D., Jones, D., 2004. Purchasing Principles andManagement 9th Edition Financial Times Prentice Hall.

Belton, V., Stewart, T.J., 2001. Multiple Criteria Decision Analysis: an IntegratedApproach. Springer.

R. Mateus et al. / Journal of Purchasing & Supply Management 16 (2010) 206–215 215

Boer, L. de, Linthorst, M., Schotanus, F., Telgen, J., 2006. An analysis of somemistakes, miracles and myths in supplier selection. In: Proceedings of the 15thIPSERA Conference, San Diego.

Cabral, M.A., 2007. Procedimentos e Instrumentos Classicos. Cadernos de Justic-aAdministrativa—IX Seminario de Justic-a Administrativa, CEJUR (Portugal),no. 64, pp. 15–27.

Cabral, M.A., 2008. O concurso publico no Codigo dos Contratos Publicos. In: Gonc-alves, P. (Ed.), Estudos de Contratac- ~ao Publica – I. Centro de Estudos de DireitoPublico e Regulac- ~ao. Faculdade de Direito de Coimbra, Coimbra Editora,Portugal, pp. 181–227.

Chen, T.H., 2008. An economic approach to public procurement. Journal of PublicProcurement 8 (3), 407–430.

Clemen, R.T., Reilly, T., 2001. Making Hard Decisions with Decisiontools. Duxbury,Belmont.

Csaba, C., 2006. Investigating the decision making practice of public procurementprocedures. In: second International Public Procurement Conference Proceed-ings, 21–23 September 2006, Rome (Italy), pp. 869–899.

Dini, F., Pacini, R., Valletti, T., 2006. Scoring rules. In: Dimitri, N., Piga, G., Spagnolo,G. (Eds.), Handbook of Procurement. Cambridge University Press, Cambridge(UK), pp. 293–321.

Dogson, J., Spackman, M., Pearman, A., Phillips, L., 2000. DTLR Multi-CriteriaAnalysis Manual. Office of the Deputy Prime Minister, UK.

European Union, 2004. Directive 2004/18/EC of the European parliament and ofthe council of 31 march 2004 on the coordination of procedures for the awardof public works contracts, public supply contracts and public service contracts.Official Journal, L 134, 30.04.2004, pp. 114–240.

Goodwin, P., Wright, G., 1997. Decision Analysis for Management Judgement. JohnWiley & Sons, Chichester.

Kaplan, R.S., Norton, D.P., 1992. The balanced scorecard: measures that driveperformance. Harvard Business Review 70 (1), 71–80.

Keeney, R.L., 1992. Value-Focused Thinking: a Path to Creative Decisionmaking.Harvard University Press, Cambridge.

Keeney, R.L., 1994. Creativity in decision making with value-focused thinking.Sloan Management Review, Summer 1994 (35), 33–41.

Keeney, R.L., 2002. Common mistakes in making value trade-offs. OperationsResearch 50 (6), 935–945.

Keeney, R.L., Raiffa, H., 1976. Decisions with Multiple Objectives: Preferences andValue Tradeoffs.. John Wiley & Sons.

MNE, 2006. Ministerio dos Negocios Estrangeiros (Portuguese Ministry of ForeignAffairs). Anteprojecto de Codigo dos Contratos Publicos—Maio de 2006/http://www.mne.gov.pt/NR/rdonlyres/791BB023-63E2-4DE1-9AF3-DF07C8AF7F9E/0/Proj_Codigo_Contratos_Publicos.pdfS, last visited in 30th January2010.

Pictet, J., Bollinger, D., 2003. Adjuger un Marche au Mieux-Disant: AnalyseMulticrit�ere, Pratique et Droit des Marches Publics. Presses Polytechniques etUniversitaires Romandes Collection Diriger l’entreprise, Lausanne.

Pictet, J., Bollinger, D., 2008. Extended use of the cards procedure as a simpleelicitation technique for MAVT: application to public procurement in Switzer-land. European Journal of Operational Research 185 (3), 1300–1307.

Portugal, 2008. Decreto-lei no. 18/2008 de 29 de Janeiro que aprova o codigodos contratos publicos. Diario da Republica, l.a Serie, 20, 29.01.2008,pp. 753–852.

Raiffa, H., Richardson, J., Metcalfe, D., 2007. Negotiation Analysis: The Science andArt of Collaborative Decision Making, Cambridge, MA.. Harvard UniversityPress.

Tavares, L.V., 2008. Gest~ao das Aquisic- ~oes Publicas: Guia de Aplicac- ~ao do Codigodos Contratos Publicos –Decreto-Lei 18/2008– Empreitadas, Bens e Servic-os.OPET – Observatorio de Prospectiva da Engenharia e da Tecnologia, Lisbon.

Tavares, L.V., Coelho, J.S., Grac-a, P.M., 2008. O Modelo e o Software SIAP 2008: Paraa Avaliac- ~ao de Propostas e Candidaturas segundo o Codigo dos ContratosPublicos (DL 18/2008). OPET—Observatorio de Prospectiva da Engenharia e daTecnologia, Lisbon.

TContas, 2006. Tribunal de Contas (Portuguese Court of Auditors). Anteprojecto deCodigo dos Contratos Publicos—Setembro de 2006 /http://www.tcontas.pt/ContratacaoPublica/Docs/CCP_vs20060510.pdfS, last visited in 30th January2010.

Von Winterfeldt, D., Edwards, W., 1986. Decision Analysis and BehaviouralResearch, Cambridge, MA.. Cambridge University Press.

WGPP, 2007. Working Group on Public Procurement (European Court of Auditors).EU public sector procurement Directive 2004/18/EC—Guideline for auditors,June 2007 /http://eca.europa.eu/portal/pls/portal/docs/1/3088304.pdfS, lastvisited in 30th January 2010.