Optimal revision of uncertain estimates in project portfolio selection

Eeva Vilkkumaa, Juuso Liesiö, Ahti Salo

Department of Mathematics and Systems Analysis,

Aalto University School of Science and Technology

Contents

• Project portfolio selection• Optimizer’s curse• Revised estimates• Discussion

Project portfolio selection

• Select a subset of projects within a budget, e.g., k out of n projects with the aim of maximizing the sum of the projects’ values μi, i=1,...,n

• The values μi are generally unknown, whereby decisions about which projects to select are made based on estimates Vi about μi.

Estimates Portfolio selection Values

t

Optimizer’s curse in portfolio selection

• Assume that the estimates are unbiased

• Portfolio maximization selects, on average, overestimated projects → the value of the portfolio is less than expected based on the estimation information (optimizer’s curse; cf. Smith and Winkler, 2006):

where is the index set of the selected projects.

],[][ ˆˆ

Ki iKi i VEE

].[][ ii EVE

K̂

Optimizer’s curse in portfolio selection

• Choosing 10 projects out of 100

• Values i.i.d with

• Unbiased estimates

• The larger the estimation error variance, the harder it is to identify the best projects, and the larger the difference between the estimated and realized portfolio value

µi ~ N(0,12)

Vi = µi + εi, εi ~ N(0,σ2)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 112

14

16

18

20

22

24

26

Standard deviation of estimation error

Por

tfol

io v

alue

Estimated

True

Standard deviation of estimation error

Optimal revision of the estimates

• Estimates do not account for the uncertainties

• Use Bayesian revised estimates instead as a basis for project selection

• For instance, with µi ~ N(mi,σi2), Vi ~ N(µi,τi

2):

• The estimate V and the prior information m are weighted according to their uncertainty.

]|[ˆ VEv ii

iiiii mVv )1(ˆ 22

2

ii

ii

where

iv̂

Optimal revision of the estimates

• With revised estimates the optimizers’ curse is eliminated, that is

where is the index set of the projects selected using revised estimates

• Previous example – Choosing 10 projects out of 100

– True values i.i.d. with

– Unbiased estimates

],ˆ[][ ~~

Ki iKi i vEE

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 112

14

16

18

20

22

24

26

Standard deviation of estimation error

Por

tfolio

val

ue

Estimated

True

µi ~ N(0,12)

Vi = µi + εi, εi ~ N(0,σ2)

K~

Standard deviation of estimation error

Revised estimates and portfolio composition• In the previous example, the projects’ values were

identically distributed, and the estimation errors had equal variances

• Then, prioritization among the projects remains unchanged when the estimates are revised, because

• In general, using revised estimates may result in a different project prioritization than estimates

jijiji VVVVvv )1()1(ˆˆ

Revised estimates and portfolio composition

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Pro

ject

val

ue

Same error variances

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Pro

ject

val

ue

Different error variances

Estimate Revisedestimate

Revisedestimate

EstimateEstimate Revised estimate

Estimate Revised estimate

Same error variances Different error variances• Choosing 3 projects out of 8

• True values i.i.d. With µi ~ N(0,12)

• On the left, estimates with equal error variance for all projects

• On the right, four projects (dashed) more difficult to estimate

Vi = µi + εi, εi ~ N(0,0.52)

Vi = µi + εi, εi ~ N(0,12)

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Pro

ject

val

ue

Same error variances

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Pro

ject

val

ue

Different error variances

Estimate Revisedestimate

Revisedestimate

Estimate

Revised estimates and portfolio composition

• On the left, equal error variances → estimates are shifted towards the common prior mean (zero) in the same proportion

• On the right: the revised estimates of the ”dashed” projects are more drawn towards zero, because the estimation information is less reliable

• Selection of 3 projects leads to different portfolios depending on whether the estimates are revised or not Estimate Revised

estimateEstimate Revised

estimate

Same error variances Different error variances

Revised estimates and portfolio value• The use of revised estimates yields at

least as high overall portfolio value as the use of initial estimates, i.e.

• Example:– Selection of 10 out of 100 projects with

values µi ~ N(3,12)

– Population contains two types of projects:

– Revised estimates yield higher portfolio value for any non-trivial division between projects with small and large estimation error variances

0 10 20 30 40 50 60 70 80 90 10042

43

44

45

46

47

48

Share of projects with large error variance [%]P

ortfo

lio v

alue

][][ ˆ~

Ki iKi i EE

1) εi ~ N(0,0.12) - small error variance2) εi ~ N(0,12) - large error variance

Share of projects with large error variance [%]

Optimal

Estimates

Revised estimates

Revised estimates and correct choices

• The share of correctly selected projects increases with revised estimates in the normally distributed case, i.e.,

where K is the index set of the projects in the optimal portfolio

• In the previous example, the difference between the two portfolios is statistically significant (α=0.05), when the share of projects with large error variance is between 25-55%

}ˆ|{#}~

|{# KKiiKKii

Share of projects with large error variance [%]

Discussion

• Selection based on project prioritization resulting from estimates– The value of the portfolio will, on average, be lower than

expected– If there are differences in the projects’ estimation error

variances, too many projects with large error variance will be selected

• Suggestions for improving the selection process– Accounting for the uncertainties by using revised estimates– Sorting the projects in terms of estimation error variances by,

e.g., budget division