1

Optimizing Decisions over the Long-term in the Presence of Uncertain

ResponseEdward Kambour

2

Key Question Given a set of choices (product offerings), which

one is optimal? For each choice

Probability of success (sale) Revenue realized given a success

Revenue Management perspective Which of the choices will yield the greatest revenue

• Elicits the best customer response

3

Deterministic world Customer response to each choice is known and

constant Plug in the known success rate and revenue for each

choice and optimize

4

Random World Customer response is random with known parameter

values For example, success rate is 0.5 or realized revenue is

Poisson(7)

How to handle random response Risk neutral – Expected Response Risk averse – Expected Utility Minimax (Minimizes the worst possible outcome) Maximin (Maximizes the best possible outcome)

5

Random World Typically don’t know precisely the demand

parameters Estimate the demand

Parametric estimators of unknown parameters Robust estimators of unknown parameters Non-parametrics

• Sample quantiles, Empirical CDF, …

6

Real World Random response

In either the success rate or the realized revenue

Uncertainty in the demand distribution Based on earlier observations (expert opinion, similar

processes, etc.) The Bayesian approach quantifies this uncertainty in

a directly

7

Effect of Uncertainty What effect does the uncertainty have with

regard to making optimal decisions? Assuming risk neutrality, should decisions be based

only on expected revenue? Is there potential value in exploring choices that do

not yield the best expected revenue as of now? Quantify value in obtaining additional information, i.e.

reducing uncertainty in the demand response

8

Simple Example Choose either to place a $1 bet on a flip of a coin

coming out heads or keep your $1 You estimate that the probability of a head is 0.4 Should you play?

Expected Revenue Don’t play: 0 Do play: 1(p) – 1(1-p) = 2p – 1 = 2(0.4) –1 = -0.2

9

Simple Example Solution Depends

10

Scenario 1 There is no uncertainty

The probability of a head is exactly 0.4 Assuming that you’re risk neutral, you should not play

11

Scenario 2 You know that the coin is either a 2-headed coin

or it is a 2-tailed coin Pr[2-headed] = 0.4 Pr[2-tailed] = 0.6 If you play once

Observe head 2-headed• Continue to play and make $1 every time

Observe tail 2-tailed• Stop after losing $1

12

Scenario 2 (cont.) If you could play up to 10 times

Expected Revenue Don’t play – 0 Do play

• Observe the first flip

» If it’s heads then play the next 9 times

» If it’s tails then quit

• 0.4(10) - 0.6(1) = 3.4

13

Scenario 3 You estimate that the probability of head is 0.4

plus or minus 0.2 Playing once no longer yields perfect information

Playing any finite number of times will not yield perfect information

There is a possibility of long-term positive revenue from playing

Head probabilities greater than 0.5 are plausible Should you play at all?

If so, under what circumstances should you stop playing?

14

Scenario 3 (Bayesian Perspective) You estimate that the probability of head is 0.4

with uncertainty of 0.1 Prior mean = 0.4 Prior standard deviation = 0.1 Beta prior: = 9.2, = 13.8

Beta(9.2,13.8)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 0.2 0.4 0.6 0.8 1 1.2

p

f(p)

15

Scenario 3 (cont.) Let X be the number of heads in any set of n flips of the

coin and assume X is binomial distributed (iid Bernoulli trials)

Updated uncertainty in the head probability after the n trials Beta * = 9.2 + X, * = 13.8 + n – X Expectation: */(*+*) Variance: **/[(*+*)(*+*)(*+*+1)] Never know for sure

16

Key Points Short term expected revenue is not the only thing

to consider There is also value in obtaining additional

information about the demand distribution because this information can be used to improve long-term revenue performance How do we calculate the value of this information in

terms of future revenue? How much experimentation is optimal?

17

One Approach Evaluate any possible choice at every time

Completely reevaluate after every observation Due to the prior uncertainty, lose independence because

observations share the same unknown parameters Enumerate all possible combinations of observations and

strategies along with their associated probabilities and search for the best

Computationally intensive Complications with continuous variables

18

More Restrictive Approach Define long-term in terms of a number of observations

For example, 10 flips of the coin

Allow for a single experiment with a single conclusion For example, 3 flips of the coin

If 3 heads continue to play over the last 7 Otherwise stop

Calculate the expected net return from an experiment Expected Cost – Expected Revenue from additional observations

19

Expected Cost of Experiment Based on prior distribution on the unknown

parameters and the conditional distribution of the customer response (given the parameters) Calculate the marginal distribution for the data

Calculate the marginal expectation over the trials

Subtract the baseline revenue

( ) ( | ) ( )m x f x

E[ ] ( )X xm x dx

20

Expected Revenue from Trial Expected revenue over the rest of the long-term period

given the decision made after the trial For each choice

Win implies the highest posterior expectation of revenue Pr[win] E[return|win] For the rest of the period (n-m)

• Expected revenue given win is (n-m)E[return|win]

The expected revenue is then the weighted sum Pr[Win](n-m)E[return|win]

Subtract the baseline revenue

21

Scenario 3 (revisited) Set n = 10 Investigate an experiment with m trials Expected cost of the experiment

.2m

Assume we observed m1 heads Updated uncertainty

Beta( = 9.2 + m1, = 13.8 + m – m1)

22

Scenario 3 (cont.) Continue to play if the new expectation on the probability

of a head is greater than 0.5 * = 9.2 + m1 > 13.8 + m – m1 = *

Expected return on the experiment Pr[* > * ](10-m){2E[*/(*+*)| * > * ]-1]} +

(1- Pr[* > * ])(10-m)0 Note

Non-negative If you let n and Pr[* > * ] > 0, then the expected return for any

finite experiment

23

Scenario 3 (Details) * > *

9.2 + m1 > 13.8 + m – m1 2m1 > 4.6 + 0.5m m1 > 2.3 + 0.5m

m1 | p ~ Bin(m,p) m1~BetaBinomial(m,9.2,13.8) */(*+*) = (9.2 + m1) / (23+m)

E[*/(*+*)| * > * ] =

(9.2 +E[m1| m1 > 2.3 + 0.5m])/(23+m)

24

Scenario 3 (Cont.) If m = 5

m1 > 2.3 + 0.5m = 4.8 m1 = 5

* = 9.2 + 5 =14.2

* = 13.8 + 0 = 13.8

Expected Return Pr[* > * ](10-m){2E[*/(*+*)| * > * ]-1]}

Pr[m1 = 5 ](10-5){2(14.2/28)-1} = 0.0175(5)0.0143 = 0.0012

25

Scenario 3 (Cont.) Value of the 5 flip experiment

0.0012 – 0.2(5) = -0.9988

Value of each possible experimentm Exp Ret Exp Cost Net Ret0 0.0000 0.0000 0.00001 0.0000 -0.2000 -0.20002 0.0000 -0.4000 -0.40003 0.0000 -0.6000 -0.60004 0.0000 -0.8000 -0.80005 0.0012 -1.0000 -0.99886 0.0017 -1.2000 -1.19837 0.0023 -1.4000 -1.39778 0.0021 -1.6000 -1.59799 0.0014 -1.8000 -1.798610 0.0000 -2.0000 -2.0000

26

Scenario 4 Equivalent to Scenario 3 except

= 0.092, = 0.138 Uncertainty (prior standard deviation) = 1

Value of Experimentm Exp Ret Exp Cost Net Ret0 0.0000 0.0000 0.00001 0.0000 -0.2000 -0.20002 0.0000 -0.4000 -0.40003 0.0000 -0.6000 -0.60004 0.0000 -0.8000 -0.80005 1.6430 -1.0000 0.64306 1.3148 -1.2000 0.11487 0.9923 -1.4000 -0.40778 0.6617 -1.6000 -0.93839 0.3320 -1.8000 -1.468010 0.0000 -2.0000 -2.0000

27

General Rules of Thumb Larger values of n will lead to longer experiments

The longer the “long-term,” the greater the return on experiments

More uncertainty (larger prior standard deviations) will lead to more experimentation The less you know, the greater the return on

experiments

28

Extensions to the Simple Example Method There are multiple choices

Each with a current prior distribution for the success probability

The revenue achieved for a success may be random for each choice The parameters of the distribution are not known

29

Example Two choices

Choice 1 and Choice 2 The success probabilities are unknown for both choices The realized revenue from a success is random

• Unknown mean and variance

Long term horizon (n) is 50 Experiment

m1 trials for choice 1 m2 trials for choice 2

30

Choice 1: Current Opinion Success Rate (Beta distributed)

Expectation: 0.50 Uncertainty: 0.15

Mean revenue on a success (Normal given the variance) Expectation: 8.00 Uncertainty: 0.41

Variance in revenue on a success (Inverted Gamma) Expectation: 2.07 Uncertainty: 0.05

31

Choice 2: Current Opinion Success Rate (Beta distributed)

Expectation: 0.45 Uncertainty: 0.28

Mean revenue on a success (Normal given the variance) Expectation: 7.00 Uncertainty: 2.11

Variance in revenue on a success (Inverted Gamma) Expectation: 2.11 Uncertainty: 0.08

32

Results Search for the best experiment in terms of net

revenue over the 50 observations

33

Results The best trial consists of a sample of 6

observations 4 with choice 1 2 with choice 2

After the trial the choice with the best posterior mean is used for the remaining 44

34

Future Work Develop efficient optimization routine

Search methods (concave envelope, gradients)

Extend the types of experimentation Multiple experiments

Further investigate the asymptotics

35