Post on 20-May-2018
Intro Utility theory Utility & Loss Functions Value of Information Summary
Decision Theory(CIV6540 - Machine Learning for Civil Engineers)
Professor: James-A. Goulet
Departement des genies civil, geologique et des mines
Polytechnique Montreal
Chapter 14 – Goulet (2018).Probabilistic Machine Learning for Civil Engineers.
Chapter 16 – Russell, S. and Norvig, P. (1995).Artificial Intelligence, A modern approach. Prentice-Hall.
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 1 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Context
Making rational decisions - Fire alarm
t
0.0
5
0.95
Pr = 0.99
Pr = 0.01
t − 1 min
0.99
0.01
1
Pr = 0.7
Pr = 0.3
t + 1 min
0.99
1
0.01
0.0
50.95
Pr = 0.1
Pr = 0.9
1
10
10
0.01
E[U] =3.7
E[U] =7.0
t + 1
Pr = 0.1
Pr = 0.9
E[U] =9
E[U] =1
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 2 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Context
Making rational decisions - Soil contamination
We have 1 m3 of soil form an industrial site. What should we do?
Pr = 0.9
Pr = 0.1
0$
100$
10K
$
100$
E[$| ] = 0$× 0.9 + 10K$× 0.1 = 1K$
E[$| ] = 100$× 0.9 + 100$× 0.1 = 100$
Optimal action
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 3 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Nomenclature
Nomenclature
A = {a1, a2, · · · , aA} A set of possible actions
x ∈ Z or ∈ R An outcome in a set of possible states
Pr(x) Probability of a state x
U(ai , x) Utility given a state x and an action ai
L(ai , x) ≡ −U(ai , x) Loss given a state x and an action ai
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 4 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Nomenclature
Soil contamination example
ai ∈ { , } ≡ {0, 1}
x ∈ { , } ≡ {0, 1}
Pr(x) = [0.9, 0.1]ᵀ
U(ai , x) = U[
, ,, ,
]≡[
0$ −10K$−100$ −100$
]
L(ai , x) = L[
, ,, ,
]≡[
0$ 10K$100$ 100$
]
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 5 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Rational decisions
Rational decisions → Expected utility maximization
The perceived benefit of an outcome xi given an action ai ismeasured by the expected utility or expected loss
E[U(a)] =∑X
i=1 U(a, xi ) · Pr(xi )
E[L(a)] =∑X
i=1 L(a, xi ) · Pr(xi )
The optimal action a∗ is the one that maximizes the expectedutility or minimizes the expected loss
a∗ = arg maxa
E[U(a)] = arg mina
E[L(a)]
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 6 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Module #9 Outline
Intro
Utility theoryUtility & LossFunctionsValue ofInformation
Topics organization
Probabilitytheory
{1 Revision probability & linear algebra
2 Probability distributions −4 −2 0 2 40
0.2
0.4 µ µ+σµ−σ
x−4 −2 0 2 40
0.5
1µ µ+σµ−σ
x
F X(x)
f X(x)
MachineLearning
Basics
0 Introduction
3 Bayesian Estimation p(A|B) =p(B|A)p(A)
p(B)
4 MCMC sampling & Newton −5 0 5−5
0
5
θ1
θ 2
−50
5
−50
50.00
0.05
0.10
θ1θ2
s = 1000
Data-drivenmethods
5 Regression [ ] cl+
Pato
gens
[ppm
]
6 Classification0 0.2 0.4 0.6 0.8 10
10
x
f X|D
(x|d
)
d=0 d=1 d=2 d=3 d=4 d=5
0 0.2 0.4 0.6 0.8 10
0.5
1
P(D
S)
S={4,5} S={2,3} S={0,1}
7 State-space model for time-series
Model drivenmethods
{8 Model Calibration −2
02
−20
2
y1y2
−2 0 2
−2
0
2
y1
y 2
DecisionMaking
{9 Decision Theory
E[U] = 7.0
E[U] = 3.7
10 AI & Sequential decision problems
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 7 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Section Outline
Utility theory2.1 Lotteries2.2 Axioms of utility theory
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 8 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Lotteries
Nomenclature for ordering preferences
A lottery: Li = [{p1, x1}, {p2, x2}, · · · , {pX, xX}]
L = [{1.0, , }{0.0, , }]L = [{0.9, , }{0.1, , }]
A decision maker
Li � Lj prefers Li over Lj
Li ∼ Lj is indifferent between Li and Lj
Li � Lj prefers Li over Lj or is indifferent
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 9 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Axioms of utility theory
Axioms of utility theory
What is defining a rational behaviour?
Orderability: Exactly one of (Li � Lj), (Lj � Li ), (Li ∼ Lj) holds
Transitivity: if (Li � Lj) ∧ (Lj � Lk), then (Li � Lk)
Continuity: if (Li � Lj � Lk), then ∃p : [{p, Li}; {1− p, Lk}] ∼ Lj
Substitutability:if (Li ∼ Lj), then [{p, Li}; {1− p, Lk}] ∼ [{p, Lj}; {1− p, Lk}]
Monotonicity:if Li � Lj , then (p ≥ q ⇔ [{p, Li}; {1− p, Lj}] � [{q, Li}; {1− q, Lj}])
Decomposability: ...no fun in gambling
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 10 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Section Outline
Utility & Loss Functions3.1 Utility3.2 Non-linear utility functions3.3 Utility and Loss functions U(v) & L(v)3.4 E[L(v(ai ,X ))] and risk aversion
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 11 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Utility
Axioms → utility
Existence of a utility function:
U(Li ) ≥ U(Lj) ⇔ Li � Lj
U(Li ) = U(Lj) ⇔ Li ∼ Lj
Expected utility of a lottery:
E[U([{p1, x1}, {p2, x2}, · · · , {pX, xX}])] =X∑
i=1
piU(xi )
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 12 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Non-linear utility functions
Do you want to take the lottery?
L = [{12 , + }, {1
2 , - }]L = [{1,+0$}]
Which lottery do you choose? Why?
E[$(L )] = 12 ×+200$ + 1
2 ×−100$ = +50$
E[$(L )] = 0$
Are you being irrational?For individuals U($) and L($) are non-linear...
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 13 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Utility and Loss functions U(v) & L(v)
Risk aversion and utility functions U(v)U(v): An utility function weight monetary value (v) as a functionof risk aversion/propension(± 1$ not the same effet if you have 1$ or 1M$)
0 0.2 0.4 0.6 0.8 1v
0
1
Util
ity,
(v)
Risk averseRisk neutralRisk seeking
People/organizationsare risk averse
U(v) = vk
k > 1 Risk seekingk = 1 Neutral0 < k < 1 Risk averse
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 14 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Utility and Loss functions U(v) & L(v)
Risk aversion and loss functions L(v)L(v): A loss function weight monetary value (v) as a function ofrisk aversion/propension
0 0.2 0.4 0.6 0.8 1v
0
1
Loss
, (
v)
Risk averseRisk neutralRisk seeking
People/organizationsare risk averse
L(v) = vk
k > 1 Risk aversek = 1 Neutral0 < k < 1 Risk seeking
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 15 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Utility and Loss functions U(v) & L(v)
Attitude toward risks
0 0.2 0.4 0.6 0.8 1v
0
1
Loss
, (
v)
Risk averseRisk neutralRisk seeking
L(v) = vk
k > 1 Risk aversek = 1 Neutral0 < k < 1 Risk seeking
A neutral attitude toward risks minimizes the expected costsover a multiple decisions
I Insurance compagnies: neutral attitude toward risksI Insured people: risk averse; they pay a premium not to be in
a risk neutral position(i.e. expected costs are higher over multiple decisions)
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 16 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Utility and Loss functions U(v) & L(v)
Expected Loss
0 0.2 0.4 0.6 0.8 1v
0
1
Loss
, (
v)
Risk averseRisk neutralRisk seeking
Value → Loss:
Value x = x =a = v( , ) v( , )a = v( , ) v( , )
→Loss x = x = E[L(v(a,X ))]
a = L(v( , )) L(v( , )) E[L(v( ,X ))]a = L(v( , )) L(v( , )) E[L(v( ,X ))]
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 17 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
E[L(v(ai , X ))] and risk aversion
E[L(v(ai ,X ))] and risk aversion (ex. discrete) [ ]
0 0.5 10
0.5
1
E[v(a
1,X
)]=
E[v(a
2,X
)]v(ai, x)
p(v|a
i,x)
Action a1Action a2
0 10
1
v(ai, x)
Loss
,L(v)
01
E[L(v(a1, X)] = E[L(v(a2, X)]
p(L(v)|ai, x)
Risk perception ¬neutral: E[L(v(a1,X ))] 6= E[L(v(a2,X ))]
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 18 / 26
[CIV ML/Decision/PCgA discrete.m]
Intro Utility theory Utility & Loss Functions Value of Information Summary
E[L(v(ai , X ))] and risk aversion
E[L(v(ai ,X ))] and risk aversion (ex. discrete) [ ]
0 0.5 10
0.5
1
E[v(a
1,X
)]=
E[v(a
2,X
)]v(ai, x)
p(v|a
i,x)
Action a1Action a2
0 10
1
v(ai, x)
Loss
,L(v)
01
E[L(v(a1, X)]
E[L(v(a2, X)]
p(L(v)|ai, x)
Risk perception ¬neutral: E[L(v(a1,X ))] 6= E[L(v(a2,X ))]
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 18 / 26
[CIV ML/Decision/PCgA discrete.m]
Intro Utility theory Utility & Loss Functions Value of Information Summary
E[L(v(ai , X ))] and risk aversion
E[L(v(ai ,X ))] and risk aversion (ex. continuous) [ ]
0 0.5 1
E[v(a
1,X
)]=
E[v(a
2,X
)]
v(ai, x)
f(v|a
i,x)
Action a1Action a2
0 10
1
v(ai, x)
Loss
,L(v)
0
E[L(v(a1, X))] = E[L(v(a2, X))]
f (L(v)|ai, x)
Risk perception ¬neutral: E[L(v(a1,X ))] 6= E[L(v(a2,X ))]
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 19 / 26
[CIV ML/Decision/PCgA continuous.m]
Intro Utility theory Utility & Loss Functions Value of Information Summary
E[L(v(ai , X ))] and risk aversion
E[L(v(ai ,X ))] and risk aversion (ex. continuous) [ ]
0 0.5 1
E[v(a
1,X
)]=
E[v(a
2,X
)]
v(ai, x)
f(v|a
i,x)
Action a1Action a2
0 10
1
v(ai, x)
Loss
,L(v)
0E[L(v(a1, X))]
E[L(v(a2, X))]
f (L(v)|ai, x)
Risk perception ¬neutral: E[L(v(a1,X ))] 6= E[L(v(a2,X ))]
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 19 / 26
[CIV ML/Decision/PCgA continuous.m]
Intro Utility theory Utility & Loss Functions Value of Information Summary
Section Outline
Value of Information4.1 Value of perfect information4.2 Value of imperfect information4.3 Exemples
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 20 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Value of perfect information
Expected utility of collecting informationIn cases where the value of a state x is imperfectly known,onepossible action is to collect information about X .
E[U(a∗)] = maxa
X∑i=1
U(a, xi ) · Pr(xi )
U(a∗, x = y) = maxa
U(a, x = y)
Because y has not been observed yet, we must consider allpossibilities Y = Xi according to their probability
E[U(a∗)] =X∑
i=1
maxa
[U(a, xi )] · Pr(xi )
Value of perfect information
VPI (y) = E[U(a∗)]− E[U(a∗)] ≥ 0
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 21 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Value of perfect information
Soil contamination exampleL(a, x) x = x =
a = 100$ 100$a = 0$ 10K$
Current expected costs conditional on actions
E[L( ,X )] = 0$× 0.9 + 10K$× 0.1 = 1K$
E[L( ,X )] = 100$× 0.9 + 100$× 0.1 = 100$ = E[L(a∗,X )]
Expected costs conditional on perfect information
E[L(a∗)] =∑X
i=1 mina
(L(a, xi )) · Pr(xi )
= 0$× 0.9︸ ︷︷ ︸y=x=
+ 100$× 0.1︸ ︷︷ ︸y=x=
= 10$
Value of perfect information
VPI (y) = E[L(a∗)]− E[L(a∗)] = 90$Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 22 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Value of perfect information
Value of information
The value of information represents how much you are willing topay for an information.
What if the information is not perfect?
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 23 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Value of imperfect information
Value of imperfect information
Discrete state case
E[U(a∗)] =X∑
i=1
X∑j=1
maxa
(U(a, yi )) · Pr(yi |xj) · Pr(xj)
Continuous state case
E[U(a∗)] =
∫ ∫maxa
(U(a, y)) · f (y |x) · f (x)dydx
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 24 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Exemples
Soil contamination example - Discrete caseL(a, x) x = x =
a = 100$ 100$a = 0$ 10K$
Pr(y = | ) = 1
Pr(y = | ) = 0.95
Current expected costs E[L(a∗,X )] = 100$
Expected costs conditional on imperfect information
E[L(a∗)] =∑X
i=1
∑Xj=1 min
a(L(a, yi )) · Pr(yi |xj) · Pr(xj)
= 0$× 0.9︸ ︷︷ ︸x=y=
+ ( 100$× 0.95︸ ︷︷ ︸y=
+ 10K$× 0.05︸ ︷︷ ︸y=
)× 0.1
︸ ︷︷ ︸x== 59.5$
Value of information
VOI (y) = E[L(a∗)]− E[L(a∗)] = 40.5$
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 25 / 26
Intro Utility theory Utility & Loss Functions Value of Information Summary
Summary
Rational Decision:
Choose the action a∗i which minimize the expected lossL(a, x) or maximizes the expected cost U(a, x)
a∗i = arg minai
E[L(ai , x)] = arg maxai
E[U(ai , x)]
L(v(a, x)) & U(v(a, x)): Subjective weight on value as afunction of the attitude toward risks(± 1$ not the same effect if you have 1$ or 1M$)
0 0.2 0.4 0.6 0.8 1v
0
1
Loss
, (
v)
Risk averseRisk neutralRisk seeking
L(v) = vk
k > 1 Risk aversek = 1 Neutral0 < k < 1 Risk seeking
Value of information:
Value you should be willing to pay for information
VOI (y) = E[U(a∗)]− E[U(a∗)] ≥ 0
Value of perfect information:
E[U(a∗)] =X∑
i=1
maxa
[U(a, xi )] · Pr(xi )
Value of imperfect information:
E[U(a∗)] =X∑
i=1
X∑j=1
maxa
(U(a, y)) · Pr(y|xj ) · Pr(xj )
Professor: J-A. Goulet Polytechnique Montreal
Decision Theory | V1.2 | Machine Learning for Civil Engineers 26 / 26