Post on 04-Jan-2016
Physics Fluctuomatics (Tohoku University) 1
Physical Fluctuomatics12th Bayesian network and belief propagation in
statistical inference
Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University
kazu@smapip.is.tohoku.ac.jphttp://www.smapip.is.tohoku.ac.jp/~kazu/
Physics Fluctuomatics (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese).
Physics Fluctuomatics (Tohoku University) 3
Joint Probability and Conditional Probability
aAaAbBbBaA
aA
bBaAaAbB
PrPr,Pr
Pr
,PrPr
A
B
Conditional Probability of Event B=b when Event A=a has happened.
Probability of Event A=a }Pr{ aAJoint Probability of Events A=a and B=b
)()(Pr,Pr bBaAbBaA
Fundamental Probabilistic Theory for Image Processing
Physics Fluctuomatics (Tohoku University) 4
Fundamental Probabilistic Theory for Image Processing
a c d
dDcCbBaAbB ,,,Pr}Pr{
Marginal Probability of Event B
A B
C D
Marginalization
Physics Fluctuomatics (Tohoku University) 5
Fundamental Probabilistic Theory for Image Processing
},Pr{
},,Pr{},|Pr{
bBaA
cCbBaAbBaAcC
Causal Independence
}|Pr{},|Pr{ bBcCbBaAcC
C
A B
C
A B
)F,F(),T,F(),F,T(),T,T(),(
}F|Pr{}T|Pr{
cb
b,BAcCb,BAcC
Physics Fluctuomatics (Tohoku University) 6
Fundamental Probabilistic Theory for Image Processing
},,Pr{
},,,Pr{
},,|Pr{
cCbBaA
dDcCbBaA
cCbBaAdD
Causal Independence
)F,F(),T,F(),F,T(),T,T(),(
}FF|Pr{}TF|Pr{
}FT|Pr{},TT|Pr{
dc
c,C,BAdDc,C,BAdD
c,C,BAdDcC,BAdD
}|Pr{},,|Pr{ cCdDcCbBaAdD
D
A B C
D
A B C
Physics Fluctuomatics (Tohoku University) 7
Fundamental Probabilistic Theory for Image Processing
aAaAbBbBcC
aAaAbBbBaAcC
bBaAcBbAcCcCbBaA
PrPrPr
PrPr,Pr
,Pr,Pr,,Pr
a b
b
cCbBaA
cCbBaA
cC
cCaA
cCaA
,,Pr
,,Pr
Pr
,Pr
PrA
B
C
}|Pr{ aAbB
bBcC
bBaAcC
Pr
,Pr
Causal Independence
Physics Fluctuomatics (Tohoku University) 8
Fundamental Probabilistic Theory for Image Processing
),(),(
PrPrPr,,Pr
},{},{ cbfbaf
aAaAbBbBcCcCbBaA
CBBA
A
B
C
aAaAbBbaf
bBcCcbf
BA
CB
PrPr),(
Pr),(
},{
},{
A
B
C
Directed Graph Undirected Graph
Physics Fluctuomatics (Tohoku University) 9
Simple Example of Bayesian Networks
? TTPrTTPr WSWR
AAAA
Physics Fluctuomatics (Tohoku University) 10
Simple Example of Bayesian Networks
CC
CCSS
CCRR
RRSSWW
CCCCSS
SSCCRR
RRSSCCWW
SSCCSSCCRR
RRSSCCWW
RRSSCC
RRSSCCWW
WWRRSSCC
Pr
Pr
Pr
,Pr
PrPr
,Pr
,,Pr
,Pr,Pr
,,Pr
,,Pr
,,Pr
,,,Pr
aA
aAaA
aAaA
aAaAaA
aAaAaA
aAaAaA
aAaAaAaA
aAaAaAaAaA
aAaAaAaA
aAaAaA
aAaAaAaA
aAaAaAaA
Physics Fluctuomatics (Tohoku University) 11
Simple Example of Bayesian Networks
TPr
TT,PrTTPr
W
WRWR
A
AAAA
),(),(
PrPrPr,,Pr
},{},{ cbfbaf
aAaAbBbBcCcCbBaA
CBBA
Physics Fluctuomatics (Tohoku University) 12
Simple Example of Bayesian Networks
2781.0T,,T,PrT,TPrFT, FT,
WRRSCCWS
C R
a a
AaAAaAAA
6471.0T,,,PrTPrFT, FT, FT,
WRRSSCCW
C S R
a a a
AaAaAaAA
4581.0T,T,,PrT,TPrFT, FT,
WRCSCCWR
C S
a a
AAaAaAAA
Physics Fluctuomatics (Tohoku University) 13
Simple Example of Bayesian Networks
7079.0
6471.0
4581.0
TPr
TT,PrTTPr
W
WRWR
A
AAAA
4298.0
6471.0
2781.0
TPr
TT,PrTTPr
W
WSWS
A
AAAA
Physics Fluctuomatics (Tohoku University) 14
Simple Example of Bayesian Networks
),(),(),,(
PrPr
Pr,Pr
,,,Pr
CS}CS,{CR}CR,{RSW}RS,,{
CCCCSS
CCRRRRSSWW
WWRRSSCC
aafaafaaaf
aAaAaA
aAaAaAaAaA
aAaAaAaA
W
C
S R
W
CCCCSSCS}CS,{
CCRRCR}CR,{
RRSSWWRSW}RS,,{
PrPr),(
Pr),(
,Pr),,(
aAaAaAaaf
aAaAaaf
aAaAaAaaaf W
C
S R
W
Directed Graph
Undirected Graph
Physics Fluctuomatics (Tohoku University) 15
Belief Propagation for Bayesian Networks
Belief propagation cannot give us exact computations in Bayesian networks on cycle graphs.Applications of belief propagation to Bayesian networks on cycle graphs provide us many powerful approximate computational models and practical algorithms for probabilistic information processing.
Physics Fluctuomatics (Tohoku University) 16
Simple Example of Bayesian Networks
22111133
22442255
443366
6677
665588
882211
PrPrPr
PrPr
,Pr
XPr
,Pr
,,,Pr
xXxXxXxX
xXxXxXxX
xXxXxX
xXx
xXxXxX
xXxXxX
),(),(),(
),(),,(),,(x
,,,Pr
311342245225
7667543346865568
882211
xxWxxWxxW
xxWxxxWxxW
xXxXxX
Physics Fluctuomatics (Tohoku University) 17
Joint Probability of Probabilistic Model with Graphical Representation including Cycles
),(),(),(
),(),,(),,(x
,,,Pr
311342245225
7667543346865568
882211
xxWxxWxxW
xxWxxxWxxW
xXxXxX
Directed Graph
1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87 Undirected Hypergraph
Physics Fluctuomatics (Tohoku University) 18
Marginal Probability Distributions
ix
iiii
xXxXxX
xPxX
\882211 ,,,Pr
)(}Pr{
x
kji xxx
kjiijkkkjjii
xXxXxX
xxxPxXxXxX
,,\882211 ,,,Pr
),,(},,Pr{
x
ji xx
jiijjjii
xXxXxX
xxPxXxX
,\882211 ,,,Pr
),(},Pr{
x
1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87
Physics Fluctuomatics (Tohoku University) 19
Approximate Representations of Marginal Probability Distributions in terms of Messages
3 4 6)()(
)()(),,(
)()(
)()(),,(
),,(
656866676
42443133643346
656866676
42443133643346
643346
x x x xMxM
xMxMxxxW
xMxM
xMxMxxxW
xxxP
1
3
2
4
6 5
87
346W
)( 3133 xM )( 4244 xM
)( 6676 xM )( 65686 xM
1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87
Physics Fluctuomatics (Tohoku University) 20
Approximate Representations of Marginal Probability Distributions in terms of Messages
6
66766568663466
6676656866346666
x
xMxMxM
xMxMxMxP
3 4
6 5
87
)( 6676 xM
)( 65686 xM
)( 63466 xM 1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87
3 4
64334666 ,,x x
xxxPxP
Physics Fluctuomatics (Tohoku University) 21
Basic Strategies of Belief Propagations in Probabilistic Model with Graphical Representation including Cycles
1
3
2
4
6 5
87
346W
)( 3133 xM )( 4244 xM
)( 6676 xM
)( 65686 xM
3 4
6 5
87
)( 6676 xM
)( 6586 xM
)( 6346 xM
667665686
42443133
643346346
643346
,,1
,,
xMxM
xMxM
xxxWZ
xxxP
667665686
634666
66
1
xMxM
xMZ
xP
Approximate Expressions of Marginal Probabilities
1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87
Physics Fluctuomatics (Tohoku University) 22
Simultaneous Fixed Pint Equations for Belief Propagations in
Hypergraph Representations
3 4 6
3 4
)()(),,(
)()(),,(
)(42443133643346
42443133643346
63466
x x x
x x
xMxMxxxW
xMxMxxxW
xM
Belief Propagation Algorithm
6
1
3
2
4)( 3133 xM )( 4244 xM
)( 63466 xM
1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87
Physics Fluctuomatics (Tohoku University) 23
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
23
12
01
MM
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics (Tohoku University) 24
Belief Propagation for Bayesian Networks
Belief propagation can be applied to Bayesian networks also on hypergraphs as powerful approximate algorithms.
Physics Fluctuomatics (Tohoku University) 25
Numerical Experiments
4393.0)0(5607.0)1( 88 PP
4360.0)0(5640.0)1( 88 PP
Belief Propagation
Exact
8\88221188 },,,Pr{)(
x
xXxXxXxPx
3629.0)0,0( 0871.0)1,0(
0764.0)0,1( 4736.0)1,1(
5858
5858
PP
PP
3636.0)0,0( 0864.0)1,0(
0724.0)0,1( 4776.0)1,1(
5858
5858
PP
PP
85 ,\8822118558 },,,Pr{),(
xx
xXxXxXxxPx
1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87
Physics Fluctuomatics (Tohoku University) 26
Numerical Experiments
8261.04393.0
3629.0
PresentPr
Present,PresentPr
PresentPresentPr
Dyspnea
DyspneaBronchitis
DyspneaBronchitis
X
XX
XX
Belief PropagationBelief Propagation
1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87
Physics Fluctuomatics (Tohoku University) 27
Linear Response Theory
}|{ Ωx ixi
ij
PPnPnPnPjj xnxnjjj
i
Nodeat Field External respect to with Nodeat Average ofDeviation
)()(~
)()(~
)( ,,)(
xx
xx
i
ji
h
jiij
h
nP
nPmPnmP
i
)(lim
)()(),(
)(
0
mxi ihP
ZP ,exp)(~
1)(
~ xx
3
1 2
4
6 5
87
Physics Fluctuomatics (Tohoku University) 28
Numerical Experiments
0187.0
4393.0
0082.0
PresentPr
Present,PresentPr
PresentPresentPr
Dyspnea
DyspneaisTuberculos
DyspneaisTuberculos
X
XX
XX
1
3
2
4
6 5
13W
67W
24W
25W346W
568W
87
Physics Fluctuomatics (Tohoku University) 29
Summary
Bayesian Network for Probabilistic InferenceBelief Propagation for Bayesian Networks
Physics Fluctuomatics (Tohoku University) 30
Practice 11-1
Compute the exact values of the marginal probability Pr{Xi} for every nodes i(=1,2,…,8), numerically, in the Bayesian network defined by the joint probability distribution Pr{X1,X2,…,X8} as follows:
Each conditional probability table and probability table is given in Figure 3.12 and Table 3.11 in
Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009.
21
132425
43667658
821
PrPr
PrPrPr
,PrXPr,Pr
,,,Pr
XX
XXXXXX
XXXXXXX
XXX
Physics Fluctuomatics (Tohoku University) 31
Practice 11-2
Each conditional probability table and probability table is given in Figure 3.12 and Table 3.11 in
Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009.The algorithm has appeared explicitly in the above textbook.
Make a program to compute the approximate values of the marginal probability Pr{Xi} for every nodes i(=1,2,…,8) by using the belief propagation method in the Bayesian network defined by the joint probability distribution Pr{X1,X2,…,X8} as follows:
21132425
43667658
821
PrPrPrPrPr
,PrXPr,Pr
,,,Pr
XXXXXXXX
XXXXXXX
XXX