Kazuyuki Tanaka Graduate School of Information Sciences Tohoku University, Japan

3 March, 2003 University of Glasgow 1

Statistical-Mechanical Approach to Probabilistic Inference--- Cluster Variation Method

and Generalized Loopy Belief Propagation ---

Statistical-Mechanical Approach to Probabilistic Inference--- Cluster Variation Method

and Generalized Loopy Belief Propagation ---

Kazuyuki TanakaGraduate School of Information Sciences

Tohoku University, [email protected]

http://www.statp.is.tohoku.ac.jp/~kazu/index-e.html

Kazuyuki TanakaGraduate School of Information Sciences

Tohoku University, [email protected]

http://www.statp.is.tohoku.ac.jp/~kazu/index-e.html


Contents

1. Introduction2. Probabilistic Inference3. Cluster Variation Method4. Generalized Loopy Belief Propagation5. Linear Response6. Numerical Experiments7. Concluding Remarks


IntroductionProbabilistic Inference and Belief Propagation

Probabilistic Inference Probabilistic Model

Bayes Formula

BeliefMarginal Probability

BeliefPropagation

Probabilistic models on tree-like networks

with no loops=>Exact Results

Probabilistic models on networks

with some loops=>Good Approximation

Generalization


IntroductionStatistical Mechanics and Belief Propagation

Belief PropagationProbabilistic model with no loop

Probabilistic model with some loops(Lauritzen, Pearl)

Probabilistic model with no loop Transfer Matrix

Recursion Formula for Beliefs and Messages

Probabilistic model with some loops

Bethe/Kikuchi MethodCluster Variation Method

Transfer Matrix =

Belief Propagation


Introduction

Purpose

•Review of generalized loopy belief propagation and cluster variation method.•Calculation of correlations between any pair of nodes by combining the cluster variation method with the linear response theory


Probabilistic Inference


Probabilistic InferenceProbabilistic Inference and Probabilistic Model

)()()()(

)()()()(

)|()|(

)|()|(

),|(

),|(1

),,,,,,,(

88776655

44332211

13132424

25256767

436346

658568

87654321

xVxVxVxV

xVxVxVxV

xxVxxV

xxVxxV

xxxV

xxxVZ

xxxxxxxxP

1X

3X

2X

4X

6X 5X

8X7X

13V

67V

24V

25V346V

568V

1V 2V

3V4V

5V

8V7V

6V


Probabilistic InferenceProbabilistic Inference and Probabilistic Model

)()(

),(),(),(

)()()(

),(),,(),,(1

),,,,,,,(

2233

311342245225

44552

66

7667543346865568

87654321

xWxW

xxWxxWxxW

xWxWxW

xxWxxxWxxxW

Z

xxxxxxxxP

)()()(),|(),,(

)()()|(),(

664433436346643346

442224244224

xVxVxVxxxVxxxW

xVxVxxVxxW

1X

3X

2X

4X

6X 5X

13W

67W

24W

25W346W

568W

8X7X

3W4W

5W

6W


Probabilistic Inference

Probabilistic Model and Beliefs

)8,7,6,5,4,3,2,1(

),,,,,,,()(\

87654321

i

xxxxxxxxPxPi

iixx

Statistics: Marginal Probability

Statistical Mechanics: One-body distribution

Probabilistic Inference: Belief

1X

3X

2X

4X

6X 5X

13W

67W

24W

25W346W

568W

8X7X

3W4W

5W

6W


Cluster Variation Method

Probabilistic Inference and Probabilistic Model

)()(

),(),(),(

)()()(

),(),,(),,(

),,,,,,,(

2233

311342245225

44552

66

7667543346865568

87654321

xQxQ

xxQxxQxxQ

xQxQxQ

xxQxxxQxxxQ

xxxxxxxxQ

)( ofy Probabilit Marginal : )()( xxxx\x

QQQ

ZDDDD

DDDDDDPfD

ln2 6543

56867252434613

0ln)(

x

xx

x P

QQPQD

x

x

x

1)(

0

Q

Q 1X

3X

2X

4X

6X 5X

13W

67W

24W

25W346W

568W

8X7X

3W4W

5W

6W

67W

1X

3X

2X

4X

6X 5X

8X7X

13W 24W

25W

346W

568W

3X

3X

6X

5X

5X

4X

4X

6X

6X

2X2X

3W

2W

4W

5W

6W



ZWQDWQD

WQDWQD

WQDWQD

WQDWQD

WQDWQDPQD

ln|2|

||

||

||

||

6655

4433

5685686767

25252424

3463461313

1 4 6

54

,),,(),()(

,),(),()(

643346311333

5225422422

x x x

xx

xxxQxxQxQ

xxQxxQxQ

1)(),()(]|[minarg}|{\}|{

xxxCxxx

QQQPQDPCQ

1X

3X

2X

4X

6X 5X

8X7X

13W 24W

25W

346W

568W

3X

3X

6X

5X

5X

4X

4X

6X

6X

2X2X

3W

2W

4W

5W

6W


Generalized Loopy Belief Propagation

Extreme Condition of Kullback-Leibler Divergence

3 4 6

3 4

)()(),,(

)()(),,(

)(42443133643346

42443133643346

63466

x x x

x x

xmxmxxxW

xmxmxxxW

xm

Message Passing Algorithm

1X

3X

2X

4X

6X

)( 3133 xm )( 4244 xm

)( 63466 xm



Expression of Marginal Probability in terms of Messages

3 4 6)()(

)()(),,(

)()(

)()(),,(

),,(

656866676

42443133643346

656866676

42443133643346

643346

x x x xmxm

xmxmxxxW

xmxm

xmxmxxxW

xxxP

1X

3X

2X

4X

6X 5X

8X7X

346W

)( 3133 xm )( 4244 xm

)( 6676 xm

)( 65686 xm



1X

3X

2X

4X

6X 5X

13W

67W

24W

25W346W

568W

8X7X

3W4W

5W

6W

67W

1X

3X

2X

4X

6X 5X

8X7X

13W 24W

25W

346W

568W

3X

3X

6X

5X

5X

4X

4X

6X

6X

2X2X

3W

2W

4W

5W

6W

)( )(

)( )( )(

),( ),,(

),,( ),(

),( ),(

6655

443322

7667865568

6433465225

42243113

xPxP

xPxPxP

xxPxxxP

xxxPxxP

xxPxxP

5 6

6

3

),,()(

),()(

),()(

86556888

766777

311311

x x

x

x

xxxPxP

xxPxP

xxPxP


Linear Response

ji

ii xiii

xiiiiii xPxxPxPxPxx

Nodeat Field External respect to with Nodeat Average ofDeviation

)()(~

)()(~

xx

xx

i

j

hjiji h

xxxxx

i

P

0

lim

)(in Covariant x

x

x)(Pxx iix

x)(Pxxxx jiji

)exp()(~1

)(~

iixhPZ

P xx


Numerical Experiments

Beliefs

0104.0)1(9896.0)1( 33 PP

0104.0)1(9896.0)1( 33 PP

4393.0)1(5607.0)1( 88 PP

4360.0)1(5640.0)1( 88 PP


Exact4500.0)1(5500.0)1( 55 PP

4500.0)1(5500.0)1( 55 PP

1X

3X

2X

4X

6X 5X

13W

67W

24W

25W346W

568W

8X7X

3W4W

5W

6W

i

xxxxxxxxPxP iixx \

87654321 ),,,,,,,()(


Numerical Experiments

Correlation Functions8544.061 xx

1334.083 xx

0890.062 xx

0828.072 xx

8544.061 xx

1402.083 xx

0890.062 xx

0828.072 xx


Exact

1X

3X

2X

4X

6X 5X

13W

67W

24W

25W346W

568W

8X7X

3W4W

5W

6W

i jx x

jiijjijiji xxPxxPxxxx ),()(x

x


Concluding Remarks

Summary

•Review of Cluster Variation Method and Loopy Belief Propagation•Calculation of Correlation Function by means of Linear Response Theory and Cluster Variation Method

Future Problem•Learning of Hyperparameters by means of Maximum Likelihood Estimation

Kazuyuki Tanaka Graduate School of Information Sciences Tohoku University, Japan

Documents

Transcript of Kazuyuki Tanaka Graduate School of Information Sciences Tohoku University, Japan