Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 1st...
-
Upload
jayson-singleton -
Category
Documents
-
view
218 -
download
0
Transcript of Physical Fuctuomatics (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 1st...
Physical Fuctuomatics (Tohoku University) 1
Physical FluctuomaticsApplied Stochastic Process
1st Review of probabilistic information processing
Kazuyuki TanakaGraduate School of Information Sciences
[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/
Webpage:http://www.smapip.is.tohoku.ac.jp/~kazu/PhysicalFluctuomatics/2010/
Physical Fuctuomatics (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) .Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., 2009 (in Japanese).
Physical Fuctuomatics (Tohoku University) 3
References of the present lecture
K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002.Y. Kabashima and D. Saad: Statistical mechanics of low-density parity-check codes (Topical Review), J. Phys. A, vol.37, no.6, pp.R1-R43, 2004. H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001.M. Opper and D. Saad D (eds): Advanced Mean Field Methods --- Theory and Practice, MIT Press, 2001.C. M. Bishop: Pattern Recognition and Machine Learning, Springer, 2006.M. J. Wainwright and M. I. Jordan: Graphical Models, Exponential Families, and Variational Inference, now Publishing Inc, 2008.M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2009.
Physical Fuctuomatics (Tohoku University) 4
Benefit of Information & Communications Technology
Ubiquitous ComputingUbiquitous Internet
Benefit of Information & Communications Technology
Demand for IntelligenceIt cannot be satisfied only with it being only cheap and being quick.
Physical Fuctuomatics (Tohoku University) 5
Field of Information Processing
Information processing according to theoriesInformation processing according to theoriesInference from propositionsInference from propositions
Realization by progress of computational processing capacityRealization by progress of computational processing capacity
Information processing in real worldInformation processing in real worldDiversity of reason in phenomenon Diversity of reason in phenomenon Compete data is not necessarily obtained.Compete data is not necessarily obtained.It is difficult to extract and select some important It is difficult to extract and select some important
information from a lot of data. information from a lot of data. Uncertainty caused by the gap of knowing simply and Uncertainty caused by the gap of knowing simply and understanding actually.understanding actually.We hope to deal successfully with such uncertainty.We hope to deal successfully with such uncertainty.
Information processing for numerical calculationsDefinite Procedure has been given for each calculation.
Physical Fuctuomatics (Tohoku University) 6
Computer for next generations
Required CapacityCapability to sympathize with a user ( Knowledge)Capability to put failure and experience to account in the next chance ( Learning )
How should we deal successfully with the uncertainty caused How should we deal successfully with the uncertainty caused by the gap of knowing simply and understanding actually?by the gap of knowing simply and understanding actually?
Formulation of knowledge and uncertainty
Realization of information processing data with uncertainty
Physical Fuctuomatics (Tohoku University) 7
Computational model for information processing in data with uncertainty
Probabilistic InferenceProbabilistic model
with graphical structure ( Bayesian network )Medical diagnosis
Failure diagnosis Risk Management
Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness.
Inference system for data with uncertainty
modeling
Node is random variable.Arrow is conditional probability.
Mathematical expression of uncertainty=>Probability and Statistics
Graph with cycles
Important aspect
Physical Fuctuomatics (Tohoku University) 8
Computational Model for Probabilistic Information Processing
Probabilistic Information Processing Probabilistic Model
Bayes Formula
Algorithm
Monte Carlo MethodMarkov Chain Monte Carlo
MethodRandomized AlgorithmGenetic Algorithm
Approximate MethodBelief PropagationMean Field Method
Randomness and Approximation
Physical Fuctuomatics (Tohoku University) 9
Probabilistic Image ProcessingProbabilistic Image Processing
Noise Reduction by Probabilistic Image
Processing
K. Tanaka: J. Phys. A, vol.35, 2002.A. S. Willsky: Proceedings of IEEE, vol.90, 2002.
173
110218100
120219202
190202192
Average
192 202 190
202 219 120
100 218 110
192 202 190
202 173 120
100 218 110
Modeling of Probabilistic Image Processing based on Conventional Filters
Markov Random Filed Model Probabilistic Image Processing
The elements of such a digital array are called pixels.At each point, the intensity of light is represented as an integer number or a real number in the digital image data.
Algorithm
Conventional Filter
Physical Fuctuomatics (Tohoku University) 10
Probabilistic Image ProcessingProbabilistic Image Processing
Degraded Image (Gaussian Noise ) Probabilistic Image Processing
Lowpass Filter Wiener Filter Median Filter
MSE:520MSE: 2137
MSE:860 MSE:767 MSE:1040
K. Tanaka: J. Phys. A, vol.35, 2002.A. S. Willsky: Proceedings of IEEE, vol.90, 2002.
Physical Fuctuomatics (Tohoku University) 11
Error Correcting Code
Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004.
High Performance Decoding Algorithm
010 000001111100000 001001011100001
0 1 0
code
010
error
decode
Parity Check Code
Turbo Code, Low Density Parity Check (LDPC) Code
majority ruleError Correcting Codes
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo ) 12
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
1
1
0
0
1
1
6
5
4
3
2
1
x
x
x
x
x
x
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo ) 13
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
1
1
0
0
1
1
6
5
4
3
2
1
x
x
x
x
x
x
1)2 (mod 100
0)2 (mod 101
0)2 (mod 011
9
8
7
X
X
X
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo ) 14
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
1
1
0
0
1
1
6
5
4
3
2
1
x
x
x
x
x
x
1
0
0
1
1
0
0
1
1
9
8
7
6
5
4
3
2
1
x
x
x
x
x
x
x
x
x
1)2 (mod 100
0)2 (mod 101
0)2 (mod 011
9
8
7
X
X
X
Code Word
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo ) 15
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
1
1
0
0
1
1
6
5
4
3
2
1
x
x
x
x
x
x
1
0
0
1
1
0
0
1
1
9
8
7
6
5
4
3
2
1
x
x
x
x
x
x
x
x
x
1
0
1
1
1
0
1
1
1
9
8
7
6
5
4
3
2
1
y
y
y
y
y
y
y
y
y
1)2 (mod 100
0)2 (mod 101
0)2 (mod 011
9
8
7
X
X
X
1 1
0
p1
1
0
0
p1
p
p
Received Word
Code Word
Binary Symmetric Channel
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo ) 16
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo ) 17
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo ) 18
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
14 January, 2010Hokkaido University GCOE Tutorial
(Sapporo ) 19
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
Fundamental Concept for Turbo Codes and LDPC CodesFundamental Concept for Turbo Codes and LDPC Codes
Physical Fuctuomatics (Tohoku University) 20
CDMA Multiuser Detectors in Mobile Phone CommunicationCDMA Multiuser Detectors in Mobile Phone Communication
Relationship between mobile phone communication and spin glass theory
T. Tanaka, IEEE Trans. on Information Theory, vol.48, 2002
Signals of User A
Spreading Code Sequence
Wireless Communication
Received Data
Decode
Spreading Code Sequence
Probabilistic model for decoding can be expressed in terms of a physical model for spin glass phenomena
Noise
Coded Signalsof Other Users
Coded Signals of User A
Physical Fuctuomatics (Tohoku University) 21
Artificial Intelligence
Bayesian Network
CA
SA RA
WA
Probabilistic inference system
Practical algorithmsby means of belief
propagation
J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).
Physical Fuctuomatics (Tohoku University) 22
Main InterestsInformation Processing:
DataPhysics:
Material, Natural Phenomena
System of a lot of elements with mutual relationCommon Concept between Information Sciences and Physics
MaterialMolecule
Materials are constructed from a lot of molecules.
Molecules have interactions of each other.
0 ,1 101101110001
010011101110101000111110000110000101000000111010101110101010Bit
Data
Data is constructed from many bits
A sequence is formed by deciding the arrangement of bits.
A lot of elements have mutual relation of each otherSome physical concepts in Physical models are useful for the design of computational models in probabilistic information processing.
Physical Fuctuomatics (Tohoku University) 23
Horizon of Computation in Probabilistic Information Processing
Compensation of expressing uncertainty using probability and statistics
It must be calculated by taking account of both events with high probability and events with low probability.
Computational Complexity
It is expected to break throw the computational complexity by introducing approximation algorithms.
Physical Fuctuomatics (Tohoku University) 24
What is an important point in What is an important point in computational complexity? computational complexity?
How should we treat the calculation of the summation over 2N configuration?
FT, FT, FT,
21
1 2
,,,x x x
N
N
xxxf
}
}
}
;,,,
F){or Tfor(
F){or Tfor(
F){or Tfor(
;0
21
2
1
L
N
xxxfaa
x
x
x
a
N fold loops
If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.
Physical Fuctuomatics (Tohoku University) 25
Why is a physical viewpoint effective in probabilistic information processing?
Matrials are constructed from a lot of molecules.(1023 molecules exist in 1 mol.)
Molecules have intermolecular forces of each other
1 2
,,, 21x x x
N
N
xxxf
Theoretical physicists always have to treat such multiple summation.
Development of Approximate Methods
Probabilistic information processing is also usually reduced to multiple summations or integrations.
Application of physical approximate methods to probabilistic information processing
Physical Fuctuomatics (Tohoku University) 26
Academic Circulation
Academic Circulation
Academic Circulation between Physics and Information Sciences
Physics Information Sciences
Understanding and prediction of properties of materials and natural phenomena
Extraction and processing of information in data
Common ConceptStatistical Mechanical Informatics
Probabilistic Information Processing
Statistical Sciences
Physical Fuctuomatics (Tohoku University) 27
Summary of the present lecture
Probabilistic information processingExamples of probabilistic information processingCommon concept in physics and information sciencesApplication of physical modeling and approximations
Future Lectures
Fundamental theory of probability and statisticsLinear model
Graphical model...