Overview 541 Probability Theory
-
Upload
albert-pang -
Category
Documents
-
view
218 -
download
0
Transcript of Overview 541 Probability Theory
-
7/28/2019 Overview 541 Probability Theory
1/51
Advanced Probability Theory (Math541)
Instructor: Kani Chen
(Classic)/Modern Probability Theory (1900-1960)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 1 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
2/51
Primitive/Classic Probability:
(16th-19th century)
Gerolamo Cardano (1501-1576)
Liber de Ludo Aleae (games of chance)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 2 / 17
http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
3/51
Primitive/Classic Probability:
(16th-19th century)
Gerolamo Cardano (1501-1576)
Liber de Ludo Aleae (games of chance)
Galilei Galileo (1564-1642)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 2 / 17
http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
4/51
Primitive/Classic Probability:
(16th-19th century)
Gerolamo Cardano (1501-1576)
Liber de Ludo Aleae (games of chance)
Galilei Galileo (1564-1642)Pierre de Fermat (1601-1665) and Blaise Pascal (1623-1662)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 2 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
5/51
Primitive/Classic Probability:
(16th-19th century)
Gerolamo Cardano (1501-1576)
Liber de Ludo Aleae (games of chance)
Galilei Galileo (1564-1642)Pierre de Fermat (1601-1665) and Blaise Pascal (1623-1662)
Jakob Bernoulli (1654 -1705) Bernoulli
trial/distribution/r.v/numbers
Daniel (1700-1782) (utility function)
Johann (1667-1748) (LHopital rule)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 2 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
6/51
Abraham de Moivre (1667-1754)Doctrine of Chances
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 3 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
7/51
Abraham de Moivre (1667-1754)Doctrine of Chances
Pierre-Simon Laplace (1749-1827)
Laplace transform
De Moivre-Laplace Theorem (the central limit theorem)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 3 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
8/51
Abraham de Moivre (1667-1754)Doctrine of Chances
Pierre-Simon Laplace (1749-1827)
Laplace transform
De Moivre-Laplace Theorem (the central limit theorem)Simeon Denis Poisson (1781-1840).
Poisson distribution/process, the law of rare events
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 3 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
9/51
Abraham de Moivre (1667-1754)Doctrine of Chances
Pierre-Simon Laplace (1749-1827)
Laplace transform
De Moivre-Laplace Theorem (the central limit theorem)Simeon Denis Poisson (1781-1840).
Poisson distribution/process, the law of rare events
Emile Borel ( 1871-1956).
Borel sets/measurable, Borel-Cantelli lemma, Borel strong law.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 3 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
10/51
Foundation of modern probability:
Keynes, J. M. (1921): A Treatise on Probability.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 4 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
11/51
Foundation of modern probability:
Keynes, J. M. (1921): A Treatise on Probability.
von Mises, R. (1928): Probability, Statistics and Truth.
(1931): Mathematical Theory of Probability and Statistics.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 4 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
12/51
Foundation of modern probability:
Keynes, J. M. (1921): A Treatise on Probability.
von Mises, R. (1928): Probability, Statistics and Truth.
(1931): Mathematical Theory of Probability and Statistics.
Kolmogorov, A. (1933): Foundations of the Theory of Probability.
Kolmogorovs axioms: Probability space trio: (,F, P).
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 4 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
13/51
Measure-theoretic Probabilities
(Chapter 1 begins)Sets, set operations (, and complement), set of sets/subsets,algebra, -algebra, measurable space (,F), product space ...
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 5 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
14/51
Measure-theoretic Probabilities
(Chapter 1 begins)Sets, set operations (, and complement), set of sets/subsets,algebra, -algebra, measurable space (,F), product space ...measures, probabilities, product measure, independence of
events, conditional probabilities, conditional expectation withrespect to -algebra.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 5 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
15/51
Measure-theoretic Probabilities
(Chapter 1 begins)Sets, set operations (, and complement), set of sets/subsets,algebra, -algebra, measurable space (,F), product space ...measures, probabilities, product measure, independence of
events, conditional probabilities, conditional expectation withrespect to -algebra.
Transformation/map, measurable transformation/map, random
variables/elements defined through mapping X = X().
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 5 / 17
http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
16/51
Measure-theoretic Probabilities
(Chapter 1 begins)Sets, set operations (, and complement), set of sets/subsets,algebra, -algebra, measurable space (,F), product space ...measures, probabilities, product measure, independence of
events, conditional probabilities, conditional expectation withrespect to -algebra.
Transformation/map, measurable transformation/map, random
variables/elements defined through mapping X = X().
Expectation/integral.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 5 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
17/51
Measure-theoretic Probabilities
(Chapter 1 begins)Sets, set operations (, and complement), set of sets/subsets,algebra, -algebra, measurable space (,F), product space ...measures, probabilities, product measure, independence of
events, conditional probabilities, conditional expectation withrespect to -algebra.
Transformation/map, measurable transformation/map, random
variables/elements defined through mapping X = X().
Expectation/integral.
Caratheodorys extension theorem, Kolmogorovs extension
theorem, Dynkins theorem, The Radon-Nikodym Theorem.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 5 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
18/51
Convergence.
Convergence modes:
convergence almost sure (strong),
in probability,
in Lp (most commonly, L1 or L2),
in distribution/law (weak).The relations of the convergence modes.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 6 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
19/51
Convergence.
Convergence modes:
convergence almost sure (strong),
in probability,
in Lp (most commonly, L1 or L2),
in distribution/law (weak).The relations of the convergence modes.
Dominated convergence theorem,
(extension to uniformly integrable r.v.s.)
monotone convergence theorem.Fatous lemma.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 6 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
20/51
Law of Large numbers.
X1, ..., Xn,... are iid random variables with mean . Let Sn = ni=1 Xi.
Weak law of Large numbers:
Sn/n , in probability.
i.e., for any > 0,
P(|Sn/n | > ) 0.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 7 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
21/51
Law of Large numbers.
X1, ..., Xn,... are iid random variables with mean . Let Sn = ni=1 Xi.
Weak law of Large numbers:
Sn/n , in probability.
i.e., for any > 0,
P(|Sn/n | > ) 0.
Kolmogorovs Strong law of Large numbers:
Sn/n , almost surely.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 7 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
22/51
Law of Large numbers.
X1, ..., Xn,... are iid random variables with mean . Let Sn = ni=1 Xi.
Weak law of Large numbers:
Sn/n , in probability.
i.e., for any > 0,
P(|Sn/n | > ) 0.
Kolmogorovs Strong law of Large numbers:
Sn/n , almost surely.
Consistency in statistical estimation.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 7 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
23/51
Large deviation (strengthening weak law)
For fixed t > 0, how fast is P(Sn/n > t) 0?Under regularity conditions,
1
n log P(Sn/n > t) (t),where is determined by the commmon distribution of Xi.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 8 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
24/51
Large deviation (strengthening weak law)
For fixed t > 0, how fast is P(Sn/n > t) 0?Under regularity conditions,
1
n log P(Sn/n > t) (t),where is determined by the commmon distribution of Xi.
Special case, Xi are iid N(0, 1), then (t) = t2/2.Note that 1
(x)
(x)/x.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 8 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
25/51
Law of iterated logarithm (strengthening strong law)
Sn/n
0 a.s.
Is there a proper an such that the limit of Sn/an is nonzero finite?
Kolmogorovs law of iterated logarithm.
lim sup
n
Sn/n
22
nlog log(n) 1, a.s.
where 2 = var(Xi).
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 9 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
26/51
Law of iterated logarithm (strengthening strong law)
Sn/n
0 a.s.
Is there a proper an such that the limit of Sn/an is nonzero finite?
Kolmogorovs law of iterated logarithm.
lim sup
n
Sn/n
22
nlog log(n) 1, a.s.
where 2 = var(Xi).
Marcinkiewicz-Zygmund strong law: for 0 < p < 2,
Sn ncn1/p
0, a.s.,
if and only if E(|Xi|p) < , where c = for 1 p < 2.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 9 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
27/51
Convergence of Series.
The convergence of Sn for independent X1,..., Xn.Khintchines convergence theorem: if E(Xi) = 0 and
nvar(Xn) < , then Sn converges a.s. as well as in L2.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 10 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
28/51
Convergence of Series.
The convergence of Sn for independent X1,..., Xn.Khintchines convergence theorem: if E(Xi) = 0 and
nvar(Xn) < , then Sn converges a.s. as well as in L2.Kolmogorovs three series theorem:
Sn converges a.s. if and only if
nP(|Xn| > 1) < ,nE(Xn1{|Xn|1}) < , and
nvar(Xn1{|Xn|1}) < .
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 10 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
29/51
Convergence of Series.
The convergence of Sn for independent X1,..., Xn.Khintchines convergence theorem: if E(Xi) = 0 and
nvar(Xn) < , then Sn converges a.s. as well as in L2.Kolmogorovs three series theorem:
Sn converges a.s. if and only if
nP(|Xn| > 1) < ,nE(Xn1{|Xn|1}) < , and
nvar(Xn1{|Xn|1}) < .
Kronecker Lemma:
If 0 < bn and
n(an/bn) < thenn
j=1 aj/bn 0.A technique to show law of large numbers via convergence of
series.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 10 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
30/51
The central limit theorem (Chapter 2).
De Moivre-Lapalace Theorem
For X1, X2... independent, under proper conditions,
Sn
E(Sn)
var(Sn) N(0, 1) in distribution.
i.e., PSn E(Sn)
var(Sn)< x
P(N(0, 1) < x)
for all x.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 11 / 17
http://find/ -
7/28/2019 Overview 541 Probability Theory
31/51
The conditions and extensions
Lyapunov.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 12 / 17
http://goforward/http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
32/51
The conditions and extensions
Lyapunov.Lindeberg (1922) condition: Xj is mean 0 with variance
2j , such
thatn
j=
1
E(X2j 1{|Xj|>sn}) = o(s2n)
for all > 0, where s2n =n
j=1 2
j .
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 12 / 17
http://goforward/http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
33/51
The conditions and extensions
Lyapunov.Lindeberg (1922) condition: Xj is mean 0 with variance
2j , such
thatn
j=
1
E(X2j 1{|Xj|>sn}) = o(s2n)
for all > 0, where s2n =n
j=1 2
j .
Feller (1935): If CLT holds, n/sn 0 and sn, then theabove Lindeberg condition holds.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 12 / 17
Th di i d i
http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
34/51
The conditions and extensions
Lyapunov.Lindeberg (1922) condition: Xj is mean 0 with variance
2j , such
thatn
j=
1
E(X2j 1{|Xj|>sn}) = o(s2n)
for all > 0, where s2n =n
j=1 2
j .
Feller (1935): If CLT holds, n/sn 0 and sn, then theabove Lindeberg condition holds.
Extensions to martingales, Markov process ...
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 12 / 17
Th diti d t i
http://find/ -
7/28/2019 Overview 541 Probability Theory
35/51
The conditions and extensions
Lyapunov.Lindeberg (1922) condition: Xj is mean 0 with variance
2j , such
thatn
j=1
E(X2j 1{|Xj|>sn}) = o(s2n)
for all > 0, where s2n =n
j=1 2
j .
Feller (1935): If CLT holds, n/sn 0 and sn, then theabove Lindeberg condition holds.
Extensions to martingales, Markov process ...Inference in statistics (accuracy justification of estimation:
confidence interval, test of hypothesis.)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 12 / 17
R t f t lit
http://find/ -
7/28/2019 Overview 541 Probability Theory
36/51
Rate of convergence to normality
Suppose X1, ..., Xn are iid.
Berry-Esseen Bound:
PSn
n
n2 < x P(N(0, 1) < x)
c/3
n1/2
for all x, where = E(|Xi|3) and c is a universal constant.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 13 / 17
R t f t lit
http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
37/51
Rate of convergence to normality
Suppose X1, ..., Xn are iid.
Berry-Esseen Bound:
PSn
n
n2 < x P(N(0, 1) < x)
c/3
n1/2
for all x, where = E(|Xi|3) and c is a universal constant.Extensions, e.g., to non iid cases, etc..
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 13 / 17
Rate of convergence to normality
http://find/ -
7/28/2019 Overview 541 Probability Theory
38/51
Rate of convergence to normality
Suppose X1, ..., Xn are iid.
Berry-Esseen Bound:
PSn
n
n2 < x P(N(0, 1) < x)
c/3
n1/2
for all x, where = E(|Xi|3) and c is a universal constant.Extensions, e.g., to non iid cases, etc..
Edgeworth expansion.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 13 / 17
Random Walk (Chapter 3)
http://find/ -
7/28/2019 Overview 541 Probability Theory
39/51
Random Walk (Chapter 3)
Given X1, X2,... iid, we study the behavior {S1, S2,...} as a sequenceof random variables.
Stopping times.
T is an integer-valued r.v. such that T = nonly depends on thevalues of X1,..., Xn, on, in other words, the values S1, ..., Sn.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 14 / 17
Random Walk (Chapter 3)
http://find/ -
7/28/2019 Overview 541 Probability Theory
40/51
Random Walk (Chapter 3)
Given X1, X2,... iid, we study the behavior {S1, S2,...} as a sequenceof random variables.
Stopping times.
T is an integer-valued r.v. such that T = nonly depends on thevalues of X1,..., Xn, on, in other words, the values S1, ..., Sn.
Walds identity/equation:
Under proper conditions, E(ST) = E(T).
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 14 / 17
Random Walk (Chapter 3)
http://find/ -
7/28/2019 Overview 541 Probability Theory
41/51
Random Walk (Chapter 3)
Given X1, X2,... iid, we study the behavior {S1, S2,...} as a sequenceof random variables.
Stopping times.
T is an integer-valued r.v. such that T = nonly depends on thevalues of X1,..., Xn, on, in other words, the values S1, ..., Sn.
Walds identity/equation:
Under proper conditions, E(ST) = E(T).
Interpretation: if Xi is the gain/loss of the i-th game, which is fair in
the sense that E(Xi) = 0. Then Sn is the cumulative gain/loss in
the first ngames. Under proper conditions, any exit strategy(stopping time) shall still break even.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 14 / 17
Tool box:
http://find/ -
7/28/2019 Overview 541 Probability Theory
42/51
Tool box:
Indicator functions.
(e.g.,Borel-Cantelli Lemma Borels strong law of large numbers
Kolmogorovs 0-1 law.)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 15 / 17
Tool box:
http://find/ -
7/28/2019 Overview 541 Probability Theory
43/51
Tool box:
Indicator functions.
(e.g.,Borel-Cantelli Lemma Borels strong law of large numbers
Kolmogorovs 0-1 law.)
Truncation.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 15 / 17
Tool box:
http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
44/51
Tool box:
Indicator functions.
(e.g.,Borel-Cantelli Lemma Borels strong law of large numbers
Kolmogorovs 0-1 law.)
Truncation.
Characteristic functions/moment generating functions. (in proving
the CLT and its convergence rates)
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 15 / 17
Inequalities
http://find/ -
7/28/2019 Overview 541 Probability Theory
45/51
Inequalities
Jensen, Holder, Cauchy-Schwartz, Lyapunov, Minkowski.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 16 / 17
Inequalities
http://find/ -
7/28/2019 Overview 541 Probability Theory
46/51
Inequalities
Jensen, Holder, Cauchy-Schwartz, Lyapunov, Minkowski.
Markov, Chebyshev.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 16 / 17
Inequalities
http://find/ -
7/28/2019 Overview 541 Probability Theory
47/51
Inequalities
Jensen, Holder, Cauchy-Schwartz, Lyapunov, Minkowski.
Markov, Chebyshev.
Kolmogorov.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 16 / 17
Inequalities
http://find/ -
7/28/2019 Overview 541 Probability Theory
48/51
Inequalities
Jensen, Holder, Cauchy-Schwartz, Lyapunov, Minkowski.
Markov, Chebyshev.
Kolmogorov.
Exponential: Berstein, Bennett, Hoeffding,
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 16 / 17
Inequalities
http://find/http://goback/ -
7/28/2019 Overview 541 Probability Theory
49/51
equa t es
Jensen, Holder, Cauchy-Schwartz, Lyapunov, Minkowski.
Markov, Chebyshev.
Kolmogorov.
Exponential: Berstein, Bennett, Hoeffding,
Doob (for martingale).
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 16 / 17
Inequalities
http://find/ -
7/28/2019 Overview 541 Probability Theory
50/51
q
Jensen, Holder, Cauchy-Schwartz, Lyapunov, Minkowski.
Markov, Chebyshev.
Kolmogorov.
Exponential: Berstein, Bennett, Hoeffding,
Doob (for martingale).
Khintchine, Marcinkiewicz-Zygmund, Burkholder-Gundy,
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 16 / 17
Martingales (Chapter 4).
http://find/ -
7/28/2019 Overview 541 Probability Theory
51/51
g ( p )
1. Conditional expectation with respect to -algebra.2. Definition of martingales,
3. Inequalities.4. Optional sampling theorem.
5. Martingale convergence theorem.
Instructor: Kani Chen (HKUST) Advanced Probability Theory (Math541) 17 / 17
http://find/