Math 341: Probability Nineteenth Lecture...

Summary for the Day Central Limit Theorem CLT and MGF CLT and Fourier Analysis

Math 341: ProbabilityNineteenth Lecture (11/17/09)

Steven J MillerWilliams College

[email protected]://www.williams.edu/go/math/sjmiller/

public html/341/

Bronfman Science CenterWilliams College, November 17, 2009

1


Summary for the Day

2


Summary for the day

Central Limit Theorem:⋄ Statement of the CLT.⋄ Poisson example.⋄ Proof with MGFs.⋄ Proof with Fourier analysis.⋄ Discuss rate of convergence.

3


Central Limit Theorem

4


Normalization of a random variable

Normalization (standardization) of a random variableLet X be a random variable with mean � and standarddeviation �, both of which are finite. The normalization, Y ,is defined by

Y :=X − E[X ]

StDev(X )=

X − �

�.

Note that

E[Y ] = 0 and StDev(Y ) = 1.

5


Statement of the Central Limit Theorem

Normal distributionA random variable X is normally distributed (or has thenormal distribution, or is a Gaussian random variable)with mean � and variance �2 if the density of X is

f (x) =1√

2��2exp

(−(x − �)2

2�2

).

We often write X ∼ N(�, �2) to denote this. If � = 0 and�2 = 1, we say X has the standard normal distribution.

6


Statement of the Central Limit Theorem

Central Limit TheoremLet X1, . . . ,XN be independent, identically distributedrandom variables whose moment generating functionsconverge for ∣t ∣ < � for some � > 0 (this implies all themoments exist and are finite). Denote the mean by � andthe variance by �2, let

X N =X1 + ⋅ ⋅ ⋅+ XN

N

and set

ZN =X N − �

�/√

N.

Then as N → ∞, the distribution of ZN converges to thestandard normal.

7


Alternative Statement of the Central Limit Theorem

Central Limit TheoremLet X1, . . . ,XN be independent, identically distributedrandom variables whose moment generating functionsconverge for ∣t ∣ < � for some � > 0 (this implies all themoments exist and are finite). Denote the mean by � andthe variance by �2, let

SN = X1 + ⋅ ⋅ ⋅+ XN

and set

ZN =SN − N�√

N�2.

Then as N → ∞, the distribution of ZN converges to thestandard normal.

8


Key Probabilities:

Key probabilities for Z ∼ N(0, 1) (i.e., Z has the standardnormal distribution).

Prob(∣Z ∣ ≤ 1) ≈ 68.2%.

Prob(∣Z ∣ ≤ 1.96) ≈ 95%.

Prob(∣Z ∣ ≤ 2.575) ≈ 99%.

9


Convergence to the standard normal

Question:Let X1,X2, . . . be iidrv with mean 0 and variance 1, and letZN = X N/(1/

√N). By the CLT ZN → N(0, 1); which

choice converges fastest? Slowest?

1 Uniform: X ∼ Unif(−√

3,√

3). Excess Kurtosis: -1.2.2 Laplace: fX (x) = e−

√2∣x∣/

√2. Excess Kurtosis: 3.

3 Normal: X ∼ N(0, 1). Excess Kurtosis: 0.4 Millered Cauchy: fX (x) =

4a sin(�/8)�

11+(ax)8 ,

a =√√

2 − 1. Excess Kurtosis: 1 +√

2 − 3 ≈ −.586.

log MX (t) = t2

2 + (�4−3)t4

4! + O(t6).

10



-4 -2 2 4

0.05

0.10

0.15

Figure: Convolutions of 5 Uniforms.

11



-4 -2 2 4

0.05

0.10

0.15

Figure: Convolutions of 5 Laplaces.

12



-4 -2 2 4

0.1

0.2

0.3

0.4

Figure: Convolutions of 5 Normals.

13



-4 -2 2 4

0.05

0.10

0.15

0.20

0.25

0.30

Figure: Convolutions of 1 Millered Cauchy.

14



-4 -2 2 4

0.05

0.10

0.15

0.20

0.25

Figure: Convolutions of 2 Millered Cauchy.

15


Central Limit Theorem

16


MGF and the CLT

Moment generating function of normal distributionsLet X be a normal random variable with mean � andvariance �2. Its moment generating function satisfies

MX (t) = e�t+�2 t2

2 .

In particular, if Z has the standard normal distribution, itsmoment generating function is

MZ (t) = et2/2.

17


MGF and the CLT

Moment generating function of normal distributionsLet X be a normal random variable with mean � andvariance �2. Its moment generating function satisfies

MX (t) = e�t+�2 t2

2 .

In particular, if Z has the standard normal distribution, itsmoment generating function is

MZ (t) = et2/2.

Proof: Complete the square.

18


Poisson Example of the CLT

ExampleLet X ,X1, . . . ,XN be Poisson random variables withparameter �. Let

X N =X1 + ⋅ ⋅ ⋅+ XN

N, Y =

X − E[X ]

StDev(X ).

Then as N → ∞, Y converges to having the standardnormal distribution.

19


Poisson Example of the CLT

ExampleLet X ,X1, . . . ,XN be Poisson random variables withparameter �. Let

X N =X1 + ⋅ ⋅ ⋅+ XN

N, Y =

X − E[X ]

StDev(X ).

Then as N → ∞, Y converges to having the standardnormal distribution.

Moment generating function: MX (t) = exp(�(et − 1)).Independent formula: MX1+X2(t) = MX1(t)MX2(t).Shift formula: MaX+b(t) = ebtMX (at).

20


General proof via Moment Generating Functions

Xi ’s iidrv,

ZN =X − �

�/√

N=

N∑

n=1

Xi − �

�√

N.

21



Xi ’s iidrv,

ZN =X − �

�/√

N=

N∑

n=1

Xi − �

�√

N.

Moment Generating Function is:

MZN (t) =N∏

n=1

e−�t�

√N MX

(t

�√

N

)= e

−�t√

N� MX

(t

�√

N

)N

22



Xi ’s iidrv,

ZN =X − �

�/√

N=

N∑

n=1

Xi − �

�√

N.

Moment Generating Function is:

MZN (t) =N∏

n=1

e−�t�

√N MX

(t

�√

N

)= e

−�t√

N� MX

(t

�√

N

)N

Taking logarithms:

log MZN (t) = −�t√

N�

+ N log MX

(t

�√

N

).

23


General proof via Moment Generating Functions (cont)

Expansion of MGF:

MX (t) = 1 + �t +�′

2t2

2!+ ⋅ ⋅ ⋅ = 1 + t

(�+

�′2t2

+ ⋅ ⋅ ⋅).

24



Expansion of MGF:

MX (t) = 1 + �t +�′

2t2

2!+ ⋅ ⋅ ⋅ = 1 + t

(�+

�′2t2

+ ⋅ ⋅ ⋅).

Expansion for log(1 + u) is

log(1 + u) = u − u2

2+

u3

3!− ⋅ ⋅ ⋅ .

25



Expansion of MGF:

MX (t) = 1 + �t +�′

2t2

2!+ ⋅ ⋅ ⋅ = 1 + t

(�+

�′2t2

+ ⋅ ⋅ ⋅).

Expansion for log(1 + u) is

log(1 + u) = u − u2

2+

u3

3!− ⋅ ⋅ ⋅ .

Combining gives

log MX (t) = t(�+

�′2t2

+ ⋅ ⋅ ⋅)−

t2(�+

�′2t2 + ⋅ ⋅ ⋅

)2

2+ ⋅ ⋅ ⋅

= �t +�′

2 − �2

2t2 + terms in t3 or higher.

26



log MX

(t

�√

N

)

=�t

�√

N+

�2

2t2

�2N+ terms in t3/N3/2 or lower in N.

27



log MX

(t

�√

N

)

=�t

�√

N+

�2

2t2

�2N+ terms in t3/N3/2 or lower in N.

Denote lower order terms by O(N−3/2). Collecting gives

log MZN (t) = −�t√

N�

+ N(

�t

�√

N+

t2

2N+ O(N−3/2)

)

= −�t√

N�

+�t√

N�

+t2

2+ O(N−1/2)

=t2

2+ O(N−1/2).

28


Central Limit Theoremand Fourier Analysis

29


Convolutions

Convolution of f and g:

h(y) = (f ∗ g)(y) =

∫

ℝ

f (x)g(y − x)dx =

∫

ℝ

f (x − y)g(x)dx .

30


Convolutions


h(y) = (f ∗ g)(y) =

∫

ℝ

f (x)g(y − x)dx =

∫

ℝ

f (x − y)g(x)dx .

X1 and X2 independent random variables with probability density p.

Prob(Xi ∈ [x , x +Δx ]) =

∫ x+Δx

xp(t)dt ≈ p(x)Δx .

Prob(X1 + X2) ∈ [x , x +Δx ] =

∫ ∞

x1=−∞

∫ x+Δx−x1

x2=x−x1

p(x1)p(x2)dx2dx1.

31


Convolutions


h(y) = (f ∗ g)(y) =

∫

ℝ

f (x)g(y − x)dx =

∫

ℝ

f (x − y)g(x)dx .

X1 and X2 independent random variables with probability density p.

Prob(Xi ∈ [x , x +Δx ]) =

∫ x+Δx

xp(t)dt ≈ p(x)Δx .

Prob(X1 + X2) ∈ [x , x +Δx ] =

∫ ∞

x1=−∞

∫ x+Δx−x1

x2=x−x1

p(x1)p(x2)dx2dx1.

As Δx → 0 we obtain the convolution of p with itself:

Prob(X1 + X2 ∈ [a, b]) =

∫ b

a(p ∗ p)(z)dz.

Exercise to show non-negative and integrates to 1.32


Statement of Central Limit Theorem

WLOG p has mean zero, variance one, finite third moment anddecays rapidly so all convolution integrals converge: p infinitelydifferentiable function satisfying∫ ∞

−∞xp(x)dx = 0,

∫ ∞

−∞x2p(x)dx = 1,

∫ ∞

−∞∣x ∣3p(x)dx < ∞.

X1,X2, . . . are iidrv with density p.

Define SN =∑N

i=1 Xi .

Standard Gaussian (mean zero, variance one) is 1√2�

e−x2/2.

33


Statement of Central Limit Theorem

WLOG p has mean zero, variance one, finite third moment anddecays rapidly so all convolution integrals converge: p infinitelydifferentiable function satisfying∫ ∞

−∞xp(x)dx = 0,

∫ ∞

−∞x2p(x)dx = 1,

∫ ∞

−∞∣x ∣3p(x)dx < ∞.

X1,X2, . . . are iidrv with density p.

Define SN =∑N

i=1 Xi .

Standard Gaussian (mean zero, variance one) is 1√2�

e−x2/2.

Central Limit Theorem Let Xi ,SN be as above and assume the thirdmoment of each Xi is finite. Then SN/

√N converges in probability to

the standard Gaussian:

limN→∞

Prob

(SN√

N∈ [a, b]

)=

1√2�

∫ b

ae−x2/2dx .

34


Proof of the Central Limit Theorem

The Fourier transform: p(y) =∫∞−∞ p(x)e−2�ixy dx .

35




Derivative of g is the Fourier transform of −2�ixg(x);differentiation (hard) is converted to multiplication (easy).

g′(y) =

∫ ∞

−∞−2�ix ⋅ g(x)e−2�ixy dx ;

g prob. density, g′(0) = −2�iE[x ], g′′(0) = −4�2E[x2].

36





g′(y) =

∫ ∞

−∞−2�ix ⋅ g(x)e−2�ixy dx ;


Natural: mean and variance simple multiples of derivatives of pat zero: p′(0) = 0, p′′(0) = −4�2.

37





g′(y) =

∫ ∞

−∞−2�ix ⋅ g(x)e−2�ixy dx ;


Natural: mean and variance simple multiples of derivatives of pat zero: p′(0) = 0, p′′(0) = −4�2.

We Taylor expand p (need technical conditions on p):

p(y) = 1 +p′′(0)

2y2 + ⋅ ⋅ ⋅ = 1 − 2�2y2 + O(y3).

Near origin, p a concave down parabola.

38


Proof of the Central Limit Theorem (cont)

Prob(X1 + ⋅ ⋅ ⋅+ XN ∈ [a, b]) =∫ b

a (p ∗ ⋅ ⋅ ⋅ ∗ p)(z)dz.

39



Prob(X1 + ⋅ ⋅ ⋅+ XN ∈ [a, b]) =∫ b

a (p ∗ ⋅ ⋅ ⋅ ∗ p)(z)dz.

The Fourier transform converts convolution to multiplication. IfFT[f ](y) denotes the Fourier transform of f evaluated at y :

FT[p ∗ ⋅ ⋅ ⋅ ∗ p](y) = p(y) ⋅ ⋅ ⋅ p(y).

40



Prob(X1 + ⋅ ⋅ ⋅+ XN ∈ [a, b]) =∫ b

a (p ∗ ⋅ ⋅ ⋅ ∗ p)(z)dz.


FT[p ∗ ⋅ ⋅ ⋅ ∗ p](y) = p(y) ⋅ ⋅ ⋅ p(y).

Do not want the distribution of X1 + ⋅ ⋅ ⋅+ XN = x , but ratherSN = X1+⋅⋅⋅+XN√

N= x .

41



Prob(X1 + ⋅ ⋅ ⋅+ XN ∈ [a, b]) =∫ b

a (p ∗ ⋅ ⋅ ⋅ ∗ p)(z)dz.


FT[p ∗ ⋅ ⋅ ⋅ ∗ p](y) = p(y) ⋅ ⋅ ⋅ p(y).


N= x .

If B(x) = A(cx) for some fixed c ∕= 0, then B(y) = 1c A

( yc

).

42



Prob(X1 + ⋅ ⋅ ⋅+ XN ∈ [a, b]) =∫ b

a (p ∗ ⋅ ⋅ ⋅ ∗ p)(z)dz.


FT[p ∗ ⋅ ⋅ ⋅ ∗ p](y) = p(y) ⋅ ⋅ ⋅ p(y).


N= x .


( yc

).

Prob(

X1+⋅⋅⋅+XN√N

= x)

= (√

Np ∗ ⋅ ⋅ ⋅ ∗√

Np)(x√

N).

43



Prob(X1 + ⋅ ⋅ ⋅+ XN ∈ [a, b]) =∫ b

a (p ∗ ⋅ ⋅ ⋅ ∗ p)(z)dz.


FT[p ∗ ⋅ ⋅ ⋅ ∗ p](y) = p(y) ⋅ ⋅ ⋅ p(y).


N= x .


( yc

).

Prob(

X1+⋅⋅⋅+XN√N

= x)

= (√

Np ∗ ⋅ ⋅ ⋅ ∗√

Np)(x√

N).

FT[(√

Np ∗ ⋅ ⋅ ⋅ ∗√

Np)(x√

N)](y) =

[p(

y√N

)]N.

44



Can find the Fourier transform of the distribution of SN :[p(

y√N

)]N

.

45




y√N

)]N

.

Take the limit as N → ∞ for fixed y .

46




y√N

)]N

.


Know p(y) = 1 − 2�2y2 + O(y3). Thus study[1 − 2�2y2

N+ O

(y3

N3/2

)]N

.

47




y√N

)]N

.



N+ O

(y3

N3/2

)]N

.

For any fixed y ,

limN→∞

[1 − 2�2y2

N+ O

(y3

N3/2

)]N

= e−2�2y2

.

48




y√N

)]N

.



N+ O

(y3

N3/2

)]N

.

For any fixed y ,

limN→∞

[1 − 2�2y2

N+ O

(y3

N3/2

)]N

= e−2�2y2

.

Fourier transform of 1√2�

e−x2/2 at y is e−2�2y2.

49



We have shown:

the Fourier transform of the distribution of SN converges toe−2�2y2

;

the Fourier transform of 1√2�

e−x2/2 at y is e−2�2y2.

Therefore the distribution of SN equalling x converges to 1√2�

e−x2/2.

50



We have shown:

the Fourier transform of the distribution of SN converges toe−2�2y2

;

the Fourier transform of 1√2�

e−x2/2 at y is e−2�2y2.

Therefore the distribution of SN equalling x converges to 1√2�

e−x2/2.We need complex analysis to justify this inversion. Must be careful:Consider

g(x) =

{e−1/x2

if x ∕= 0

0 if x = 0.

All the Taylor coefficients about x = 0 are zero, but the function is notidentically zero in a neighborhood of x = 0.

51

Math 341: Probability Nineteenth Lecture...

Documents

Transcript of Math 341: Probability Nineteenth Lecture...