1

Lecture 15: Statistics and Their Distributions, Central Limit Theorem

Devore, Ch. 5.3-5.5

Topics

I. The Concept of a “Statistic”

II. Independent, Identically Distributed (iid) Samples

III. Deriving Sampling Distribution of Statistic– By Probability Rules– By Simulation

• Application – Tolerances

IV. Distribution of the Sample Mean / Total

V. Central Limit Theorem

VI. Distribution of a Linear Combination

I. Concepts of a “Statistic”

• Consider taking two samples of size n from the same population distribution.– A: 30.7, 29.4, 31.1 Mean 30.4– B: 28.8, 30.0, 31.1 Mean 29.97

• Which group has the larger mean?

• Propositions

– The uncertainty of individual values xi when sampling from a population distribution implies a r.v.

– This uncertainty further implies that any statistic calculated from the population distribution also varies from sample to sample.

Example: Minitab

• Suppose X ~ Weibull (shape= 2, scale = 5)– E(X) = 4.4311; V(X) = 5.365

• Using Minitab generate samples of 10 and observe differences in mean and variance.

Sample 1 2 3 4 5 6Mean 4.401 5.928 4.229 4.132 3.620 5.761Median 4.360 6.144 4.608 3.857 3.221 6.342Std Dev 2.642 2.062 1.611 2.124 1.678 2.496

Results shown are from Devore, p. 226-227

Point Estimates / Sampling Distributions

• Point Estimate – value for a sample statistic from a particular sample.

• Statistic – rv whose value may be calculated from a sample of data -- lowercase letter indicates the calculated or observed value of the statistic.

S s

• Probability Distribution of a Statistic is known as its Sampling Distribution.

xX

valuecomputed theis where)( xxXP

II. iid Random Samples

• Sampling Distribution depends on several items:– Population Distribution (parameters)– Sample size, n– Method of Sampling (with or without replacement)

• rv’s X1, X2, .. Xn form a random sample of size n if:1. Xi’s are independent rv’s (independent)2. Every Xi has the same probability distribution (identically distributed)

• If satisfy above two conditions we say Xi’s are iid

– sampling with replacement or from infinite population iid– sampling w/o replacement requires sample sizes n much smaller than

population N to assume iid (rule: n/N <= 0.05).

III. Deriving Sampling Distribution of a Statistic

• By Probability Rules– used for simple cases with a few Xi’s– cases where derivation is already done.

• By Simulation (more common!)– typically used when derivation via probability

rules is complicated, or if:– Underlying distribution of interest in unknown

(assumed).

Deriving Via Simple Probability

• Example: Suppose you sell two brands of DVD players for A: $150, and B: $200.

• Sales records indicate the following:– A – 60% of Sales; B: 40% of Sales

• Let X1 – revenue from selling A; X2 revenue from B

• Suppose you take samples of size n=2.– List the possible outcome, p(x1,x2), sample mean and variance.

DVD Example: Sampling Distribution

• What is the relationship between the expected value of X-bar and variance of X-bar and the original statistics?

22)(][

)(][

xx

x

xpxXV

xpxXE

Compute:

E(X) 170 V(X) 600

n=2X1 X2 p(x1,x2) x-bar s2

150 150 0.36 150 0150 200 0.24 175 1250200 150 0.24 175 1250200 200 0.16 200 0

x-bar 150 175 200px-bar(x-bar) 0.36 0.48 0.16 1

s2 0 1250

ps2(s2) 0.52 0.48 1

DVD Example n=3

n=3

X1 X2 X3 p(x1,x2) x-bar s2

150 150 150 0.22 150.00 0.00150 150 200 0.14 166.67 833.33150 200 150 0.14 166.67 833.33150 200 200 0.10 183.33 833.33200 150 150 0.14 166.67 833.33200 150 200 0.10 183.33 833.33200 200 150 0.10 183.33 833.33200 200 200 0.06 200.00 0.00

x-bar 150 166.67 183.33 200px-bar(x-bar) 0.22 0.43 0.29 0.06 1E[X-bar] 170

s2 0 833.33ps2(s2) 0.28 0.72 1

V[X-bar] 200

Now, whatis the relationshipbetween meanand variance of originaldistribution X VersusX-bar?

Deriving Sampling Distributions for Continuous

Variables• Similar to discrete distributions, we can

also derive the sampling distributions of continuous variables.

22)()()(

)()(

XEdxxfXXV

dxxfXXE

x

x

Example: Two Exponential

• Exponential Distribution– f(x) = e- x

– E(X) = 1/ V(X) = 1/

• Suppose you have two independent rv’s, each following an exponential distribution and you are interested in the sum of the two rv’s (n=2).

• It can be shown that:

)2/(1)( ;/1)( 2 XVXE

Practical Applications

• For many well-known distributions, the sampling distributions of their primary statistics (mean, variance) have already been determined.

• In those cases where the sampling distribution is unknown or complicated, a very useful alternative is simulation.

Simulation Experiment

• To perform a simulation, you need:

– statistic of interest (e.g., X-bar, S, median, ..)

– population distribution (e.g., normal, uniform, ..)

– sample size n (e.g., n=10, n=100)

– number of k replications (e.g., k=500)

Simulation #1: Range Vs. S

• Conduct an experiment to determine the relationship between Range and S for n=2, n=5, and n=100.– Assume X ~ N(0,12)

Simulation #2: Jointly Distributed Variables with

Tolerance Stack-Up• Develop tolerances for the mean +/- 4 for

the volume of an engine cylinder whose:– bore ~ N(81 mm, 0.252 mm) and– stroke ~ N(83.5 mm, 0.202 mm)

• What is the volume equation?

IV. Distribution of Sample Mean/ Total

• Proposition -– Let X1, X2, .. Xn random sample from a distribution with mean

value and std deviation of , then:

– Let Total, To = X1 + X2 + .. Xn , then:

– Note difference between average and summing rv’s.

nXV

XE

X

X

/)(

)(

22

2)( )( nTVnTE oo

Sample Problem: Using the Avg or Sum of rv’s

• Let Y = # Parking Tickets issued on any given weekday.– Suppose Y has Poisson distribution with = 50.– Assuming you may approximate with normal,

a) What are the mean and variance of the avg # tickets per 5-day week?

b) What are the mean and variance of the sum of tickets per 5-day week?

c) What is the probability that the average # tickets per 5-day week is less than 48?

d) What is the probability that the total # tickets per 5-day week is between 225 and 275?

V. Central Limit Theorem (CLT)

• Theorem

– Let X1, X2, .. Xn be a random sample from a distribution with mean value and variance , and if n is sufficiently large, then

n

X

X

X

/

:with ondistributi normal aely approximat has

22

Rule of Thumb: n > 30.

But can be much less!

Understanding the CLT

• Using Minitab, let us generate 100 groups of service times (4 samples per group) from an exponential distribution with mean = 20 min.– Describe what is happening to the distribution?

150100500

100

50

0

exp-all

Fre

qu

ency

Histogram - 400 times

80706050403020100

20

10

0

Avg1-4

Fre

qu

ency

Histogram - 100 Group Avgs

Increasing sample size

• What is happening to the distribution of the sample averages? (Note: underlying distribution - exponential)

3532292623201714118

20

10

0

avg-e20

Fre

qu

ency

Average Multiple Distributions

• Suppose you have samples from 3 different distributions (e.g., exp, weibull, and uniform).– Minitab results from exponential ( = 20), weibull

(shape = 2, scale = 12) and uniform (20, 80).

10896847260483624120

100

50

0

stack4-6

Fre

qu

ency

55504540353025201510

25

20

15

10

5

0

Avg4-6

Fre

qu

ency

ALL 300 Observations Sample Averages

Summarizing the CLT• Regardless of the underlying distribution, averaging produces a

distribution which is more bell-shaped than before.

• Usefulness of CLT– If n becomes sufficiently large and we wish to compute a

probability of the sample mean, we may approximate with a normal.

– CLT provides analytical robustness!

• Issue of how robust depends on n and the underlying distribution -- the closer the underlying distribution resembles a normal (bell-shape) the smaller the n that is needed.

Other Applications

• Bernoulli Trials (Binomial Distribution)– Let a sample n consist of Xi Bernoulli trials (where each trial

equals 0 for failure, 1 for success).– As n (# of trials) becomes large and both:

• np > 10 and nq > 10 then the distribution of the sample mean (np) will become normally distributed.

– Consider the following example:• 10K bernoulli trials, if you group them in samples of size 100,

what will be the distribution of the groups?

Bernoulli Trial Example

• What does this experiment show about the importance of sample size, particularly for binary attributes?

0.330.310.290.270.250.230.210.190.170.15

20

10

0

bern-avg

Fre

qu

ency

Rules of Thumb with CLT

• How large a sample size do you need to invoke the CLT?

– Uniform n >= 4– Symmetric Triangular n >= 3– Normal n >= 1– Unimodal with extreme n >= 30

• (e.g., exponential)

– Discrete - apply normal approx rules• Binomial ~ np >= 10 for p < 0.5 (Or, np >= and nq >= 10)• Poisson ~ >= 15

VI. Distribution of Linear Combination (Independent

Xi’s)• Let X1, X2, .. Xn be a collection of random

variables with constraints a1, a2, .. an then,

• Linear Combination Y =

• If X1, X2, .. Xn are independent:

XnaXaXaY n...2211

)()...()()( 2211 XnEaXEaXEaYE n

)()...()()( 22

221

21 XnVaXVaXVaYV n

Differences Between Variables

• If Y = X1 - X2,

– E(X1 - X2) = a1E(X1) - a2E(X2)

– V(X1 - X2) = a12V(X1) + a2

2V(X2)

• Regardless of whether Xi are added or subtracted, the variances are additive!

Linear Combination: Tolerance

• Suppose you need to slide tube A into tube B.• What is the linear combination of assembly

clearance if tube A is N(24.8, 0.052) and tube B N(25, 0.052)?

• Assume the tube measurements are independent.

Tube A Tube BTube A Tube B

Weighted Linear Combination: Tolerance

• Suppose you are welding two pieces of metal together: a thick piece and a thin piece.– Let Xthin be the position of the thin piece.– Let Xthick be the position of the thick piece.

• From experience, you find the final position is based on the following:– Yassembly = 0.2 Xthin + 0.8 Xthick

– What is the expected variance of the assembly if the standard deviation thin piece is 0.4 mm, and the standard deviation of the thick piece is 0.15 mm? (assume the measurements of each piece is independent)