Statistics Sampling Distributions and Point Estimation of Parameters Contents, figures, and...

StatisticsStatisticsSampling Distributions and Point Estimation of Parameters

Contents, figures, and exercises come from the textbook: Applied Statistics and Probability for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

StatisticA function of observations, ,

,…,Also a random variableSample meanSample variance Its probability distribution is

called a sampling distribution

Point EstimationPoint Estimation


1X 2X nX

X

2S

Point estimator of◦A statistic

Point estimate of◦A particular numerical value


),...,,(ˆ21 nXXXh

Mean ◦The estimate

Variance ◦The estimate

Proportion◦The estimate◦ is the number of items that belong to the class

of interestDifference in means,

◦The estimateDifference in two proportions

◦The estimate

x2

22ˆ sp

nxp /ˆ

x

21

2121 ˆˆˆˆ xx

21 pp

21 ˆˆ pp

Sampling Distributions and Sampling Distributions and the Central Limit Theoremthe Central Limit TheoremRandom sample

◦The random variables , ,…, are a random sample of size if (a) the ‘s are independent random variables, and (b) every has the same probability distribution

1X nX2Xn iX

iX

If , ,…, are normally and independently distributed with mean and variance◦ has a normal

distribution ◦with mean

◦variance


1X nX2X

nXXXX n

21

2

nX

nnX

2

2

2222

Central Limit Theorem◦If , ,…, is a random sample of

size taken from a population (either finite or infinite) with mean and finite variance , and if is the sample mean, the limiting form of the distribution of

◦as , is the standard normal distribution.

Works when◦ , regardless of the shape of the

population◦ , if not severely nonnormal

1X nX2X n

2X

nXZ

/

n

30n

30n

Two independent populations with means and , and variances and

◦is approximately standard normal, or◦is exactly standard normal if the two

populations are normal


21

21

22

2221

21

2121

//

)(

nn

XXZ

Example 7-1 Resistors◦An electronics company manufactures

resistors that have a mean resistance of 100 ohms and a standard deviation of 10 ohms. The distribution of resistance is normal. Find the probability that a random sample of resistors will have an average resistance less than 95 ohms

Example 7-2 Central Limit Theorem◦ Suppose that has a continuous uniform

distribution

◦Find the distribution of the sample mean of a random sample of size

25n

X

otherwise0

642/1)(

xxf

40n

Example 7-3 Aircraft Engine Life◦ The effective life of a component used in a jet-

turbine aircraft engine is a random variable with mean 5000 hours and standard deviation 40 hours. The distribution of effective life is fairly close to a normal distribution. The engine manufacturer introduces an improvement into the manufacturing process for this component that increases the mean life to 5050 hours and decreases the standard deviation to 30 hours. Suppose that a random sample of components is selected from the “old” process and a random sample of components is selected from the “improved” process. What is the probability that the difference in the two sample means is at least 25 hours? Assume that the old and improved processes can be regarded as independent populations.

161 n

252 n

12 XX

Exercise 7-10◦ Suppose that the random variable has

the continuous uniform distribution

◦ Suppose that a random sample of observations is selected from this distribution. What is the approximate probability distribution of ? Find the mean and variance of this quantity.


X

otherwise0

101)(

xxf

12n

6X

General Concepts of Point General Concepts of Point EstimationEstimationBias of the estimator

is an unbiased estimator if

Minimum variance unbiased estimator (MVUE)◦ For all unbiased estimator of , the one

with the smallest variance is the MVUE for

◦ If , ,…, are from a normal distribution with mean and variance

)ˆ(E

)ˆ(E

X

1X nX2X 2

Standard error of an estimator

Estimated standard error◦ or or If is normal with mean and variance

and


)ˆ(ˆ

V

s )ˆ(se

X

n/2

nX

nS

X

Mean squared error of an estimate

Relative efficiency of to


2

22

22

2

2

)()ˆ(

))ˆ((]))ˆ(ˆ[(

))ˆ(()]ˆ(ˆ[2]))ˆ([(]))ˆ(ˆ[(

]))ˆ()ˆ(ˆ[(

])ˆ[()ˆ(MSE

biasV

EEE

EEEEEEE

EEE

E

2 1

)ˆ(MSE)ˆ(MSE

2

1

Example 7-4 Sample Mean and Variance Are Unbiased◦ Suppose that is a random variable with

mean and variance . Let , ,…., be a random sample of size from the population represented by . Show that the sample mean and sample variance are unbiased estimators of and , respectively.

Example 7-5 Thermal Conductivity◦ Ten measurements of thermal conductivity

were obtained:◦ 41.60, 41.48, 42.34, 41.95, 41.86◦ 42.18, 41.72, 42.26, 41.81, 42.04◦ Show that and

X 2 1X 2X nXn

X X2S 2

0898.010284.0ˆ

ns

x924.41x

Exercise 7-31◦ and are the sample mean and sample

variance from a population with mean and variance . Similarly, and are the sample mean and sample variance from a second independent population with mean and variance . The sample sizes are and , respectively.

◦ (a) Show that is an unbiased estimator of ◦ ◦ (b) Find the standard error of . How

could you estimate the standard error?◦ (c) Suppose that both populations have the same

variance; that is, . Show that

◦ Is an unbiased estimator of .

1X2

1S1 2

1

2X22S

2 22 1n

2n

21 XX 21

21 XX

222

21

2)1()1(

21

222

2112

nn

SnSnS p2

Moments◦Let , ,…, be a random sample

from the probability distribution , where can be a discrete probability mass function or a continuous probability density function. The th population moment (or distribution moment) is , = 1, 2,…. The corresponding th sample moment is = 1, 2, ….


Methods of Point Methods of Point EstimationEstimation

nX2X1X)(xf)(xf

k

kk)( kXE

k

n

ikiXn 1

1

Moment estimators◦Let , ,…, be a random

sample from either a probability mass function or a probability density function with unknown parameters , ,…, . The moment estimators , ,…, are found by equating the first population moments to the first sample moments and solving the resulting equations for the unknown parameters.


nX2X1X

m

1 2 m

21 mm

m

Maximum likelihood estimator◦Suppose that is a random variable

with probability distribution , where is a single unknown parameter. Let , ,…, be the observed values in a random sample of size . Then the likelihood function of the sample is

◦Note that the likelihood function is now a function of only the unknown parameter . The maximum likelihood estimator (MLE) of is the value of that maximizes the likelihood function .

X

);( xf 1x 2x nx

n

);();();()( 21 nxfxfxfL

)(L

Properties of a Maximum Likelihood Estimator◦Under very general and not restrictive

conditions, when the sample size is large and if is the maximum likelihood estimator of the parameter ,

◦(1) is an approximately unbiased estimator for

◦(2) the variance of is neatly as small as the variance that could be obtained with any other estimator, and

◦(3) has an approximate normal distribution.


n

])ˆ([ E

Invariance property◦Let , ,…., be the

maximum likelihood estimators of the parameters , , …, . Then the maximum likelihood estimator of any function of these parameters is the same function

◦of the estimators , ,…, .


1 2 k

1 2

k),...,,( 21 kh

)ˆ,...,ˆ,ˆ( 21 kh

1 2 k

Bayesian estimation of parameters◦Sample , ,…, ◦Joint probability distribution

◦Prior distribution for

◦Posterior distribution for

◦Marginal distribution

)()|,...,,(),,...,,( 2121 fxxxfxxxf nn

nX2X1X

)(f

),...,,(),,...,,(),...,,|(

21

2121

n

nn xxxf

xxxfxxxf

continuous ),,...,,(

discrete ),,...,,(),...,,(

21

21

21θdxxxf

θxxxfxxxf

n

n

n

Example 7-6 Exponential Distribution Moment Estimator◦ Suppose that , ,…, is a random sample

from an exponential distribution with parameter . For the exponential, .

◦ Then results in .Example 7-7 Normal Distribution

Moment Estimators◦ Suppose that , ,…, is a random sample

from a normal distribution with parameters and . For the normal distribution, and

◦ . Equating to and to gives

◦ and◦ Solve these equations.

nX2X1X

)(XEXXE )( X/1 X/1ˆ

nX2X1X 2

)(XE222 )( XE )(XE X )( 2XE

n

i in X1

21

X

n

i in X1

2122

Example 7-8 Gamma Distribution Moment Estimators◦ Suppose that , ,…, is a random

sample from a gamma distribution with parameters and , For the gamma distribution, and

◦ . Solve◦ and


nX2X1Xr

/)( rXE 22 /)1()( rrXE

Xr /

n

i in Xrr1

212/)1(

Example 7-9 Bernoulli Distribution MLE◦ Let be a Bernoulli random variable. The

probability mass function is

◦ where is the parameter to be estimated. The likelihood function of a random sample of size is

◦ Find that maximizes .


X

otherwise01,0)1(

);(1 xpp

pxfxx

pn

n

ii

n

ii

ii

xnxn

i

xx pppppL 11 )1()1()(1

1

p )( pL

Example 7-10 Normal Distribution MLE◦ Let be normally distributed with unknown

and known variance . The likelihood function of a random sample of size , say , ,…, , is

◦ Find .Example 7-11 Exponential Distribution

MLE◦ Let be exponentially distributed with

parameter . The likelihood function of a random sample of size , say , ,…, , is

◦ Find .

X 2

n 1X 2X nX

n

ii

i

x

nx

n

i

eeL 1

2222

)(2

1

2/2)2/()(

1 )2(1

21)(

X n

1X 2X nX

n

ii

ix

nn

i

x eeL 1

1

)(

Example 7-12 Normal Distribution MLEs for and◦ Let be normally distributed with mean

and variance , where both and are unknown. The likelihood function of a random sample of size is

◦ Find and .Example 7-13

◦ From Example 7-12, to obtain the maximum likelihood estimator of the function

◦ Substitute the estimators and into the function , which yields

X 2

n

n

ii

i

x

nx

n

i

eeL 1

2222

)(2

1

2/2)2/()(

1

2

)2(1

21),(

2

2

2

22 ),(h 2

2/1

1

212 )(ˆˆ

n

iin XX

Example 7-14 Uniform Distribution MLE◦ Let be uniformly distributed on the

interval 0 to . Since the density function is for and zeros otherwise, the likelihood function of a random sample of size is

◦ for , ,…,◦ Find .


X aaxf /1)( ax0

n

n

inaa

aL1

11)(

ax 10 ax 20 axn 0a

Example 7-15 Gamma Distribution MLE◦ Let , ,…, be a random sample from

the gamma distribution. The log of likelihood function is

◦ Find that


nX2X1X

n

ii

n

ii

n

i

xri

r

xrnxrnr

rexrL

i

11

1

1

)](ln[)ln()1()ln(

)(ln),(ln

xrˆ

)ˆ()ˆ(')ln()ˆln(

1 rrnxn

n

ii

Example 7-16 Bayes Estimator for the Mean of a Normal Distribution◦ Let , ,…, be a random sample from

the normal distribution with mean and variance , where is unknown and is known. Assume that the prior distribution for is normal with mean and variance ; that is,

◦ The joint probability distribution of the sample is


nX2X1X 2

2 0

20

)2/()(

0

20

20

21)(

ef

n

iix

nn exxxf 1

22 )()2/1(

2/221 )2(1)|,...,,(

◦ Show that

◦ Then the Bayes estimate of is


2

/)/(

)/

11)(2/1(

21

220

022

022

0),...,,|(

nnx

nn exxxf

nnx/

)/(22

0

022

0

Exercise 7-42◦ Let , ,…, be uniformly distributed on

the interval 0 to . Recall that the maximum likelihood estimator of is .

◦ (a) Argue intuitively why cannot be an unbiased estimator for .

◦ (b) Suppose that . Is it reasonable that consistently underestimates ? Show that the bias in the estimator approaches zero as gets large.

◦ (c) Propose an unbiased estimator for .◦ (d) Let . Use the fact that if

and only if each to derive the cumulative distribution function of . Then show that the probability density function of is

nX2X1Xaa )max(ˆ iXa

aa

)1/()ˆ( nnaaEa a

n

a

)max( iXY yY yX i

YY

◦ Use this result to show that the maximum likelihood estimator for is biased.

◦ (e) We have two unbiased estimators for : the moment estimator and

◦ , where is the largest observation in a random sample of size . It can be shown that and that

◦ . Show that if , is a better estimator than . In what sense is it a better estimator of ?


otherwiae0

0)(1

ayany

yf n

n

aa

Xa 2ˆ1 )max(]/)1[(ˆ2 iXnna )max( iX

n)3/()ˆ( 2

1 naaV )]2(/[)ˆ( 2

2 nnaaV 1n 2a1a

a

Exercise 7-50◦ The time between failures of a machine

has an exponential distribution with parameter . Suppose that the prior distribution for is exponential with mean 100 hours. Two machines are observed, and the average time between failures is hours.

◦ (a) Find the Bayes estimate for .◦ (b) What proportion of the machine do you

think will fail before 1000 hours?


1125x

Statistics Sampling Distributions and Point Estimation of Parameters Contents, figures, and...

Documents

Transcript of Statistics Sampling Distributions and Point Estimation of Parameters Contents, figures, and...