Post on 27-Mar-2015
STATISTICS POINT ESTIMATION
Professor Ke-Sheng ChengDepartment of Bioenvironmental Systems Engineering
National Taiwan University
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Annual flood peak water level (in cm)230 288 295282 275 462309 294 400249 245 299348 305 285360 375 330220 287 237295 210 278255 286 307195 500 300
2
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Is it originated from a normal (or gamma) distribution???
3
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Statistical inference(統計推論 )
• Given a random sample from the distribution of a population, we often are interested in making inferences about the population.
• Two important statistical inferences are – Estimation (推估 )– Test of hypotheses (假設檢定 ).
?
300Normal distribution?
4
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Parameter estimation
• Assume that some characteristics of the elements in a population can be represented by a RV X with pdf fX
( ． ;θ), where the “form” of the density is assumed known except that it contains some unknown parameters θ.
• We want to estimate an unknown parameter θ or some function of the unknown parameter, τ(θ), using observed values of a random sample, x1,x2,…,xn.
5
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• Point estimationLet the value of some statistic, say , represent the unknown parameter θ or τ(θ); such a statistic is called a point estimator.
),,( 1 nXXt
For example, and
respectively are point estimator of and .
n
iin x
nx
1
1
n
inin xx
ns
1
2)(1
1
6
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• Interval estimationTwo statistics and , where , so that
( , ) constitutes an interval for which the probability can be determined that it contains the unknown parameter θ or
τ(θ).
),,( 11 nXXt ),,( 12 nXXt
),,( 11 nXXt ),,( 12 nXXt ),,( 11 nXXt ),,( 12 nXXt
7
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Point Estimation of Distribution Parameters
• What parameters are to be estimated? – Parameters are variables that characterize
distributions. For example mean and standard deviation are parameters.
• How can we estimate the parameters?– methods of finding estimators – desired properties of point estimators
8
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• To estimate parameters we need to have a random sample. For example, a random sample (x1,x2,…,xn) of f( ． ;θ) is collected in order to estimate the parameter θ. Therefore,
• There can be many (or infinite) ways of estimation and we need to establish some kind of criteria in order to have an adequate estimator.
),...,,(ˆ21 nxxxt
9
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Estimators and Estimates
• Estimator: Any “statistic” whose values are used to estimate τ(θ), where τ( ． ) is some function of the parameter θ, is defined to be an estimator of . Note that an estimator is a random variable.
• Estimate: The estimated value given by an estimator.
)(
10
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Methods of finding estimator
• Assume that x1,x2,…,xn is a random sample from a density f( ． ;θ), where the form of the density is known but the parameter θ is unknown. Further assume that θ is a vector of real numbers, say θ=(θ1,θ2,…,θk).
• We often let , called the parameter space, denote the set of possible values that the parameter θ can assume. Our objective is to find statistics to be used as estimators of certain functions, say , of θ= (θ1,…,θk).
)(),(),( 21 r
11
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Method of moments
• Let f( ． ;θ1,θ2,…,θk) be the density of random variable X which has k parameters .
• In general will be a known function of the k parameters , i.e.
• Let x1, x2,…, xn be a random sample from the density f( ． ; ). From the k equations, we have
)( rr XE
k ,,1
k ,,1
k ,,1
12
krf krr ,...,2,1),...,(' 1
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• We can solve a solution of expressed in terms of x1, x2,…, xn. The solution, i.e.
, is called an estimator of .
krfXn
M kr
n
i
rir ,...,2,1)ˆ,...,ˆ(
1' 1
1
),,( 1 k
),,( 1 k )ˆ,,ˆ( 1 k
13
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 14
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Given a random sample of a normal distribution with mean μ and variance σ2. Estimate the parameters μ and σ by the method of moments.
ˆ1
','1
11
n
ini XX
nM
n
iiX
nM
1
2222
222122
1ˆˆ)(
n
XX
n
XnXXX
nX
nX
nn
ii
n
iin
ii
n
i
n
iii
1
2
2
1
2
2
1
2
1
2
1
22
)(
)()
1()
1(
1
15
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• Note that the method of moment estimator of σ2 is NOT the sample variance.
n
ii XX
n 1
22 )(1
n
ii XX
nS
1
22 )(1
1
16
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• Let x1, x2,…, xn be a random sample from a
Poisson distribution with parameter λ.
Estimate using MOM.
!);(
x
exf
x
,...2,1,0x
ˆ1''
111
n
iiX
nM
nX
17
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Let be a random sample from a uniform distribution on . What are the method of
moments estimator of and ?
nXXX ,...,, 21
]3,3[
)(XE
3
3
22
2
)(32
1
)()(
dxx
XEXVar
n
ii XX
nM
111 ˆ
1
22
1
22
222 ˆˆ
1
n
iiX
nM
n
ii XX
n 1
221
18
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Maximum Likelihood Method Definition of the likelihood function
19
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 20
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Rationale of using likelihood function for parameter estimation
21
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 22
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 23
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 24
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Definition of the maximum likelihood estimator
25
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 26
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 27
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Example
28
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 29
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 30
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Example
31
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Example
32
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
21 2
1
Lower bound for y1
Upper bound for yn
21
1 y 21ny
21
21
1 yyn
)(
)();(),...,;(
],[
1],[
11
21
121
21
21
yy
n
ii
n
iin
nI
xIxfxxL
33
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 34
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 35
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Example
36
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 37
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 38
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
)()()()32
1(
)()()32
1(),...,;,(
),[),
3[),
3[
]3,[1],3[1
11
1
nyyyn
nyy
nn
yIII
yIyIxxL
n
n
39
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 40
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 41
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Parameter estimators of various distributions
42
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 43
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Properties of point estimators
• Among different estimators, we want to know whether one estimator is “better” than others or what properties an estimator may or may not possess.
• Consider the case that we estimate using a statistic of a random sample from a density . Intuitively, we look for an estimator that is “close” to . However, what is the definition of “closeness”? (Or, how do we measure closeness?)
)(
)(
),...,( 1 nXXt);( xf
44
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Most concentrated estimator
45
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Pitman-closer and Pitman-closest
46
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Unbiased estimator
• An estimator ( ) of is said to be unbiased if
• An unbiased estimator is said to be more efficient than any other unbiased estimator of , if
for all .
.)ˆ( E
ˆ )ˆ()ˆ( VarVar
47
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Mean squared error of an estimator
48
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• Naturally, we would prefer to find an estimator that has the smallest mean-squared error, however, such estimators rarely exist. In general, the mean-squared error of an estimator depends on .
• What should we look for if a uniformly minimum MSE estimator rarely exists?
49
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Uniformly minimum MSE estimator
50
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• Let’s define = bias of T . [Note: is an estimator of .] Then,
2
22
22
22
12
1
)]()([)(
)]()([})]({[
})]()([)]()()][([2)]({[
])]}()([)]([{[})]({[
)),...,((})](),...,({[)(
TETVar
TETETE
TETETETTETE
TETETETE
XXtTLetXXtEMSE nnt
)]ˆ([)()()( bTETb )(T )ˆ(
22 )ˆ()()]([)()( bTVarTbTVarMSEt
51
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 52
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 53
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 54
Bias
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• We look for an estimator that has a uniformly minimum MSE within the class of unbiased estimators. Such an estimator is called a uniformly minimum-variance unbiased estimator (UMVUE).
55
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Example
56
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 57
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
• The MSE of is
n
ii XX
n 1
22 )(1
2
1
22
1
2
2
1
22
]})(1
[{])(1
[
}])(1
{[
n
ii
n
ii
n
ii
XXn
EXXn
Var
XXn
E
2
44
42
2
22222
)1
3(
1)1(
]1
[)()1
(
nn
n
nn
n
n
nSVar
n
nn
58
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 59
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Standard Error
• The “standard error” of a statistic is the standard deviation of its sampling distribution. If the standard error involves unknown parameters whose values can be estimated, substitution of those estimates into the standard error results in an “estimated standard error”.
60
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 61
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.
Mean-squared-error consistency
• We discuss the mean-squared error of an estimator derived from a random sample of “fixed” size n. Properties of point estimators that are defined for a “fixed” sample size are referred to as “small-sample” properties, whereas properties that are defined for increasing sample size are referred to as “large-sample” properties.
62
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 63
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 64
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 65
Properties of the maximum likelihood estimators
• Asymptotic properties of the MLEs
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 66
• Invariance property of the MLEs
• The invariance property does not hold for unbiasedness.
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 67
04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 68