Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that...
-
date post
21-Dec-2015 -
Category
Documents
-
view
216 -
download
0
Transcript of Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that...
![Page 1: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/1.jpg)
Topic4
Ordinary Least Squares
![Page 2: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/2.jpg)
• Suppose that X is a non-random variable• Y is a random variable that is affected by X in a
linear fashion and by the random variable with E() = 0That is,
E(Y) = + X
Or, Y = + X +
![Page 3: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/3.jpg)
O X
Y
..
. ..
Observed points
![Page 4: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/4.jpg)
O X
Y
ActualLine
. .Y= 1 + 2x..
.
![Page 5: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/5.jpg)
O X
Y
ActualLine
.Y= 1 + 2x..
..
![Page 6: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/6.jpg)
O X
Y
ActualLine
.Y= 1 + 2x.
.
..
![Page 7: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/7.jpg)
O X
Y
ActualLine
Y= 1 + 2x.
.
. ..
![Page 8: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/8.jpg)
O X
Y
ActualLine
Y= 1 + 2x
.
. ..
.
![Page 9: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/9.jpg)
O X
Y
. ActualLine
Y= 1 + 2x
.
. ..
![Page 10: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/10.jpg)
O X
Y
. ActualLine
Y= 1 + 2x. ..
Y= b1 + b2xFitted Line
.
BC is an error of EstimationAC is an effect of the random factor
C
B
. A.
![Page 11: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/11.jpg)
• The Ordinary Least Squares (OLS) estimates are obtained by minimising the sum of the squares of each of these errors.
• The OLS estimates are obtained from the values of X and the actual Y values (YA) as follows:
![Page 12: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/12.jpg)
Error of estimation (e) YA –YE |
where YE is the estimated value of Y.e2 YA
–YE ]2
e2 YA –(b1 + b2 X)]2
e2/b1 YA –(b1 + b2X)] (-1) =0
e2 /b2 YA –(b1 + b2X)] (-X) = 0
![Page 13: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/13.jpg)
Y –(b1 + b2X)] (-1) = 0
-NYMEAN + N b1 + b2NXMEAN = 0
b1 = YMEAN – b2XMEAN ….. (1)
![Page 14: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/14.jpg)
e2/b2 Y –(b1+ b2X)] (-X) = 0
Y –(b1 + b2X)] (-X) = 0
b1X –b2X2 = XY ………..(2)
b1 = YMEAN - b2XMEAN ….. (1)
![Page 15: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/15.jpg)
• These estimates are given below (with the superscripts for Y dropped).
^1 = (∑Y)(ΣX2) – (∑X)(∑XY)
N∑ X2 - (∑X)2
^2 = N∑YX – (∑X)(∑Y)
N∑ X2 - (∑X)2
![Page 16: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/16.jpg)
• Alternatively,
^1 = YMEAN - ^2XMEAN
^2 = Covariance(X,Y) Variance(X)
![Page 17: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/17.jpg)
(a) ei (Yi– YiE) = 0 and
(b) X2iei X2i(Yi– YiE) = 0
where YiE is the estimated value of Yi.
X2i is the same as Xi from before Proof: (Yi– YiE) = Yi– ^1 - ^2 X2i)
= Yi– ^1 - ^2 X2i
= nYMEAN – n^1 - n^2 XMEAN
= n(YMEAN – ^1 - ^2 XMEAN)
= 0 [ since ^1 = YMEAN - ^2XMEAN ]
Two Important Results
![Page 18: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/18.jpg)
See the lecture notes for a proof of part (b)
Total sum of squares (TSS) (Yi– YMEAN )
2
Residual sum of squares (RSS) (Yi– Yi
E )
2
Explained sum of squares (ESS) (Yi
E – YMEAN )
2
![Page 19: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/19.jpg)
To prove that
TSS = RSS + ESS
TSS ≡ (Yi– YMEAN)2
= {(Yi– YiE + Yi
E– YMEAN)}2
= (Yi– YiE)2 + (Yi
E– YMEAN)}2
(Yi– Yi E)(Yi
E– YMEAN)
= RSS + ESS (Yi– YiE)(Yi
E– YMEAN)
![Page 20: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/20.jpg)
(Yi– YiE)(Yi
E– YMEAN)
Yi– YiE)(Yi
E ) -YMEAN Yi– YiE)
Yi– YiE)(Yi
E ) [by (a) above]
Yi– YiE)(Yi
E ) = Yi– YiE)(^1^2
Xi)
= ^1 Yi– YiE)^2 XiYi– Yi
E)
= 0 [by (a) and (b) above]
![Page 21: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/21.jpg)
R2 ≡ ESS/TSS
Since TSS = RSS + ESS, it follows that
0 R2
![Page 22: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/22.jpg)
Topic 5
Properties of Estimators
![Page 23: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/23.jpg)
In the discussion that follows, ^ is an estimator of the parameter of interest,
Bias of ^ ≡ E(^) -
^ is unbiased if Bias of ^ = 0.
^ is negatively biased if Bias of ^ < 0.
^ is positively biased if Bias of ^ > 0.
![Page 24: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/24.jpg)
Mean Squared Errors (MSE) of estimation for ^ is given asMSE^ ≡ E[(^-)]2
MSE^ ≡ E[(^-)2]≡ E[{^-E(^) +E(^)-≡ E[{^-E(^)}2] + E[{E(^)- 2E[{^-E(^)}*{E(^)-≡ Var(^) + {E(^)- 2E[{^-E(^)}*{E(^)-
![Page 25: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/25.jpg)
Now, E[{^-E(^)}*{E(^)-
≡ {E(^)-E(^)}*{E(^)-
MSE^ ≡ Var(^) + {E(^)-
MSE^ ≡ Var(^) + (bias)2 .
≡ 0*{E(^)-
![Page 26: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/26.jpg)
If ^ is unbiased, that is, if E( ^)- = 0. then we have,
MSE^ ≡ Var(^)
An unbiased estimator ^ of a parameter is efficient if and only if it has the smallest variance of all unbiased estimatorsThat is, for any other unbiased estimator p of
Var(^)≤ Var(p)
![Page 27: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/27.jpg)
An estimator ^ is said to be consistent if it converges in probability to . That is,
Lim n Prob(|^- | > ) = 0 for every > 0.
![Page 28: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/28.jpg)
When the above condition holds, ^ is said to be the probability limit of , that is,plim^
Sufficient conditions for consistency: If the mean of ^convergesto and var(^) converges to zero (as n approaches ) then ^is consistent.
![Page 29: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/29.jpg)
That is, ^n is consistent if it can be shown that
Lim n E(^n
And Lim n Var(^n
![Page 30: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/30.jpg)
The Regression Model with TWO Variables
The Model :: Y = X +
Y is the DEPENDENT variable
X is the INDEPENDENT variable
Yi X1i X2i i
![Page 31: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/31.jpg)
The OLS estimates ^1 and ^2 are sample
statistics used to estimate 1and2 respectively
Yi X1i X2i i
Here X1i ≡ 1 for all i and X2 is
nothing but X .
![Page 32: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/32.jpg)
Assumptions about X2:
(1a) X2 is non-random (chosen by the
investigator) (1b) Random sampling is performed from a population of fixed values of X2 .
(1c) : Lim (1/n)x22i) = Q > 0
n [ where x2i X2i – X2MEAN.]
(1c) : Lim (1/n)X2i) = P > 0
n
![Page 33: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/33.jpg)
Assumptions about the disturbance term
2a. E() = 0
2b. Var(i) = 2 for all i.
2c. Cov(ij ) = 0 for i j. (The values
are uncorrelated across observations). 2d. The i all have a normal distribution
Homoskedasticity
![Page 34: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/34.jpg)
Result^2 is linear in the dependent variable Yi
^2 = Covariance(X,Y)
Variance(X)
^2 = Yi–YMEAN )Xi–XMEAN )
Xi–XMEAN )2
Proof:
![Page 35: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/35.jpg)
^2 = YiXi–XMEAN )
Xi–XMEAN )2
+ K
CiYiK
where the Ci andK are constants
![Page 36: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/36.jpg)
Therefore,
^2 is a linear function of Yi
Since, Yi
X1i X2i i
^2 is a linear function of i and hence
is normally distributed
![Page 37: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/37.jpg)
Similarly,
^1 is a linear function of Yi (and
hence i ) and is normally distributed
Both ^1 and ^2 are unbiased estimates of 1 and 2 respectively.
That is, E( ^1 ) = 1 and
E( ^2 ) = 2
![Page 38: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/38.jpg)
Each of ^1 and ^2 is an efficient estimators of 1 and 2 respectively.
Thus, each of ^1 and ^2 is a
Best (efficient)
Linear (in the dependent variable Yi )
Unbiased
Estimator of 1 and 2 respectively.
Each of ^1 and ^2 is a consistent
estimator of 1 and 2 respectively.
Also,
![Page 39: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/39.jpg)
Var(^1 ) = (1/n +X 2mean2x2i
2)
Var(^2 ) = x2i2)
. Cov(^1, ^2 ) = -X 2meanx2i2
![Page 40: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/40.jpg)
LimVar(^2 )
n = Lim x2i
2
n = Lim /nx2i
2/n
n = 0/Q [using assumption (1c)]
= 0
![Page 41: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/41.jpg)
Because ^2 is an unbiased estimator of 2 and
LimVar(^2 ) = 0
n
^2 is a consistent estimator of 2
![Page 42: Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random.](https://reader030.fdocuments.net/reader030/viewer/2022032522/56649d6d5503460f94a4d0ea/html5/thumbnails/42.jpg)
The variance of the random term, , is not known
To perform statistical analysis, we estimate by
^2 RSS/(n-2)
This is because ^2 is an unbiased estimator of 2