Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson11-1 Lesson 11: Regressions Part II.
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson10-1 Lesson 10: Regressions Part I.
-
date post
19-Dec-2015 -
Category
Documents
-
view
216 -
download
2
Transcript of Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Lesson10-1 Lesson 10: Regressions Part I.
Lesson10-1 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Lesson 10:
Regressions Part IRegressions Part I
Lesson10-2 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Outline
Correlation Analysis
Regression Analysis
Standard error of estimate
Confidence interval and prediction interval
Inference about the regression slope
Cautions about the interpretation of significance
Evaluating the model
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Correlation Analysis
Correlation Analysis is a group of statistical techniques used to measure the strength of the association between two variables.
A Scatter Diagram is a chart that portrays the relationship between the two variables.
The Dependent Variable is the variable being predicted or estimated.
The Independent Variable provides the basis for estimation. It is the predictor variable.
Lesson10-4 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Example
Suppose a university administrator wishes to determine whether any relationship exists between a student’s score on an entrance examination and that student’s cumulative GPA. A sample of eight students is taken. The results are shown below
Student Exam Score GPA
A 74 2.6
B 69 2.2
C 85 3.4
D 63 2.3
E 82 3.1
F 60 2.1
G 79 3.2
H 91 3.8
Lesson10-5 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Scatter Diagram: GPA vs. Exam Score
| | | | | | | | | |
50 55 60 65 70 75 80 85 90 95
Exam Score
4.00 -3.75 -3.50 -3.25 -3.00 -2.75 -2.50 -2.25 -2.00 -
Cu
mu
lati
ve G
PA
We would like to know whether there is a strong linear relationship between the two variables.
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
The Coefficient of Correlation, r
The Coefficient of Correlation (r) is a measure of the strength of the linear relationship between two variables. It can range from -1.00 to 1.00. Values of -1.00 or 1.00 indicate perfect and strong
correlation. Values close to 0.0 indicate weak correlation. Negative values indicate an inverse relationship
and positive values indicate a direct relationship.
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Formula for r
n
1i
2i
n
1i
2i
n
1iii
n
1i
2i
n
1i
2i
n
1iii
yx
2xy
)yy()xx(
)yy)(xx(
)1n(
)yy(
)1n(
)xx(
)1n(
)yy)(xx(
ss
sr
We calculate the coefficient of correlation from the following formulas.
Sample covariance between x and y
Sample standard deviation of x
Sample standard deviation of y
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Coefficient of Determination
The coefficient of determination (r2) is the proportion of the total variation in the dependent variable (Y) that is explained or accounted for by the variation in the independent variable (X). It is the square of the coefficient of correlation. It ranges from 0 to 1. It does not give any information on the direction of the
relationship between the variables.
Special cases: No correlation: r=0, r2=0. Perfect negative correlation: r=-1, r2=1. Perfect positive correlation: r=+1, r2=1.
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
EXAMPLE 1
Dan Ireland, the student body president at Toledo State University, is concerned about the cost to students of textbooks. He believes there is a relationship between the number of pages in the text and the selling price of the book. To provide insight into the problem he selects a sample of eight textbooks currently on sale in the bookstore. Draw a scatter diagram. Compute the correlation coefficient.
Book Page Price ($)
Intro to History 500 84
Basic Algebra 700 75
Intro to Psyc 800 99
Intro to Sociology 600 72
Bus. Mgt. 400 69
Intro to Biology 500 81
Fund. of Jazz 600 63
Princ. Of Nursing 800 93
Lesson10-10 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Example 1 continued
400 500 600 700 800
60
70
80
90
100
Page
Scatter Diagram of Number of Pages and Selling Price of Text
Price ($)
Lesson10-11 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Example 1 continued
Book Page Price ($)
X Y
Intro to History 500 84
Basic Algebra 700 75
Intro to Psyc 800 99
Intro to Sociology 600 72
Bus. Mgt. 400 69
Intro to Biology 500 81
Fund. of Jazz 600 63
Princ. Of Nursing 800 93
Total 4,900 636
n
1i
2i
n
1i
2i
n
1iii
)yy()xx(
)yy)(xx(r
The correlation between the number of pages and the selling price of the book is 0.614. This indicates a moderate association between the variable.
Lesson10-12 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
EXAMPLE 1 continued
Is there a linear relation between number of pages and price of books?
Test the hypothesis that there is no correlation in the population. Use a .02 significance level.
Under the null hypothesis that there is no correlation in the population. The statistic
)2/()1( 2
nr
rt
follows student t-distribution with (n-2) degree of freedom.
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
EXAMPLE 1 continued
Step 1: H0: The correlation in the population is zero. H1: The correlation in the population is not zero.
Step 2: H0 is rejected if t>3.143 or if t<-3.143. There are 6 degrees of freedom, found by n – 2 = 8 – 2 = 6.
Step 3: To find the value of the test statistic we use:
905.1)614(.1
28614.
)2/()1( 22
nr
rt
Step 4: H0 is not rejected. We cannot reject the hypothesis that there is no correlation in the population. The amount of association could be due to chance.
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Regression Analysis
In regression analysis we use the independent variable (X) to estimate the dependent variable (Y).
The relationship between the variables is linear.
Lesson10-15 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Simple Linear Regression Model
Relationship Between Variables Is a Linear Function
iii XY 10
Y intercept Slope Random Error
Dependent (Response) Variable
Independent (Explanatory) Variable x
y
0 Run
Rise
1 = Rise/Run
0 and 1 are unknown,therefore, are estimated from the data.
Lesson10-16 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Finance Application: Market Model
One of the most important applications of linear regression is the market model.
It is assumed that rate of return on a stock (R) is linearly related to the rate of return on the overall market (Rm).
Rate of return on a particular stock
Rate of return on some major stock index
The beta coefficient measures how sensitive the stock’s rate of return is to changes in the level of the overall market.
R = 0 + 1Rm +
Lesson10-17 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
EstimationMethod of moments
Given a set of data (x1,y1),…,(xn,yn), how should we estimate the parameters 0 and 1?
Start from the simple case – suppose 1 is known to be 0. That is, we have only one parameter to estimate. yi=0 + i
Possible assumption #1 E() = 5. That is E(y) =0 + E() = 0 + 5
Possible assumption #2 E() = 0. That is E(y) =0 + E() = 0
0 has the interpretation of mean of y.
Better assumption!!
Lesson10-18 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 0
Method of moments
Given a set of data (x1,y1),…,(xn,yn), how should we estimate the parameters 0?
yi=0 + i
Assumption: E() = 0. That is E(y) =0 + E() = 0
0 has the interpretation of mean of y.
Use the sample analog # 1: E() = 0 implies 0 = E(y)
Take the estimate as b0 = (y1 + y2 + … + yn)/n = ∑ yi /n
Lesson10-19 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 0
Method of moments
Given a set of data (x1,y1),…,(xn,yn), how should we estimate the parameters 0?
yi=0 + i
Assumption: E() = 0. That is E(y) =0 + E() = 0
0 has the interpretation of mean of y.
Use the sample analog # 1: E() = 0 in population. Set the (e1 + e2 + … + en )/n = 0
(y1- b0) + (y2- b0) + … + (yn- b0) = 0
(y1 + y2 + … + yn) - nb0 = 0
b0 = (y1 + y2 + … + yn)/n = ∑ yi /n
(yn-b0): deviation of y from the assumed value of 0
Lesson10-20 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 1
Method of moments
Given a set of data (x1,y1),…,(xn,yn), how should we estimate the parameters 0 and 1?
Start from another simple case – suppose 0 is known to be 0. That is, we have only one parameter to estimate. yi=1xi + i
Possible assumption #1 E() = 0. That is E(y) =1E(x) + E() = 1E(x) and
1=E(y)/E(x) Possible assumption #2
E(x) = 0. That is E(yx) =1E(x2) + E(x) = 1E(x2) and 1=E(yx)/E(x2)
Lesson10-21 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 0 and 1
Method of moments
Given a set of data (x1,y1),…,(xn,yn), how should we estimate the parameters 0 and 1?
yi=0 + 1 xi+ i
Assumption #1 E() = 0.
If x is known and non-random: E(y) =0 + 1 x + E() = 0 + 1 x
If x is unknown and random: E(y) =0 + 1 E(x) + E() = 0 + 1 E(x)
This assumption gives us only one equation. Need an extra equation….
Assumption #2….
Lesson10-22 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 0 and 1
Method of moments
Given a set of data (x1,y1),…,(xn,yn), how should we estimate the parameters 0 and 1?
yi=0 + 1 xi+ i
Assumption #1 E() = 0 implies E(y) – 0 – 1 E(x) = 0
Assumption #2 How about E(x) = 0? Since Cov(x) = E(x) – E()E(x) = E(x), the
assumption really imply and x are uncorrelated. E(x) =0 implies E[(y – 0 – 1x)x]=0
E[yx – 0x – 1x2]= E(yx) – 0E(x) – 1E(x2) = 0Two equations are adequate to solve two unknowns.
Lesson10-23 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 0 and 1
Method of moments
Two equations are adequate to solve two unknowns. E(y) – 0 – 1 E(x) = 0
E(y) E(x) – 0 E(x) – 1 E(x)2 = 0
E(yx) – 0E(x) – 1E(x2) = 0
E(yx) – E(y) E(x) – 1E(x2) + 1 E(x)2 = 0
Cov(x,y) – 1Var(x) = 0
1= Cov(x,y)/Var(x)
E(y) – 0 – 1 E(x) = 0
0 = E(y)– 1 E(x)
Lesson10-24 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 0 and 1
Method of moments
The two assumptions: Assumption #1: E() = 0 Assumption #2: E(x) = 0
implies 1= Cov(x,y)/Var(x) and 0 = E(y)– 1 E(x)
To estimate the parameters, use the sample analog: Suffice to compute the
Sample covariance between x and y, Sx,y
Sample variance of x, Sxx
Sample mean of y, my
Sample mean of x, mx
and use them to replace the corresponding population quantities.
Lesson10-25 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 0 and 1
Method of moments
The two assumptions: Assumption #1: E() = 0 Assumption #2: E(x) = 0
To estimate the parameters, we can also use the sample analog of the two assumptions directly: Sample analog of E() = 0:
∑ (yi – b0 – b1 xi) =0∑yi– nb0 – b1 ∑xi =0
Sample analog of E(x) = 0?∑ (yi – b0 – b1 xi)xi =0∑(yi xi) – b0 ∑xi – b1 ∑xi
2 =0
Lesson10-26 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation of 0 and 1
Which way is better?
Matching the moments of the two assumptions directly: Assumption #1: E() = 0 Assumption #2: E(x) = 0
Or the implied parameters in terms of the moments 1= Cov(x,y)/Var(x) and 0 = E(y)– 1 E(x)
It is not surprise that both approaches yield the same estimator. b1 = Sxy / Sxx and b0 = my – b1 mx
Lesson10-27 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
E(x) = 0 one assumption implies two
By law of iterated expectations E(x) = 0 implies E [E(x) ] = E[ ] = 0
(assumption #1) E(x) = 0 implies E(xx) = 0,
implies E[ E(xx) ] = E[ x ] = 0 (assumption #2)
What does E(x) = 0 mean? E(x) =0 E[ (y – 0 – 1 x) | x] =
E (y|x) –0 – 1 x = E (y|x) = 0 + 1 x
That the expectation of y conditional on x is 0 + 1 x
Lesson10-28 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
What if the assumption E(x) = 0 fails?
Still have: E[ ] = 0 (assumption #1)
Need to find another moment conditions, say another variables z such that E[ z ] = 0 (assumption #2’)
instrumental variable
Lesson10-29 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Maximum likelihood
Maximum likelihood estimation (MLE) is a popular statistical method used to make inferences about parameters of the underlying probability distribution from a given data set. That is to say, you have a sample of data x1, x2, …, xn
and you want to infer the distribution of the random variable x.
Commonly, one assumes the data are independent, identically distributed (iid) drawn from a particular distribution with unknown parameters and uses the MLE technique to create estimators for the unknown parameters, that define the distribution.
Lesson10-30 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Maximum likelihood
Given a sample of data x1, x2, …, xn
We would like to find the distribution P0 such that for all feasible distribution P
Prob(P0 | x1,x2,…,xn) Prob(P | x1,x2,…,xn)
For example, the distribution P0 = N(0,02)
Lesson10-31 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
An example: Maximum likelihood
Consider tossing an unfair coin 80 times (i.e., we sample something like x1=H, x2=T, ..., x80=T, and count the number of HEADS "H" observed). Call the probability of tossing a HEAD p, and the probability of tossing TAILS 1-p (so here p is θ above).
Suppose we toss 49 HEADS and 31 TAILS, and suppose the coin was taken from a box containing three coins: one which gives HEADS with probability p=1/3, one which gives HEADS with probability p=1/2 and another which gives HEADS with probability p=2/3.
The coins have lost their labels, so we don't know which one it was. Using maximum likelihood estimation we can calculate which coin has the largest likelihood, given the data that we observed.
Lesson10-32 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
An example: Maximum likelihood
The likelihood function (defined below) takes one of three values: Pr(H=49 | p=1/3) = C(80,49) (1/3)49 (1-1/3)31≈0.000 Pr(H=49 | p=1/2) = C(80,49) (1/2)49 (1-1/2)31≈0.012 Pr(H=49 | p=2/3) = C(80,49) (2/3)49 (1-2/3)31≈0.054
Which coin is more likely? The one with p=2/3
because among the three possibilities, the observation of the sample data has the highest probability when p=2/3.
Lesson10-33 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
An example: Maximum likelihood
Now suppose we had only one coin but its p could have been any value 0 ≤ p ≤ 1. We must maximize the likelihood function:
L(p) = C(80,49) p49 (1-p)31
Over all possible values of 0 ≤ p ≤ 1.
To find the maximum, take the first derivative of L(p) with respect to p and set to zero.
p48(1-p)30(49-80p) = 0 Possible solutions of p are:
P=0, P=1 and p=49/80. L(0)=L(1)=0. L(49/80) >0. Hence the maximum likelihood
estimator for p is 49/80.
Lesson10-34 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
L(p) = C(80,49) p49 (1-p)31
over all possible values of 0 ≤ p ≤ 1.
Lesson10-35 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
To estimate 0 and 1 using ML
Assume i to be independent identically distributed with normal distribution of zero mean and variance 2. Denote the normal density for i be
f(i)=f(yi-0-1xi)
normal density (which has a rather ugly formula)
The joint likelihood of observing 1, 2, …, n:
L = f(1)*f(2)*…*f(n)
i = yi - 0 - 1xi
Lesson10-36 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
To estimate 0 and 1 using ML (Computer)
We do not know 0 and 1. Nor do we know i. In fact, our objective is estimate 0 and 1.
The procedure of ML:
1. Assume a combination of 0 and 1, call it b0 and b1. Compute the implied ei = yi-b0-b1xi and f(ei)=f(yi-b0-b1xi)
2. Compute the joint likelihood conditional on the assumed values of b0 and b1:
L(b0,b1) = f(e1)*f(e2)*…*f(en)
Assume many more combination of 0 and 1, and repeat the above two steps, using a computer program (such as Excel).
Choose the b0 and b1 that yield a largest joint likelihood.
Lesson10-37 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
To estimate 0 and 1 using ML (Calculus)
The procedure of ML:
1. Assume a combination of 0 and 1, call it b0 and b1. Compute the implied ei = yi-b0-b1xi and f(ei)=f(yi-b0-b1xi)
2. Compute the joint likelihood conditional on the assumed values of b0 and b1:
L(b0,b1) = f(e1)*f(e2)*…*f(en)
Choose b0 and b1 to maximize the likelihood function L(b0,b1) – using calculus. Take the first derivative of L(b0,b1) with respect to b0, set it
to zero. Take the first derivative of L(b0,b1) with respect to b1, set it
to zero. Solve b0 and b1 using the two equations.
Lesson10-38 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
MM versus ML in the estimation of 0 and 1
It turns out that in this special case, the estimators b0 and b1 turn out to the same as the one obtained by method of moments and OLS (to be discussed later).
Lesson10-39 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Alternative assumptionX and Y are bivariate normal
Suppose (X1, X2) ~ BVN(1,2,12,2
2, 12),
The marginal distribution of X1 is normal: X1~N(1,1
2)
The conditional distribution of X2 given x1 is normal:
X2|x1 ~ N( + x1, 2)where = 2-1, = 1
2/12, 2 = 2
2(1-2)=22-21
2
Reference: Goldberger, Arthur S. (1991): A course in Econometrics, Harvard University Press. See page 75.
E(X2|X1)
Lesson10-40 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Estimation Ordinary least squares
For each value of X, there is a group of Y values, and these Y values are normally distributed.
Yi~ N(E(Y|X),i2), i=1,2,…,n
The means of these normal distributions of Y values all lie on the straight line of regression.
E(Y|X) = 0+1X
The standard deviations of these normal distributions are equal.
i2= 2 i=1,2,…,n
i.e., homoskedasticity
Lesson10-41 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Assumptions Underlying Linear Regression
yi and yk are independently drawn from the population, say, as in sampling with replacement. Cov(i,j) = 0 for all i ≠ j
Note that independence implies much more than zero covariance. For two discrete random variables, they are
independent if P(x,y) = P(x)P(y)
Lesson10-42 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Choosing the line that fits best
The question is:Which straight line fits best?
x
y
Lesson10-43 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data3
3
41
1
4
(1,2)
2
2
(2,4)
(3,1.5)
Sum of squared differences = (2 - 1)2 + (4 - 2)2 + (1.5 - 3)2 +
(4,3.2)
(3.2 - 4)2 = 6.89Sum of squared differences = (2 -2.5)2 + (4 - 2.5)2 + (1.5 - 2.5)2 + (3.2 - 2.5)2 = 3.99
2.5
Let us compare two lines
The second line is horizontal
The smaller the sum of squared differences the better the fit of the line to the data. That is, the line with the least sum of squares (of differences) will fit the line best.
The best line is the one that minimizes the sum of squared vertical differences between the points and the line.
Choosing the line that fits best
Lesson10-44 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Choosing the line that fits bestOrdinary Least Squares (OLS) Principle
Straight lines can be described generally by Y = b0 + b1X
Finding the best line with smallest sum of squared difference is the same as
Let b0* and b1
* be the solution of the above problem.
Y* = b0* + b1
*Xis known as the “average predicted value” (or simply “predicted value”) of y for any X.
Min S(b0,b1) = [yi – (b0 + b1xi)]2
Lesson10-45 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Coefficient estimates from the ordinary least squares (OLS) principle
Solving the minimization problem implies the first order conditions:
S(b0,b1) = [yi – (b0 + b1xi)]2
∂S(b0,b1) /∂ b0= (2)[yi – (b0 + b1xi)](-1) = 0
yi – (b0 + b1xi)] = 0
∂S(b0,b1) /∂ b1= (2)[yi – (b0 + b1xi)](-xi) = 0
[yi – (b0 + b1xi)](xi) = 0
ei = 0
ei xi = 0
The same as what we had earlier in the discussion of method of moments.
Lesson10-46 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
The consistency of b0 and b1 as estimators of 0 and 1
Same equations implies same estimator: b1 = Sxy / Sxx and b0 = my – b1 mx
The estimators are consistent as long as Sxy, Sxx, my and mx
are consistent estimators of the corresponding population quantities. Sxy converges to Cov(x,y) as sample size increases.
Sxx converges to Var(x) as sample size increases.
my converges to E(y) as sample size increases.
mx converges to E(x) as sample size increases
Lesson10-47 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
The b0 and b1 are linear in yi
n
1i iii
n
1i n
1i
2i
i
n
1i
2i
n
1i ii
n
1i n
1i
2i
i
n
1i
2i
n
1i i
n
1i
2i
n
1i ii
n
1i
2i
n
1i ii1
ycyxx
xx
yxx
xxy
xx
xx
xx
yxx
xx
yxx
xx
yyxxb
)(
)(
)(
)(
)(
)(
)(
)(
)(
)(
)(
))((
Similarly, it can be shown that b0 is also a linear combination of yi.
Lesson10-48 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
The unbiasedness of b0 and b1 as estimators of 0 and 1
The estimator.
b1 = ci yi
E(b1|x) = ci E(yi) = ci E(0+1xi+i) = 0 ci +1 ci xi+ ci E(i) = 0 +1 + = 1
n
1i
2i
ii
)xx(
)xx(c
0)xx(
)xx(
)xx(
)xx(c n
1i
2i
n
1iin
1in
1i
2i
in
1ii
-
-
-
-
1)xx(
)xx(
)xx(
x)xx(
)xx(
x)xx(
)xx(
x)xx(xc
n
1i
2i
n
1i
2i
n
1i n
1i
2i
in
1i n
1i
2i
ii
n
1i n
1i
2i
iii
n
1i i
Lesson10-49 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Best estimator
It can also be shown that the estimators b0 and b1 have the smallest variance among all unbiased estimators that are linear combination of yi.
That is, if there is another unbiased estimator b1’ that is linear combination of yi, say, b1’ = ci’ yi, we must have
Var(b1) Var(b1’)
Lesson10-50 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
BLUE
Best: smallest variance Linear: linear combination of yi
Unbiased: E(b0) = 0, E(b1) = 1
Estimator
Lesson10-51 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
yi = b0 + b1xi + ei
1. Slope (b1)
Estimated Y changes by b1 for each 1 unit increase in X
y* + y*= b0 + b1(x+1) y*= b1
More generally,y* + y*= b0 + b1(x+x) y*/x = b1
Y-Intercept (b0 )
1. Estimated value of Y when X = 0
Interpretation of Coefficients
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
EXAMPLE 2 continued from Example 1
Develop a regression equation for the information given in EXAMPLE 1. The information there can be used to estimate the selling price based on the number of pages.
The regression equation is:
Y* = 48.0 + .05143X
The equation crosses the Y-axis at $48. A book with no pages would cost $48.
The slope of the line is .05143. Each additional page costs about $0.05 or five cents.
Note: the sign of the b value and the sign of r will always be the same.
Lesson10-53 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Result part 1
Coefficients Standard Error t Stat P-value
Intercept 48 16.94157037 2.833267456 0.029829
X 0.051428571 0.026998733 1.904851306 0.105458
=0.051428571/ 0.026998733=(b1-0)/Sb1
Pr(t>t Stat)
H0: 1 = 0
b1 Sb1
Lesson10-54 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Example 2 continued from Example 1
We can use the regression equation to estimate values of Y.
The estimated selling price of an 800 page book is $89.14, found by
Y* = 48.0 + .05143X = 48.0 + .05143(800) = 89.14
Lesson10-55 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Standard Error of Estimate (denoted se or Sy.x)
Additional assumption: Var(i) = Var(j) = Var() = 2 for all i,j
Recall yi=0 + 1 xi+ i
Var() = 0 implies y are perfectly linear with x.
Lesson10-56 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
More Accurate Estimatorof X, Y Relationship
Less Accurate Estimatorof X, Y Relationship
Scatter Around the Regression Line
Small Var() Large Var()
Lesson10-57 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Standard Error of Estimate (denoted se or Sy.x)
Additional assumption: Var(i) = Var(j) = 2 for all i,j.
Se2 is a estimate of
2.
Hence, it may be interpreted as a measure of the reliability of the estimating equation
A measure of dispersion Measures the variability, or scatter of the observed values
around the regression line
2n
)xbby(
2n
)yy(ss
2i10
n
1ii
2n
1i
*ii
x.ye
Lesson10-58 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Assumptions: Observed Y values are normally distributed
around each estimated value of Y*
Constant variance
se measures the dispersion of the points around the regression line If se = 0, equation is a “perfect” estimator
se may be used to compute confidence intervals of the estimated value
Interpreting the Standard Error of the Estimate
Lesson10-59 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
X1
X2
X
Y
f(e)y values are normally distributed around the regression line.
For each x value, the “spread” or variance around the regression line is the same.
Regression Line
Variation of Errors Around the Regression Line
Lesson10-60 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
X2
X
Y
Variation of Errors Around the Regression Line
E(y|x)=0+1x
y is distributed normal with mean E(y|x)=0+1x, and variance 2.Confidence intervals may be constructed in the usual fashion.
Lesson10-61 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
De
pen
de
nt
Va
ria
ble
( Y
)
Independent Variable (X)
Y = b0 + b1X + 2se
Y = b0 + b1X + 1se
Y = b0 + b1X - 1se
Y = b0 + b1X - 2se
Y = b0 + b1X regression line
2se (95.5% Lie in this Region)
Scatter around the Regression Line
1se (68% Lie in this Region)
Lesson10-62 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Example 3 continued from Example 1 and 2.
Find the standard error of estimate for the problem involving the number of pages in a book and the selling price.
408.1028
)200,397(05143.0)636(48606,51
2n
)xbby(s
2i10
n
1ii
e
Lesson10-63 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Equations for the Interval Estimates
htsy e*
2
1
2
)(
)(1
xx
xx
nh
n
ii
htsy e 1*
Confidence Interval for the Mean of y
Prediction Interval for y
y*=b0 + b1 x
y=b0 + b1 x + e
Minimized when
xx
Lesson10-64 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Confidence Interval Estimate for Mean Response
X
The following factors influence the width of the interval: Std Error, Sample Size, X Value
y* = b0+b1xi
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Confidence Interval continued from Example 1, 2 and 3.
For books of 800 pages long, what is that 95% confidence interval for the mean price? This calls for a confidence interval on the
average price of books of 800 pages long.
31.1514.898
)4900(000,150,3
)5.612800(
8
1)408.10(447.214.89
)(
)(1
2
2
2
1
2**
xx
xx
ntsyhtsy
n
ii
ee
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Prediction Interval continued from Example 1, 2 and 3.
For a book of 800 pages long, what is the 95% prediction interval for its price? This calls for a prediction interval on the price of an
individual book of 800 pages long.
72.2914.898
)4900(000,150,3
)5.612800(
8
11)408.10(447.214.89
)(
)(111
2
2
2
1
2**
xx
xx
ntsyhtsy
n
ii
ee
Lesson10-67 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
1. Tests if there is a linear relationship between X & Y
2. Involves population slope
3. Hypotheses H0: = 0 (no linear relationship)
H1: 0 (linear relationship)
4. Theoretical basis is sampling distribution of slopes
Test of Slope Coefficient (b1)
Lesson10-68 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Sampling Distribution of the Least Squares Coefficient Estimator
If the standard least squares assumptions hold, then b1 is an unbiased estimator of 1 and has a population variance
2
2
1
2
22
)1()(
1
xn
ii
b snxx
and an unbiased sample variance estimator
2
2
1
2
22
)1()(
1
x
en
ii
eb sn
s
xx
ss
n
1i iii
n
1i n
1i
2i
i1 ycy
)xx(
)xx(b
Lesson10-69 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Basis for Inference About the Population Regression Slope
Let 1 be a population regression slope and b1 its least squares estimate based on n pairs of sample observations. Then, if the standard regression assumptions hold and it can also be assumed that the errors i are normally distributed, the random variable
1
11 -
bs
bt
is distributed as Student’s t with (n – 2) degrees of freedom. In addition the central limit theorem enables us to conclude that this result is approximately valid for a wide range of non-normal distributions and large sample sizes, n.
Lesson10-70 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Tests of the Population Regression Slope
If the regression errors i are normally distributed and the standard least squares assumptions hold (or if the distribution of b1 is approximately normal), the following tests have significance value :
1. To test either null hypothesis H0: 1 = 1
* or H0:1 1*
against the alternative H1: 1 > 1
*
The decision rule is to reject if
2),(n-b
*11 t
sβ-b
t1
Lesson10-71 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Tests of the Population Regression Slope
2. To test either null hypothesis H0: 1 = 1
* or H0:1 > 1*
against the alternative H1: 1 1
*
the decision rule is to reject if
2),(nb
*11 t
sβ-b
t1
-≤
Lesson10-72 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Tests of the Population Regression Slope
3. To test either null hypothesis H0: 1 = 1
* against the alternative H1: 1 1
*
the decision rule is to reject if
/22),(nb
*11
/22),(nb
*11 t-
sβ-b
t or t s
β-bt
11
-- ≤≥
/22),(n-b
*11 t
sβb
t 1
≥ -
Equivalently
Lesson10-73 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Confidence Intervals for the Population Regression Slope 1
If the regression errors i , are normally distributed and the standard regression assumptions hold, a 100(1 - )% confidence interval for the population regression slope 1 is given by
11 b/22),(n11b/22),(n-1 stbβst-b -
Lesson10-74 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Some cautions about the interpretation of significance tests
Rejecting H0: b1 = 0 and concluding that the relationship between x and y is significant does not enable us to conclude that a cause-and-effect relationship is present between x and y.
Causation requires: Association Accurate time sequence Other explanation for correlation
Correlation Causation Correlation Causation
Lesson10-75 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Some cautions about the interpretation of significance tests
Just because we are able to reject H0: 1 = 0 and demonstrate statistical significance does not enable us to conclude that there is a linear relationship between x and y.
Linear relationship is a very small subset of possible relationship among variables.
A test of linear versus nonlinear relationship requires another batch of analysis.
Lesson10-76 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Variation Measures Coeff. Of Determination Standard Error of Estimate
Test Coefficients for Significance
yi* = b0 +b1xi
Evaluating the Model
Lesson10-77 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Y
X
Y
Xi
Total Sum of Squares (Yi - Y)2
Unexplained Sum of Squares (Yi -Yi
*)2
Explained Sum of Squares (Yi
* - Y)2
Yi
SST
SSE
SSR
yi* = b0 +b1xi
Variation Measures
Lesson10-78 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Total Sum of Squares (SST) Measures variation of observed Yi around the
mean,Y Explained Variation (SSR)
Variation due to relationship between X & Y
Unexplained Variation (SSE) Variation due to other factors
SST=SSR+SSE
Measures of Variation in Regression
Lesson10-79 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Variation in y (SST) = SSR + SSE
n
1i
2i )yy(
n
1i
2**i )yyyy(
n
1i
**i
2*2*i )yy)(yy()yy()yy(
n
1i
**i
n
1i
2*n
1i
2*i )yy)(yy()yy()yy(
n
1i
2*n
1i
2*i )yy()yy(
SST:
SSE SSR
=0, as imposed in the estimation, E(x)=0.
Lesson10-80 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
R2 (=r2, the coefficient of determination) measures the proportion of the variation in y that is explained by the variation in x.
n
ii
n
ii
n
ii
n
ii yy
SSR
yy
SSEyy
yy
SSER
1
2
1
2
1
2
1
2
2
)()(
)(
)(1
R2 takes on any value between zero and one. R2 = 1: Perfect match between the line and the data
points. R2 = 0: There are no linear relationship between x and
y.
Variation in y (SST) = SSR + SSE
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Summarizing the Example’s results (Example 1, 2 and 3)
The estimated selling price for a book with 800 pages is $89.14.
The standard error of estimate is $10.41.
The 95 percent confidence interval for all books with 800 pages is $89.14 ± $15.31. This means the limits are between $73.83 and $104.45.
The 95 percent prediction interval for a particular book with 800 pages is $89.14 ± $29.72. The means the limits are between $59.42 and $118.86.
These results appear in the following output.
Lesson10-82 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Example 3 continued
Regression Analysis: Price versus Pages
The regression equation isPrice = 48.0 + 0.0514 Pages
Predictor Coef SE Coef T PConstant 48.00 16.94 2.83 0.030Pages 0.05143 0.02700 1.90 0.105
S = 10.41 R-Sq = 37.7% R-Sq(adj) = 27.3%
Analysis of VarianceSource DF SS MS F PRegression 1 393.4 393.4 3.63 0.105Residual Error 6 650.6 108.4Total 7 1044.0
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Testing for Linearity
Key Argument: If the value of y does not change linearly with the
value of x, then using the mean value of y is the best predictor for the actual value of y. This implies is preferable.
If the value of y does change linearly with the value of x, then using the regression model gives a better prediction for the value of y than using the mean of y. This implies y=y* is preferable.
yy
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Three Tests for Linearity
Testing the Coefficient of CorrelationH0: = 0 There is no linear relationship between x and y.
H1: 0 There is a linear relationship between x and y.
Test Statistic:
1b
1
sb
t
Testing the Slope of the Regression LineH0: = 0 There is no linear relationship between x and y.
H1: 0 There is a linear relationship between x and y.
Test Statistic:
)2/()1( 2
nr
rt
Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Three Tests for Linearity
The Global F-testH0: There is no linear relationship between x and y.
H1: There is a linear relationship between x and y.
Test Statistic:
)2/()(
)(
)2/(
1/
2
1
*
2
1
*
nyy
yy
nSSE
SSR
MSE
MSRF
n
iii
n
ii
[Variation in y] = SSR + SSE. Large F results from a large SSR. Then, much of the variation in y is explained by the regression model. The null hypothesis should be rejected; thus, the model is valid.
Note: At the level of simple linear regression, the global F-test is equivalent to the t-test on b1. When we conduct regression analysis of multiple variables, the global F-test will take on a unique function.
Lesson10-86 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Purposes Examine Linearity Evaluate violations of assumptions
Graphical Analysis of Residuals Plot residuals versus Xi values
Difference between actual Yi & predicted Yi*
Studentized residuals:Allows consideration for the magnitude of the
residuals
Residual Analysis
Lesson10-87 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Residual Analysis for Linearity
Not Linear LinearOK
X
e e
X
For example, if truth isy = 0 + 1 x + 2x2 + The estimated residuals are likely e=2x2 +
Lesson10-88 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Heteroscedasticity OK Homoscedasticity
Using Standardized Residuals (e/se)
SR
X
SR
X
Residual Analysis for Homoscedasticity
When the requirement of a constant variance (homoscedasticity) is violated we have heteroscedasticity.
For example, for xi>xj
Var(i|xi)>var(j|xj)
Lesson10-89 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Residual Analysis for Independence
Not Independent Independent
X
SR
X
SR
OK
Using Standardized Residuals (e/se)
Lesson10-90 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
A time series is constituted if data were collected over time.
Examining the residuals over time, no pattern should be observed if the errors are independent.
When a pattern is detected, the errors are said to be autocorrelated.
Autocorrelation can be detected by graphing the residuals against time.
Non-independence of error variables
Lesson10-91 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
+
+++ +
++
++
+ +
++ + +
+
++ +
+
+
+
+
+
+Time
Residual Residual
Time+
+
+
Note the runs of positive residuals,replaced by runs of negative residuals
Note the oscillating behavior of the residuals around zero.
0 0
Patterns in the appearance of the residuals over time indicates that autocorrelation exists.
Lesson10-92 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
n
ii
n
iii
e
eeD
1
2
2
21)( Should be close to 2.
If not, examine the model for autocorrelation.
Used when data is collected over time to detect autocorrelation (Residuals in one time period are related to residuals in another period)
Measures Violation of independence assumption
The Durbin-Watson Statistic
Intuition: If x and y are independent, Var(x-y)= Var(x) + Var(y)
Lesson10-93 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
An outlier is an observation that is unusually small or large.
Several possibilities need to be investigated when an outlier is observed: There was an error in recording the value. The point does not belong in the sample. The observation is valid.
Identify outliers from the scatter diagram. It is customary to suspect an observation is an
outlier if its |standard residual| > 2
Outliers
Lesson10-94 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
+
+
+
+
+ +
+ + ++
+
+
+
+
+
+
+
The outlier causes a shift in the regression line
… but, some outliers may be very influential
++++++++++
An outlier An influential observation
Lesson10-95 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Nonnormality or heteroscedasticity can be remedied using transformations on the y variable.
The transformations can improve the linear relationship between the dependent variable and the independent variables.
Many computer software systems allow us to make the transformations easily.
Remedying violations of the required conditions
Lesson10-96 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
A brief list of transformations
Y’ = log y (for y > 0) Use when the se increases with y, or Use when the error distribution is positively skewed
y’ = y2
Use when the se2 is proportional to E(y), or
Use when the error distribution is negatively skewed y’ = y1/2 (for y > 0)
Use when the se2 is proportional to E(y)
y’ = 1/y Use when se
2 increases significantly when y increases beyond some value.
Lesson10-97 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
Example: Transformation to get linearity
Not Linear OK
X
e
Linear
e
X
Yi = b0 + b Xi + ei Yi = b0 + b1 Xi + b2 Xi2 +
ei
Lesson10-98 Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data
- END -
Lesson 10:Lesson 10: Regressions Part IRegressions Part I