Lecture07 - Least Squares Regression
Transcript of Lecture07 - Least Squares Regression
-
8/9/2019 Lecture07 - Least Squares Regression
1/32
Curve Fitting and Interpolation
Lecture 7:
Least Squares Regression
MTH2212 Computational Methods and Statistics
-
8/9/2019 Lecture07 - Least Squares Regression
2/32
Dr. M. HrairiDr. M. Hrairi MTH2212MTH2212 -- Computational Methods and StatisticsComputational Methods and Statistics 22
Objectives
Introduction
Linear regression
Polynomial regression
Multiple linear regression
General linear least squares
Nonlinear regression
-
8/9/2019 Lecture07 - Least Squares Regression
3/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 33
Curve Fitting
Experimentation
Data available at discrete points or times
Estimates are required at points between the discrete values
Curves are fit to data in order to estimate the intermediate values
-
8/9/2019 Lecture07 - Least Squares Regression
4/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 44
Curve Fitting
Two methods - depending on error in data
Interpolation
- Precise data
- Force through each data point
Regression- Noisy data
- Represent trend of the data
0
1
2
3
4
5
6
7
8
9
0 2 4 6 8 10 12 14 16
x
f(x)
0
20
40
60
80
100
120
0 1 2 3 4 5
Time (s)
Tem
perature(degF)
-
8/9/2019 Lecture07 - Least Squares Regression
5/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 55
Least Squares Regression
Experimental Data
Noisy (contains errors or inaccuracies)
x values are accurate, y values are not
Find relationship betweenxand y = f(x)
Fit general trend without matching individual points
Derive a curve that minimizes the discrepancy between the data
points and the curve Least-squaresregression
-
8/9/2019 Lecture07 - Least Squares Regression
6/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 66
x y
2.10 2.90
6.22 3.83
7.17 5.98
10.5 5.71
13.7 7.74
Linear Regression: Definition
0
1
2
3
4
5
6
7
8
9
0 2 4 6 8 10 12 14 16
x
f(x)
Straight line characterizes trend without
passing through particular point
Noisy Data From
Experiment
xaay10
!
-
8/9/2019 Lecture07 - Least Squares Regression
7/32
Dr. M. HrairiDr. M. Hrairi MTH2212MTH2212 -- Computational Methods and StatisticsComputational Methods and Statistics 77
Linear Regression: criteria for a best fit
How do we measure goodness of fit of the line to the data?
0
1
2
3
4
5
6
7
8
9
0 2 4 6 8 10 12 14 16
x
f(x)
y1 y2
y4
y5
y3
e 3
e2
Regression Model
y = a 0+ a 1x
Residual
e = y - (a 0+ a 1x)
Data points
-
8/9/2019 Lecture07 - Least Squares Regression
8/32
Dr. M. HrairiDr. M. Hrairi MTH2212MTH2212 -- Computational Methods and StatisticsComputational Methods and Statistics 88
Linear Regression: criteria for a best fit
Use the curve that minimizes the residual between the data
points and the line
Model:
Find the values of a0 and a1 that minimize Sr
xy10
!
ii xaay 10 !
iii xaa= ye 10
n
i=ii
n
i=ir xaay=e=S
1
210
1
20
1
2
3
4
5
6
7
8
9
0 2 4 6 8 10 12 14 16
x
f(x)
y1 y2
y4
y5
y3
e 3
e 2
Regression Model
y = a 0+ a 1x
Residual
e = y - (a 0+ a 1x)
Data points
x y
2.10 2.90
6.22 3.83
7.17 5.98
10.5 5.71
13.7 7.74
-
8/9/2019 Lecture07 - Least Squares Regression
9/32
Dr. M. HrairiDr. M. Hrairi MTH2212MTH2212 -- Computational Methods and StatisticsComputational Methods and Statistics 99
Linear Regression: Finding a0 and a1
Minimize Sr by taking
derivatives WRT
a0and a1,
First a0
? A
? A ? A
!
!
n
i=
n
i=
a
a
1 0
1
2
0
2 ..
.
x
x
x
x
-
n
i=
ii
r x=S
1
2
10
00 x
x
x
x
!
-
n
i=i
n
i=i yaxna
1
1
1
0
? A
0
12
1
10
)(xaayn
i=
ii
!
! Finally
-
8/9/2019 Lecture07 - Least Squares Regression
10/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 1010
Linear Regression: Finding a0 and a1
Finally
Minimize Sr by taking
derivatives WRT
a0and a1
Second a1
-
n
i ii
r aaa
a 1
2
10
11x
x
x
x
? A
? A ? A
!
!
n
i=
n
i=
a
a
1 1
1
2
1
2 ..
.
x
x
x
x
? A
0
2
1
10
)(i
iii
!
!
!
-
-
iii
ii
ii
1
1
1
20
1
-
8/9/2019 Lecture07 - Least Squares Regression
11/32
Dr. M. HrairiDr. M. Hrairi MTH2212MTH2212 -- Computational Methods and StatisticsComputational Methods and Statistics 1111
Linear Regression: Normal equations
!
-
-
n
i=ii
n
i=i
n
i=i yxaxax
1
1
1
20
1
!
-
n
i=i
n
i=i yaxna
11
10
Set of two simultaneous linear equations with two unknowns( a0 and a1):
-
8/9/2019 Lecture07 - Least Squares Regression
12/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 1212
Linear Regression: Solution of normal equations
2
11
2
111
1
2
11
2
111
2
1
0
1
1
1
11
-
!
-
!
n
i=i
n
i=i
n
i=i
n
i=i
n
i=ii
n
i=i
n
i=i
n
i=ii
n
i=i
n
i=i
n
i=i
xn
x
yxn
yx
a
xn
x
yxxn
xyn
a
The normal equations can be solved simultaneously for:
-
8/9/2019 Lecture07 - Least Squares Regression
13/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 1313
Example 1
Fit a straight line to the values in the following table
x y
2.10 2.90
6.22 3.83
7.17 5.98
10.5 5.71
13.7 7.74
-
8/9/2019 Lecture07 - Least Squares Regression
14/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 1414
Example 1 - Solution
The intercept and the slope can be calculated with:
i xi yi xi2
xiyi
1 2.10 2.90 4.41 6.09
2 6.22 3.83 38.69 23.82
3 7.17 5.98 51.41 42.88
4 10.5 5.71 110.25 59.96
5 13.7 7.74 187.69 106.04
39.69 26.1
6
392.32 238.7
4
2
11
2
1111
2
11
2
111
2
10
1
1
1
11
-
!
-
!
n
i=i
n
i=i
n
i=i
n
i=i
n
i=ii
n
i=i
n
i=i
n
i=ii
n
i=i
n
i=i
n
i=i
xn
x
yxn
yx
a
xn
x
yxxn
xyn
a
-
8/9/2019 Lecture07 - Least Squares Regression
15/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 1515
Example 1 - Solution
? A
? A
4023.0
69.39
5
13.392
)16.26)(69.39(5
17.238
038.2
69.39513.392
)7.238)(69.39(5
1)3.392)(16.26(
5
1
21
2
0
!
!
!
!
a
a
x..y 402300382 !
The values of the intercept and the slope:
The equation of the straight line linear regression
-
8/9/2019 Lecture07 - Least Squares Regression
16/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 1616
Linear Regression: Quantification of error
Suppose we have data points (xi, yi) and modeled (orpredicted) points (xi, i) from the model = f(x).
Data {yi} have two types of variations;(i) variation explained by the model and
(ii) variation not explained by the model.
Residual sum of squares: variation not explained by themodel
Regression sum of squares: variation explained by themodel
The coefficient of determination r2
!
!n
iiir yyS
1
2)(
!
!n
i
it yyS1
2)(
t
rt
S
SSr
!2
-
8/9/2019 Lecture07 - Least Squares Regression
17/32
Dr. M. HrairiDr. M. Hrairi MTH2212MTH2212 -- Computational Methods and StatisticsComputational Methods and Statistics 1717
Linear Regression: Quantification of error
For a perfect fit
Sr=0 and r=r2=1, signifying that the line explains 100% of
the variability of the data.
For r=r2
=0, Sr=St, the fit represents no improvement.
x1 x2
y1
y2
y
Total variation in y = Variation explained by the model + Unexplained variation (error)
-
8/9/2019 Lecture07 - Least Squares Regression
18/32
Dr. M. HrairiDr. M. Hrairi MTH2212MTH2212 -- Computational Methods and StatisticsComputational Methods and Statistics 1818
Linear Regression: Another measure of fit
In addition to r2, r
Define
= standard error of the estimate- Represents the distribution of the residuals around the
regression line
- Large Sy|xlarge residuals
- Small Sy|xsmall residuals
2| n
SS rxy
-
8/9/2019 Lecture07 - Least Squares Regression
19/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 1919
Example 2
Compute the total standard deviation, the standard error of
the estimate, and the correlation coefficient for the data in
Example 1.
x y
2.10 2.90
6.22 3.83
7.17 5.98
10.5 5.71
13.7 7.74
-
8/9/2019 Lecture07 - Least Squares Regression
20/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2020
Example 2 - Solution
The standard deviation is
The standard error of the estimate is
The correlation coefficient r is
9028.115
4819.14
1!
!
!
n
SS
t
y
8092.025
9643.1
2| !
!
!
n
SS
r
xy
8644.4819.14
9643.14819.142!
!
!
t
rtr
i xi yi (yi-y)2
(yi-a0-a1xi)2
1 2.1 2.9 5.4382 0.0003
2 6.22 3.83 1.9656 0.5045
3 7.17 5.98 0.5595 1.11834 10.5 5.71 0.2285 0.3049
5 13.7 7.74 6.2901 0.0363
39.69 26.1
6
14.481
9
1.9643
9297.08644.0 !!r
-
8/9/2019 Lecture07 - Least Squares Regression
21/32
-
8/9/2019 Lecture07 - Least Squares Regression
22/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2222
Linearization of Nonlinear Relationships
-
8/9/2019 Lecture07 - Least Squares Regression
23/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2323
Polynomial Regression
Minimize the residual between the data points and the curve
-- least-squaresregression
Must find values ofa0 ,a1,a2, am
ii xaay 10 !Linear
2
210 iii xaxaay !Quadratic
2
210 iiii xaxaxaay !Cubic
Generalm
imiiii xxxxy .3
32
210
-
8/9/2019 Lecture07 - Least Squares Regression
24/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2424
Polynomial Regression
Residual
Sum of squared residuals
Minimize by taking derivatives
)(3
32
210
m
imiiiii xxxxe .
n
i
mm
n
iir aaaaaeS
1
233
2210
1
2 )]([ .
-
8/9/2019 Lecture07 - Least Squares Regression
25/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2525
Polynomial Regression
Normal Equations
-
!
-
-
n
i=i
mi
n
i=ii
n
i=ii
n
i=
i
mn
i=
mi
n
i=
mi
n
i=
mi
n
i=
mi
n
i=
mi
n
i=i
n
i=i
n
i=i
n
i=
mi
n
i=i
n
i=i
n
i=i
n
i=
mi
n
i=
i
n
i=
i
yx
yx
yx
y
a
a
aa
xxxx
xxxx
xxxx
xxxn
1
1
2
1
1
2
1
0
1
2
1
2
1
1
1
1
2
1
4
1
3
1
2
1
1
1
3
1
2
1
11
2
1
/
/
/1///
-
8/9/2019 Lecture07 - Least Squares Regression
26/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2626
Example 3
Fit a third-order polynomial to the data given in the Table
below
x 0 1.0 1. 2.3 2. 4.0 5.1 6.0 6.5 7.0 8.1 9.0
y 0.2 0.8 2.5 2.5 3.5 4.3 3.0 5.0 3.5 2.4 1.3 2.0
x 9.3 11.0 11.3 12.1 13.1 14.0 15.5 16.0 17.5 17.8 19.0 20.0
y -0.3 -1.3 -3.0 -4.0 -4.9 -4.0 -5.2 -3.0 -3.5 -1.6 -1.4 -0.1
-
8/9/2019 Lecture07 - Least Squares Regression
27/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2727
Example 3 - Solution
-
!
-
-
n
iii
n
iii
n
iii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
n
ii
a
a
a
a
n
1
3
1
2
1
1
3
2
1
0
1
6
1
5
1
4
1
3
1
5
1
4
1
3
1
2
1
4
1
3
1
2
1
1
3
1
2
1
-
!
-
-
3643
2603
316301
22351 11612 014252 3546342
12 014252 354634223060
252 354634223060622463422306062224
3
2
1
0
.
.
.
.
....
....
....
...
-
-
01210
35320
30512
35930
3
2
1
0
.
.
.
.
a
a
a
a
-
8/9/2019 Lecture07 - Least Squares Regression
28/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2828
Example 3 - Solution
Regression Equation
y = - 0.359 + 2.305x - 0.353x2 + 0.012x3
-6
-4
-2
0
2
4
6
0 5 10 15 20 25
x
f(x)
-
8/9/2019 Lecture07 - Least Squares Regression
29/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 2929
Multiple Linear Regression
y = a0 + a1x1 + a2x2 + e
Again very similar.
Minimize e
Polynomial and multipleregression fall within
definition of General
Linear Least Squares.
-
8/9/2019 Lecture07 - Least Squares Regression
30/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 3030
General Linear Least Squares
_ a ? A_ a _ a
? A
_ a_ a
_ a residualsE
tscoefficienunknownA
variabledependenttheofvaluedobservedY
t variableindependentheofvaluesmeasuredat thefunctionsbasistheofvaluescalculatedtheofmatrix
functionsbasis1are10
221100
!
!
Z
EAZY
m, z,, zz
ezazazazay
m
mm
-
.
2
1 0
! !
!
n
i
m
j
jijir zayS
Minimized by taking its partial
derivative w.r.t. each of the
coefficients and setting the
resulting equation equal to zero
-
8/9/2019 Lecture07 - Least Squares Regression
31/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 3131
Nonlinear Regression
Not all equations can be broken down into General Linear
Least Squares model i.e.
Solve with nonlinear least squares using iterative methods
like Gauss-Newton method
Equation could possibly be transformed into linear form
Caveat: when fitting transformed data you minimize the residuals of the
data you are working with
May not give you the exact same fit as nonlinear regression on
untransformed data
eeayxa! )1( 10
-
8/9/2019 Lecture07 - Least Squares Regression
32/32
Dr. M. HrairiDr. M. Hrairi MTHMTH22122212 -- Computational Methods and StatisticsComputational Methods and Statistics 3232
Nonlinear Regression
eeayxa! )1( 10