8/8/2019 Final Versiongauss and the Method of Least Squares
1/25
1
Gauss and the Method of Least Squares
Teddy Petrou Hongxiao Zhu
8/8/2019 Final Versiongauss and the Method of Least Squares
2/25
2
Outline
Who was Gauss?
Why was there controversy in finding the method of leastsquares?
Gauss treatment of error
Gauss derivation of the method of least squares
Gauss derivation by modern matrix notation
Gauss-Markov theorem Limitations of the method of least squares
References
8/8/2019 Final Versiongauss and the Method of Least Squares
3/25
3
Johann Carl Friedrich Gauss
Born:1777 Brunswick, Germany
Died: February 23, 1855, Gttingen, Germany
By the age of eight during arithmetic class heastonished his teachers by being able to
instantly find the sum of the first hundredintegers.
8/8/2019 Final Versiongauss and the Method of Least Squares
4/25
4
Facts about Gauss Attended Brunswick College in 1792, where he
discovered many important theorems before even
reaching them in his studies Found a square root in two different ways to fifty
decimal places by ingenious expansions andinterpolations
Constructed a regular 17 sided polygon, the firstadvance in this matter in two millennia. He was only18 when he made the discovery
8/8/2019 Final Versiongauss and the Method of Least Squares
5/25
5
Ideas of Gauss
Gauss was a mathematical scientist with interests in so manyareas as a young man including theory of numbers, to algebra,analysis, geometry, probability, and the theory of errors.
His interests grew, including observational astronomy, celestialmechanics, surveying, geodesy, capillarity, geomagnetism,electromagnetism, mechanism optics, and actuarial science.
8/8/2019 Final Versiongauss and the Method of Least Squares
6/25
6
Intellectual Personality and Controversy
Those who knew Gauss best found him to be cold and
uncommunicative.
He only published half of his ideas and found no one to sharehis most valued thoughts.
In 1805 Adrien-Marie Legendre published a paper on themethod of least squares. His treatment, however, lacked aformal consideration of probability and its relationship to leastsquares, making it impossible to determine the accuracy of themethod when applied to real observations.
Gauss claimed that he had written colleagues concerning theuse of least squares dating back to 1795
8/8/2019 Final Versiongauss and the Method of Least Squares
7/25
7
Formal Arrival of Least Squares
Gauss
Published The theory of the Motion of Heavenly Bodies in 1809.He gave a probabilistic justification of the method,which wasbased on the assumption of a normal distribution of errors.Gauss himself later abandoned the use of normal error function.
Published Theory of the Combination of Observations LeastSubject to Errors in 1820s. He substituted the root mean squareerror for Laplaces mean absolute error.
Laplace Derived the method of least squares (between1802 and1820) from the principle that the best estimate should have thesmallestmean error -the mean of the absolute value of the error.
8/8/2019 Final Versiongauss and the Method of Least Squares
8/25
8
Treatment of Errors
Using probability theory to describe error
Error will be treated as a random variable
Two types of errors
Constant-associated with calibration
Random error
8/8/2019 Final Versiongauss and the Method of Least Squares
9/25
9
Error Assumptions
Gauss began his study by making two assumptions
Random errors of measurements of the same type lie withinfixed limits
All errors within these limits are possible, but not necessarily
with equal likelihood
8/8/2019 Final Versiongauss and the Method of Least Squares
10/25
10
Density Function
)()(likely,equallyaremaginitudesametheoferrorsnegativeandPositive
oneslargeoccur thanlikely tomoreareerrorsSmall
)(isdx)x(x,intervalewithin thlyingerrorsofyprobabilitThe
.propertiesfollowingthe
ithfunction wdensityaasmeaningsamewith the)(functionthedefineWe
xx
dxx
x
!
JJ
J
J
8/8/2019 Final Versiongauss and the Method of Least Squares
11/25
11
Mean and Variance
Define . In many cases
assume k=0 Define mean square error as
If k=0 then the variance will equal
dxxxk ! )(J
dxxxm )(22 J
g
g
!
2m
8/8/2019 Final Versiongauss and the Method of Least Squares
12/25
12
Reasons for is always positive and is simple.
The function is differentiable and integrable unlikethe absolute value function.
The function approximates the average value in
cases where large numbers of observations are beingconsidered,and is simple to use when consideringsmall numbers of observations.
2
m2m
8/8/2019 Final Versiongauss and the Method of Least Squares
13/25
13
More on VarianceIf then variance equals .
Suppose we have independent random variables
with standard deviation 1 and expected value 0. Thelinear function of total errors is given by
Now the variance of E is given as
This is assuming every error falls within standarddeviations from the mean
0{k22 km
,...}'',',{ eee
...! eeE PP
!!
!!k
i
i
k
i
iieM1
2
1
222PP
P
8/8/2019 Final Versiongauss and the Method of Least Squares
14/25
14
Gauss Derivation of the Method of Least Squares
Suppose a quantity, V=f(x), where V, x are unknown. Weestimate V by an observation L.
If x is calculated by L, L~f(x), error will occur.
But if several quantities V,V,Vdepend on the sameunknown x and they are determined by inexact observations,then we can recover x by some combinations of theobservations.
Similar situations occur when we observe several quantities thatdepend on several unknowns.
8/8/2019 Final Versiongauss and the Method of Least Squares
15/25
15
Gauss Derivation of the Method of Least Squares
)variancesamethehavetheysoerrorsthescaledWe:(
ns'.observatiotheoferrorsmean'theofweightstheares'thewhere
,'
)''(',
)(:
:bensobservatioin theerrorsLet the
),,,("
),,,('
),,,(
.,,,unkownsoffunctionsare.,'',',where
.,'',',:nsobservatiotindependenby taking,'',',estimatewant toWe
:Problem
3
2
1
Note
p
p
LVv
p
LVv
zyxfV
zyxfV
zyxfV
zyxVVV
LLLVVV
.
.
.
.
.
..
..
!
!
!
!
!
VT
8/8/2019 Final Versiongauss and the Method of Least Squares
16/25
16
Gauss Derivation of the Method of Least Squares
)"...,,space(nobservatio...),,(spaceparameter:or,RR:
:mappingadescribessystemThis2.
since,inedoverdetermissystem1.This:Note
kno n.are,,tscoe icienthehere
,...,,unko nsounctionslinearasrittenare,,,
vvvzyxF
cba
zyxvvv
pp
""
TV
VT
VT
.
.
!!
!
.
..
.
'''''''''''''''
:systemlinearfollowingheConsider t
lzcybxavlzcybxav
lczbyaxv
8/8/2019 Final Versiongauss and the Method of Least Squares
17/25
17
_ a
componet.otherthe
orconditionsimilaranteandpossible,assmallasbeto2
antWe
...."""...),,(
Then.ocomponentirsttheis"...),,(uppose
:beloasdescribedcondition,optimalityanstatis ies2.Ronidentiytheis1.
:such thatRtoRrom),",,G(mappinglinearaorlookingareWe
:asproblemthestatecanWe
.z,y,x,otindependenkconstantsomeor
.:s.t
,",,otscoe icienare,",,here
:problemonoptimizatianolve
222min
!!
!
!
O
OOO
V
VT
OOO
OOO
OOO
kvvvvvvgx
Gvvvgx
GFG
vvv
kxetcvvv
vvv
Q
.
.
..
.
8/8/2019 Final Versiongauss and the Method of Least Squares
18/25
18
Gauss Derivation of the Method of Least Squares
Solutions:
Its still not obvious:
How do these results relate with the least squares estimation?
.)()()(.
222222
222
etce
tc !
EOEOEOEEE
OOO .
.,"",,enminimun hitsattains
sumthat theobviousisithichFromsystem.theo
neliminatiobyderivedetscoe icienthedenotetheallhere
222
etc
s
EOEOEO
OOO
E
!!!
.
8/8/2019 Final Versiongauss and the Method of Least Squares
19/25
19
Gauss Derivation of the Method of Least Squares
It can be proved that
...'ofonminimizatitheasresultssameget thewillwe
,0,0..
vanish.,,,
partialstheallwhere,minimizethatvaluesparameterthepickssquaresLeast
'
)'),,,('()),,,((
"'
22
22222
!x
;x!
x
;x
x
;x
x
;x
x
;x
;
!!;
OO
.
.
.
..
.
yxei
zyx
p
LzyxV
p
LzyxVvvvLet
8/8/2019 Final Versiongauss and the Method of Least Squares
20/25
20
Gauss derivation by modern matrix notation:
lAxv
pv
xV
cb
RcbcxbxbV
xxx
VVV
iiii
ii
ii
iij
iijiiii
!
!
!
:becomessystemThe
/)(
:settingbysystemcoordinateneaitch to
onobservatioanisAssume
.theovaluesin er theattempt toaninthemeasureWe
.andtheallovaluestheknoe
,,...
such that,,,parametersounctions
linearare,,,quantitiesobservablethatAssume
11
21
21
VV
V
T
.
.
8/8/2019 Final Versiongauss and the Method of Least Squares
21/25
21
Gauss derivation by modern matrix notation:
:Proof
norm.minimunofrowshas)(matrixthematricessuchallamongand
,
:holdsfollowingthesuch thatmatrixaiseThen ther.rankofmatrix)(aisSuppose
1 TTAAAE
xKAxx
KA
!
!
v"v
V
TVVVTVT
Gauss results are equivalent to the following lemma:
possible.assmallasbeshould
ofentriesdiagonaltheofsumthat thedemandingtoequivalentisThis
possible.assmallasbeshould...
:thatisconditiononoptimizatiThe
condition.firstthesatisfies)(
222
1
T
iiii
TT
KK
KKK
AAAE
T!
!
8/8/2019 Final Versiongauss and the Method of Least Squares
22/25
22
optimal.isGofpartconsistent-nontheinverses,leftlinearallamongand
))(()(functiontheofinverselefttheis)(:)(
thatshowslemmaour,:equationoriginalourtoeturning
diagonal.
itsonentriespositivestrictlyhavewill))((entries,zero-nonany
has)(ifsinceone,optimalfact theinissolutionthat theshowsThis
))(())())(((inally,
0)(getwe,thatnotingandgmultiplyinightmatrix.zerotheis)(Thus.
0)(,:getweg,Subtractin
;havewelso,;and,:Thus,)(
)(,Thus.)(denote,invertibleis
:continuedProof
1
11
xxlAxxEllAxEvG
lAxv
EKEK
EKE
EKEKEEEKEEKEKK
EEKEADD
AE
KA
xE
Kx
xKAxxEAxDAEAAAE
AxDAAxAAAxxAADAA
T
TTTT
TTTT
TTT
TTTTT
!!!
!
!!!
!! !
!!!!
!!!
Q
.
8/8/2019 Final Versiongauss and the Method of Least Squares
23/25
23
Gauss-Markov theorem
ariance.smallest vwith theestimatorunbiasedbesttheisestimatorsquares-leasttheed,uncorrelatareandvariancesamethehave'whens,other wordIn
)~
()Var(Cand)E(havewe,Cof~
estimatorunbiasedanyforthen,)Var(and0)E(Ifor.error vect
theisandctor,unknown veanis,rankhmatrix witaniswhere
modellinearaIn
TT
2
s
Var
I
ppn
Ax
LSLS
I
FUUUUFF
WII
IU
IU
e!!
!!
v
!
8/8/2019 Final Versiongauss and the Method of Least Squares
24/25
24
Limitation of the Method of Least Squares
Nothing is perfect:
This method is very sensitive to the presence ofunusual data points. One or two outliers cansometimes seriously skew the results of a leastsquares analysis.
8/8/2019 Final Versiongauss and the Method of Least Squares
25/25
25
References
Gauss, Carl Friedrich, Translated by G. W. Stewart. 1995. Theory of theCombination ofObservations Least Subject to Errors: PartOne, PartTwo, Supplement. Philadelphia: Society for Industrial and Applied
Mathematics. Plackett, R. L. 1949. A Historical Note on the Method of Least Squares.
Biometrika. 36:458460.
Stephen M. Stiger, Gauss and the Invention of Least Squares. TheAnnals of Statistics, Vol.9, No.3(May,1981),465-474.
Plackett, Robin L. 1972. The Discovery of the Method of Least Squares.
Plackett, Robin L. 1972. The Discovery of the Method of Least Squares.
Belinda B.Brand, Guass Method of Least Squares: A historically-basedintroduction. August2003
http://www.infoplease.com/ce6/people/A0820346.html
http://www.stetson.edu/~efriedma/periodictable/html/Ga.html
Top Related