LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is...
Transcript of LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is...
![Page 1: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/1.jpg)
EE613Machine Learning for Engineers
LINEAR REGRESSION I
Sylvain CalinonRobot Learning & Interaction Group
Idiap Research InstituteOct. 31, 2019
![Page 2: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/2.jpg)
EE613 - List of courses
2
19.09.2019 (JMO) Introduction26.09.2019 (JMO) Generative I03.10.2019 (JMO) Generative II10.10.2019 (JMO) Generative III17.10.2019 (JMO) Generative IV24.10.2019 (JMO) Decision-trees31.10.2019 (SC) Linear regression I07.11.2019 (JMO) Kernel SVM14.11.2019 (SC) Linear regression II21.11.2019 (FF) MLP28.11.2019 (FF) Feature-selection and boosting05.12.2019 (SC) HMM and subspace clustering12.12.2019 (SC) Nonlinear regression I 19.12.2019 (SC) Nonlinear regression II
![Page 3: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/3.jpg)
Outline
3
Linear Regression I (Oct 31)• Least squares• Singular value decomposition (SVD)• Kernels in least squares (nullspace)• Ridge regression (Tikhonov regularization)• Weighted least squares• Iteratively reweighted least squares (IRLS)• Recursive least squares
Linear Regression II (Nov 14)• Logistic regression• Tensor-variate regression
Hidden Markov model (HMM) & subspace clustering (Dec 5)
Nonlinear Regression I (Dec 12)• Locally weighted regression (LWR)• Gaussian mixture regression (GMR)
Nonlinear Regression II (Dec 19)• Gaussian process regression (GPR)
![Page 4: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/4.jpg)
Labs
4
Teguh Lembono
Python notebooks and labs exercises:https://github.com/teguhSL/ee613-python
![Page 5: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/5.jpg)
PbDlib
http://www.idiap.ch/software/pbdlib/ 5
![Page 6: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/6.jpg)
LEAST SQUAREScirca 1795
6
![Page 7: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/7.jpg)
Least squares: a ubiquitous tool
Weighted least squares?
Regularized least squares?
L1-norm instead of L2-norm?
Nullspace structure?
Recursive computation?7
![Page 8: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/8.jpg)
![Page 9: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/9.jpg)
![Page 10: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/10.jpg)
Linear regression
Python notebooks:demo_LS.ipynb, demo_LS_polFit.ipynb
Matlab codes: demo_LS01.m, demo_LS_polFit01.m
10
![Page 11: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/11.jpg)
Linear regression
• Least squares is everywhere: from simple problems to large scale problems.
• It was the earliest form of regression, which was published by Legendre in 1805 and by Gauss in 1809. They both applied the method to the problem of determining the orbits of bodies around the Sun from astronomical observations.
• The term regression was only coined later by Galtonto describe the biological phenomenon that the heights of descendants of tall ancestors tend to regress down towards a normal average.
• Pearson later provided the statistical context showing that the phenomenon is more general than a biological context. Karl Pearson
Francis Galton
Adrien-Marie Legendre
Carl Friedrich Gauss
11
![Page 12: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/12.jpg)
Multivariate linear regression
Moore-Penrosepseudoinverse 12
Sample 1
Sample 2
Sample N
![Page 13: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/13.jpg)
Multiple multivariate linear regression
Moore-Penrosepseudoinverse 13
![Page 14: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/14.jpg)
Example of multivariate linear regression
14
![Page 15: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/15.jpg)
Polynomial fitting with least squares
15
![Page 16: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/16.jpg)
Singular value decomposition (SVD)
Unitary matrix(orthogonal)
Unitary matrix(orthogonal)
Matrix with non-negative diagonal entries
(singular values of X)
10
![Page 17: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/17.jpg)
Least squares with SVD
17
![Page 18: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/18.jpg)
Kernels in least squares(nullspace projection)
Python notebook:demo_LS_polFit.ipynb
Matlab code: demo_LS_polFit_nullspace01.m
18
![Page 19: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/19.jpg)
Kernels in least squares (nullspace)
19
![Page 20: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/20.jpg)
Kernels in least squares (nullspace)
20
![Page 21: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/21.jpg)
21
Example with polynomial fitting
4 samples
![Page 22: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/22.jpg)
Forward kinematics
22
Example with robot inverse kinematics
![Page 23: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/23.jpg)
Find q to have f(q)=0
![Page 24: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/24.jpg)
Gauss-Newton algorithm
![Page 25: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/25.jpg)
25
Gauss-Newton algorithm
Example with robot inverse kinematics
![Page 26: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/26.jpg)
Example with robot inverse kinematics
26
Task space /operational space
coordinates
Joint space /configuration space
coordinates
![Page 27: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/27.jpg)
Example with robot inverse kinematics
27
Secondary constraint:trying to move the first joint
Primary constraint:keeping the tip
of the robot still
![Page 28: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/28.jpg)
Example with robot inverse kinematics
28
Trackingtarget withleft hand
Tracking target with right hand, if possible
![Page 29: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/29.jpg)
Ridge regression (Tikhonov regularization,penalized least squares)
Python notebook: demo_LS_polFit.ipynb
Matlab example: demo_LS_polFit02.m
29
![Page 30: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/30.jpg)
Ridge regression (Tikhonov regularization)
30
![Page 31: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/31.jpg)
Ridge regression (Tikhonov regularization)
31
![Page 32: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/32.jpg)
Ridge regression (Tikhonov regularization)
32
![Page 33: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/33.jpg)
Ridge regression (Tikhonov regularization)
33
![Page 34: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/34.jpg)
Weighted least squares(Generalized least squares)
Python notebook:demo_LS_weighted.ipynb
Matlab example: demo_LS_weighted01.m
34
![Page 35: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/35.jpg)
Weighted least squares
35
![Page 36: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/36.jpg)
Weighted least squares
36
Color intensityproportional
to weight
![Page 37: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/37.jpg)
37
Weighted least squares - Example I
![Page 38: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/38.jpg)
Weighted least squares - Example II
38
![Page 39: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/39.jpg)
Iteratively reweighted least squares (IRLS)
Python notebook:demo_LS_weighted.ipynb
Matlab code: demo_LS_IRLS01.m
39
![Page 40: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/40.jpg)
Iteratively reweighted least squares (IRLS)
40
• Iteratively Reweighted Least Squares generalizes least squares by raising the error to a power that is less than 2: can no longer be called “least squares”
• The strategy is that an error |e|p can be rewritten as |e|p = |e|p-2 e2.
• |e|p-2 can be interpreted as a weight, which is used to minimize e2 with weighted least squares.
• p=1 corresponds to least absolute deviation regression.
![Page 41: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/41.jpg)
41
Iteratively reweighted least squares (IRLS)
|e|p = |e|p-2 e2
![Page 42: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/42.jpg)
42
IRLS as regression robust to outliers
Color darknessproportional
to weight
![Page 43: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/43.jpg)
Recursive least squares
Python notebook:demo_LS_recursive.ipynb
Matlab code: demo_LS_recursive01.m
43
![Page 44: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/44.jpg)
Recursive least squares
44
![Page 45: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/45.jpg)
Recursive least squares
45
![Page 46: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/46.jpg)
Recursive least squares
46
![Page 47: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/47.jpg)
Linear regression:
Examples of applications
47
![Page 48: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/48.jpg)
Linear in state space of higher dimension
Koopman operators in controlNonlinear
Main challenge in Koopman analysis:How to find these basis functions?
48
![Page 49: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/49.jpg)
state variable (position+velocity)
desired state
control command (acceleration)
precision matrix
control weight matrix
System dynamics
Use low control commands!Track path!
Linear quadratic tracking (LQT)
49
![Page 50: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/50.jpg)
How to solve this objective function?
System dynamics
Use low control commands!Track path!
Pontryagin’s max. principle, Riccati equation,
Hamilton-Jacobi-Bellman
(the Physicist perspective)
Dynamic programming
(the Computer Scientist perspective)
Linear algebra
(the Algebraist perspective)
50
![Page 51: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/51.jpg)
Let’s first re-organize the objective function…
51
![Page 52: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/52.jpg)
Let’s then re-organize the constraint…
52
![Page 53: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/53.jpg)
Linear quadratic tracking (LQT)
The constraint can then be put into the objective function:
Solving for u results in the analytic solution:
53
![Page 54: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/54.jpg)
The distribution in control space can be projected back to the state space
Passing through 3 keypoints withvarying precision
t=0
t=0.3t=0.6
t=1
t=0.3t=0.6
t=1
54
Linear quadratic tracking (LQT)
![Page 55: LINEAR REGRESSION Iodobez/TEACHING/EE613/EE613-lecture7...Linear regression • Least squares is everywhere: from simple problems to large scale problems. • It was the earliest form](https://reader033.fdocuments.net/reader033/viewer/2022060521/604f45539079573162360e4e/html5/thumbnails/55.jpg)
Main references
Regression
F. Stulp and O. Sigaud. Many regression algorithms, one unified model – a review. Neural Networks, 69:60–79, September 2015
W. W. Hager. Updating the inverse of a matrix. SIAM Review, 31(2):221–239, 1989
G. Strang. Introduction to Applied Mathematics. Wellesley-Cambridge Press, 1986
The Matrix Cookbook
Kaare Brandt PetersenMichael Syskind Pedersen