Matrix Calculus Algebra
-
Upload
marcospaizante -
Category
Documents
-
view
120 -
download
1
Transcript of Matrix Calculus Algebra
![Page 1: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/1.jpg)
Matrix Calculus and Algebra
Yi ZhangYi Zhang
![Page 2: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/2.jpg)
Outline
Matrix calculus and algebra Dimensions of derivatives Dimensions of derivatives Basic calculations of matrix derivatives Rules for product chain trace determinant and Rules for product, chain, trace, determinant and
norms
![Page 3: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/3.jpg)
Matrix Derivatives: Dimensions
The basic case: y scalar, x scalar dy/dx: scalar dy/dx: scalar
Generalize to vectors/matrices
![Page 4: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/4.jpg)
Matrix Derivatives: Dimensions [T. Minka]
![Page 5: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/5.jpg)
Basic Calculations
Step 1: know the dimensions Step 2: element-wise calculations Step 2: element-wise calculations Step 3: put into the vector/matrix form
![Page 6: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/6.jpg)
Basic Calculations
Example:
Dimensions? Element-wise? Vector/matrix form?
![Page 7: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/7.jpg)
Basic Calculations
Example:
![Page 8: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/8.jpg)
Basic Calculations
Too simple? Try this … [J. Rennie]
We want:
![Page 9: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/9.jpg)
Basic Calculations
… …
![Page 10: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/10.jpg)
Basic Calculations
![Page 11: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/11.jpg)
Basic Calculations
![Page 12: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/12.jpg)
Basic Calculations
![Page 13: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/13.jpg)
Basic Calculations
![Page 14: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/14.jpg)
Basic Calculations
![Page 15: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/15.jpg)
Some Examples
![Page 16: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/16.jpg)
Gradient and Hessian
![Page 17: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/17.jpg)
Gradient and Hessian
![Page 18: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/18.jpg)
Gradient and Hessian
![Page 19: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/19.jpg)
Some Basic Rules
Take derivatives w.r.t. z (omitted)
![Page 20: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/20.jpg)
Product Rules
Take derivatives w.r.t. z (omitted)
![Page 21: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/21.jpg)
Derivatives of Determinants
Take derivatives w.r.t. z (omitted)
![Page 22: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/22.jpg)
The Chain Rule
Suppose
The chain rule:
![Page 23: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/23.jpg)
The Chain Rule
Suppose
The chain rule:
![Page 24: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/24.jpg)
Derivatives of Traces [K.B. Petersen]
Assume F(X) is an element-wise differentiable functiondifferentiable function f() is the scalar derivative of F()
![Page 25: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/25.jpg)
Derivatives of Traces [K.B. Petersen]
Given:
![Page 26: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/26.jpg)
Derivatives of Frobenius Norm
Frobenius norm
Derivatives
![Page 27: Matrix Calculus Algebra](https://reader034.fdocuments.net/reader034/viewer/2022050905/54fb1d894a7959f9348b467e/html5/thumbnails/27.jpg)
References
T. Minka. Old and New Matrix Algebra Useful for Statisticsfor Statistics
K. B. Petersen. The Matrix CookbookJ D M R i A Si l E i M t i J. D. M. Rennie. A Simple Exercise on Matrix Derivatives.