BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR...
Transcript of BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR...
![Page 1: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/1.jpg)
IFT 6113BACKGROUND: LINEAR ALGEBRA
AND OPTIMIZATIONhttp://tiny.cc/6113
Mikhail Bessmeltsev
![Page 2: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/2.jpg)
MOTIVATION
Quick summary!Mostly for common ground: You may already know this material.
First half is important; remainder summarizes interesting recent tools.
Numerical problems are everywhere in geometric modeling!
![Page 3: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/3.jpg)
OUR BIAS
Numerical analysis is a huge field.
Patterns, algorithms, & examples common in geometry.
![Page 4: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/4.jpg)
EXAMPLES
How to flatten a mesh?
RI3
RI2
u
v
Pi
ui ,vi
Object space (3D) Texture space (2D)
“Mesh Parameterization: Theory and Practice” by Kai Hormann, Bruno Lévy, Alla Sheffer
![Page 5: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/5.jpg)
![Page 6: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/6.jpg)
EXAMPLES
How to flatten a mesh?
min𝑢𝑖,𝑣𝑖∈ℝ
2𝑓(𝑢1, 𝑣1, … , 𝑢𝑛, 𝑣𝑛)
Fit a target 2D shape?
Triangle distortion?
More sharp corners?
![Page 7: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/7.jpg)
“Boundary First Flattening” by Rohan Sawhney and Keenan Crane
![Page 8: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/8.jpg)
EXAMPLES
How to animate a character?
“Linear Subspace Design for Real-Time Shape Deformation” by Yu Wang, Alec Jacobson, Jernej Barbič, Ladislav Kavan
![Page 9: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/9.jpg)
EXAMPLES
How to animate a character?
min𝑥𝑖,𝑦𝑖∈ℝ
2𝑓(𝑥1, 𝑦1, … , 𝑥𝑛, 𝑦𝑛)
s.t. 𝑥𝑗 − 𝑥𝑗′ = 0
𝑦𝑗 − 𝑦𝑗′ = 0
Triangle distortion?
Some smoothness?
Known positions
𝑗 ∈ 𝐽
![Page 10: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/10.jpg)
min𝑥∈ℝ𝑛
𝑓(𝑥)
s.t. 𝑔 𝑥 = 0ℎ 𝑥 ≥ 0
![Page 11: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/11.jpg)
min𝑥∈ℝ𝑛
𝑓(𝑥)
s.t. 𝑔 𝑥 = 0ℎ 𝑥 ≥ 0
Optimized function(Energy)
![Page 12: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/12.jpg)
min𝑥∈ℝ𝑛
𝑓(𝑥)
s.t. 𝑔 𝑥 = 0ℎ 𝑥 ≥ 0
Equalityconstraints
![Page 13: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/13.jpg)
min𝑥∈ℝ𝑛
𝑓(𝑥)
s.t. 𝑔 𝑥 = 0ℎ 𝑥 ≥ 0
Inequality constraints
![Page 14: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/14.jpg)
min𝑥∈ℝ𝑛
𝐴𝑥 − 𝑏 22
𝐴𝑥 = 𝑏 ↔
EXAMPLES
![Page 15: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/15.jpg)
min𝑥∈ℝ𝑛
𝐴𝑥 22
s.t. 𝑥 − 1 = 0𝐴𝑥 = 𝜆𝑥 ↔
EXAMPLES
![Page 16: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/16.jpg)
ROUGH PLAN
• (intro) Matrices and Eigenvalues
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 17: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/17.jpg)
ROUGH PLAN
• (intro) Matrices and Eigenvalues
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 18: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/18.jpg)
MATRICES
How are those matrices special?
1−2
5
1 −1 0−1 2 00 0 5
0 −1 −21 0 42 −4 0
cos(0.4) − sin 0.4 0
sin(0.4) cos(0.4) 00 0 1
1 0 30 1 4
1
![Page 19: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/19.jpg)
MATRICES
How are those matrices special?
1−2
5
1 −1 0−1 2 00 0 5
0 −1 −21 0 42 −4 0
cos(0.4) − sin 0.4 0
sin(0.4) cos(0.4) 00 0 1
1 0 30 1 4
1
Diagonal Symmetric𝐴 = 𝐴𝑇
Skew-symmetric𝐴𝑇 = −𝐴
Orthogonal𝐴−1 = 𝐴𝑇
Um something?
![Page 20: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/20.jpg)
MATRICESWhat is the geometric meaning of those?
1−2
5
cos(0.4) − sin 0.4 0
sin(0.4) cos(0.4) 00 0 1
1 0 30 1 4
1
Diagonal
Orthogonal𝐴−1 = 𝐴𝑇
Um something?
![Page 21: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/21.jpg)
INTRO TO INTROWhat is the geometric meaning of those?
1−2
5
cos(0.4) − sin 0.4 0
sin(0.4) cos(0.4) 00 0 1
1 0 30 1 4
1
ScalingNon-uniform scaling
Rotation Translation in 2D*
![Page 22: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/22.jpg)
MATRICES
Eigenvalues and eigenvectors
𝐴𝑥 = 𝜆𝑥
or
ℒ 𝑓(𝑥) = 𝜆𝑓(𝑥)
discrete
continuous
![Page 23: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/23.jpg)
MATRICES
• Geometric transformations• Linear operators on functions
![Page 24: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/24.jpg)
EIGENVECTORS/VALUES
Geometric meaning?
Vectors which only scale
![Page 25: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/25.jpg)
MATRIX IS POSITIVE DEFINITE
𝐴 ≽ 0
All eigenvalues are nonnegative
Geometric meaning?
![Page 26: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/26.jpg)
min𝑥∈ℝ𝑛
𝑓(𝑥)
s.t. 𝑔 𝑥 = 0ℎ 𝑥 ≥ 0
![Page 27: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/27.jpg)
ROUGH PLAN
• (intro) Matrices and Eigenvalues
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 28: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/28.jpg)
VECTOR SPACES AND LINEAR OPERATORS
![Page 29: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/29.jpg)
ABSTRACT EXAMPLE
Eigenvectors?
![Page 30: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/30.jpg)
IN FINITE DIMENSIONS
![Page 31: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/31.jpg)
LINEAR SYSTEM OF EQUATIONS
Simple “inverse problem”
![Page 32: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/32.jpg)
COMMON STRATEGIES
• Gaussian elimination– O(n3) time to solve Ax=b or to invert
• But: Inversion is unstable and slower!
• Never ever compute A-1 if you can avoid it.
![Page 33: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/33.jpg)
SIMPLE EXAMPLE
![Page 34: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/34.jpg)
STRUCTURE?
![Page 35: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/35.jpg)
LINEAR SOLVER CONSIDERATIONS
• Never construct 𝑨−𝟏 explicitly(if you can avoid it)
• Added structure helpsSparsity, symmetry, positive definiteness, bandedness
![Page 36: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/36.jpg)
LINEAR SYSTEMS: SOLVERS
• Direct (explicit matrix)– Dense: Gaussian elimination/LU, QR for least-
squares– Sparse: Reordering (SuiteSparse, Eigen)
• Iterative (apply matrix repeatedly)– Positive definite: Conjugate gradients– Symmetric: MINRES, GMRES– Generic: LSQR
![Page 37: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/37.jpg)
GENERIC ADVICE
Generic tools are often not too effective!
![Page 38: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/38.jpg)
GENERIC ADVICE
Try the
simplest solver first.
![Page 39: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/39.jpg)
VERY COMMON: SPARSITY
Induced by the connectivity of the triangle mesh.
Iteration of CG has local effect⇒ Precondition!
![Page 40: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/40.jpg)
FOR IFT 6113• No need to implement a linear solver
• If a matrix is sparse, your code should store it as a sparse matrix!
https://eigen.tuxfamily.org/dox/group__TutorialSparse.html
![Page 41: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/41.jpg)
ROUGH PLAN
• (intro) Matrices and Eigenvalues
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 42: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/42.jpg)
UNCONSTRAINEDOPTIMIZATION
min𝑥∈ℝ𝑛
𝑓(𝑥)
Trivial when 𝑓 𝑥 is linear
Easy when 𝑓(𝑥) is quadratic
Hard in case of generic non-linear.
![Page 43: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/43.jpg)
UNCONSTRAINEDOPTIMIZATION
min𝑥∈ℝ𝑛
𝑓(𝑥)
![Page 44: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/44.jpg)
NOTIONS FROM CALCULUS
Gradient
𝑓 𝑥, 𝑦 = 3𝑥2 − 𝑦
∇𝑓 =𝜕𝑓
𝜕𝑥,𝜕𝑓
𝜕𝑦= (6𝑥,−1)
Geometric meaning?
![Page 45: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/45.jpg)
NOTIONS FROM CALCULUS
Gradienthttps://en.wikipedia.org/?title=Gradient
![Page 46: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/46.jpg)
NOTIONS FROM CALCULUS
Gradienthttps://en.wikipedia.org/?title=Gradient
![Page 47: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/47.jpg)
NOTIONS FROM CALCULUS
Jacobianhttps://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant
![Page 48: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/48.jpg)
NOTIONS FROM CALCULUS
Hessianhttp://math.etsu.edu/multicalc/prealpha/Chap2/Chap2-5/10-3a-t3.gif
![Page 49: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/49.jpg)
OPTIMIZATION TO ROOT-FINDING
Critical point
(unconstrained)
Saddle point
Local min
Local max
![Page 50: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/50.jpg)
CONVEX FUNCTIONS
https://en.wikipedia.org/wiki/Convex_function
𝑓′′ 𝑥 > 0
![Page 51: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/51.jpg)
CONVEX FUNCTIONS
https://en.wikipedia.org/wiki/Convex_function
𝐻 𝑥 ≽0
![Page 52: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/52.jpg)
SPECIAL CASE: LEAST-SQUARES
Normal equations(better solvers for this case!)
![Page 53: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/53.jpg)
USEFUL DOCUMENT
http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/3274/pdf/imm3274.pdf
The Matrix CookbookPetersen and Pedersen
![Page 54: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/54.jpg)
UNCONSTRAINED OPTIMIZATION
Unstructured.
![Page 55: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/55.jpg)
BASIC ALGORITHMS
Gradient descent
Line search
Multiple optima!
![Page 56: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/56.jpg)
BASIC ALGORITHMS
Accelerated gradient descent
Quadratic convergence on convex problems!(Nesterov 1983)
A very cool intro: https://distill.pub/2017/momentum/
![Page 57: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/57.jpg)
BASIC ALGORITHMS
Newton’s Method
1
2
3
Line search for stability
![Page 58: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/58.jpg)
BASIC ALGORITHMS
Quasi-Newton: BFGS and friends
Hessian approximation
• (Often sparse) approximation from previous samples and gradients
• Inverse in closed form!
![Page 59: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/59.jpg)
EXAMPLE: SHAPE INTERPOLATION
Fröhlich and Botsch. “Example-Driven Deformations Based on Discrete Shells.” CGF 2011.
![Page 60: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/60.jpg)
INTERPOLATION PIPELINERoughly:
1. Linearly interpolate edge lengths and dihedral angles.
2. Nonlinear optimization for vertex positions.
Sum of squares: Gauss-Newton
![Page 61: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/61.jpg)
SOFTWARE
• Matlab: fminunc or minfunc• C++: libLBFGS, dlib, others
Typically provide functions for functionand gradient (and optionally, Hessian).
Try several!
![Page 62: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/62.jpg)
SOME TRICKS
Regularization
![Page 63: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/63.jpg)
SOME TRICKS
Multiscale/graduated optimization
![Page 64: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/64.jpg)
ROUGH PLAN
• (intro) Matrices and Eigenvalues
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 65: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/65.jpg)
LAGRANGE MULTIPLIERS: IDEA
![Page 66: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/66.jpg)
LAGRANGE MULTIPLIERS: IDEA
- Decrease f: −𝛁𝒇- Violate constraint: ±𝛁𝒈
![Page 67: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/67.jpg)
LAGRANGE MULTIPLIERS: IDEA
Want:
![Page 68: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/68.jpg)
USE OF LAGRANGE MULTIPLIERS
Turns constrained optimization into
unconstrained root-finding.
![Page 69: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/69.jpg)
min𝑥∈ℝ𝑛
𝑓(𝑥)
s.t. 𝑔 𝑥 = 0
quadratic
linear
![Page 70: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/70.jpg)
QUADRATIC WITH LINEAR EQUALITY
(assume A is symmetric and positive definite)
![Page 71: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/71.jpg)
QUADRATIC WITH LINEAR EQUALITY
(assume A is symmetric and positive definite)
![Page 72: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/72.jpg)
min𝑥∈ℝ𝑛
𝑓(𝑥)
s.t. 𝑔 𝑥 = 0
nonlinear
nonlinear
![Page 73: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/73.jpg)
MANY OPTIONS
• ReparameterizationEliminate constraints to reduce to unconstrained case
• Newton’s methodApproximation: quadratic function with linear constraint
• Penalty methodAugment objective with barrier term, e.g. 𝒇 𝒙 + 𝝆|𝒈 𝒙 |
![Page 74: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/74.jpg)
EXAMPLE: SYMMETRIC EIGENVECTORS
![Page 75: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/75.jpg)
EXAMPLE: MESH EMBEDDING
G. Peyré, mesh processing course slides
![Page 76: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/76.jpg)
LINEAR SOLVE FOR EMBEDDING
• 𝒘𝒊𝒋 ≡ 𝟏: Tutte embedding• 𝒘𝒊𝒋 from mesh: Harmonic embedding
Assumption: 𝒘 symmetric.
𝑥𝑖 ∈ ℝ2
![Page 77: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/77.jpg)
What if 𝑽𝟎 = {}?
![Page 78: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/78.jpg)
NONTRIVIALITY CONSTRAINT
Prevents trivial solution 𝒙 ≡ 𝟎.
Extract the smallest eigenvalue.
![Page 79: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/79.jpg)
Mullen et al. “Spectral Conformal Parameterization.” SGP 2008.
Easy fix
![Page 80: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/80.jpg)
BASIC IDEA OF EIGENALGORITHMS
![Page 81: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/81.jpg)
TRUST REGION METHODS
Example: Levenberg-Marquardt
Fix (or adjust) damping
parameter 𝝀 > 𝟎.
![Page 82: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/82.jpg)
EXAMPLE: POLYCUBE MAPS
Huang et al. “L1-Based Construction of Polycube Maps from Complex Shapes.” TOG 2014.
Align with coordinate axes
Preserve areaNote: Final method includes more terms!
![Page 83: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/83.jpg)
Convex Optimization Tools
Try lightweight options
versus
![Page 84: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/84.jpg)
ITERATIVELY REWEIGHTED LEAST SQUARES
Repeatedly solve linear systems
“Geometric median”
![Page 85: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/85.jpg)
ALTERNATING PROJECTION
d can be aBregman divergence
![Page 86: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/86.jpg)
AUGMENTED LAGRANGIANS
Add constraint to objective
Does nothing when constraint is satisfied
![Page 87: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/87.jpg)
ALTERNATING DIRECTIONMETHOD OF MULTIPLIERS
(ADMM)
https://web.stanford.edu/~boyd/papers/pdf/admm_slides.pdf
![Page 88: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/88.jpg)
FRANK-WOLFE
https://en.wikipedia.org/wiki/Frank%E2%80%93Wolfe_algorithm
Linearize objective, preserve constraints
![Page 89: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/89.jpg)
ROUGH PLAN
• (intro) Matrices and Eigenvalues
• Linear problems
• Unconstrained optimization
• Equality-constrained optimization
• Variational problems
![Page 90: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/90.jpg)
VARIATIONAL CALCULUS: BIG IDEA
Sometimes your unknowns
are not numbers!Can we use calculus to optimize anyway?
![Page 91: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/91.jpg)
ON THE BOARD
![Page 92: BACKGROUND: LINEAR ALGEBRA AND OPTIMIZATIONbmpix/teaching/6113/2019/lectures/Numerics.pdf · OUR BIAS Numerical analysis is a huge field. Patterns, algorithms, & examples common in](https://reader030.fdocuments.net/reader030/viewer/2022040402/5e81a8a26269b34d4859428c/html5/thumbnails/92.jpg)
GÂTEAUX DERIVATIVE
Analog of derivative at u in ψ direction
Vanishes for all ψ at a critical point!