Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares...
Transcript of Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares...
![Page 1: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/1.jpg)
9/7/2014
1
Iterative Reweighted Least Squares
ECCV, Sept 7, 2014
120
120
120
What point minimizes the distance to the three points of a triangle?
Exercise: find this point using ruler and compass construction.
Toricelli
![Page 2: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/2.jpg)
9/7/2014
2
Ruler and Compass Constructionto find Fermat point
Alfred Weber
![Page 3: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/3.jpg)
9/7/2014
3
Weber’s Solution (1906)
Gustav Weler
Gustav Weler was a political decoy (doppelgänger or Body‐double) of Adolf Hitler. At the end of the Second World War, he was executed by a gunshot to the forehead in an attempt to confuse the Allied troops when Berlin was taken.[citation needed] He was also used "as a decoy for security reasons".[2] When his corpse was discovered in the Reich Chancellery garden by Soviet troops, it was mistakenly believed to be that of Hitler because of his identical moustache and haircut. The corpse was also photographed and filmed by the Soviets.
One servant from the bunker declared that the dead man was one of Hitler's cooks. He also believed this man "had been assassinated because of his startling likeness to Hitler, while the latter had escaped from the ruins of Berlin".[3]
Weler's body was brought to Moscow for investigations and buried in the yard at Lefortovo prison.[4]
![Page 4: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/4.jpg)
9/7/2014
4
Andrew Vázsonyi (1916–2003), also known as Endre Weiszfeld and Zepartzatt Gozinto) was a mathematician and operations researcher. He is known for Weiszfeld's algorithm for minimizing the sum of distances to a set of points, and for founding The Institute of Management Sciences.[1][2][3]
Weiszfeld
E. Weiszfeld, Sur le point pour lequel la somme des distances de n points donnés est minimum, Tôhoku Mathematics Journal 43 (1937), 355 - 386.
![Page 5: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/5.jpg)
9/7/2014
5
Weighted L2 cost function
Robust (L1) cost function
![Page 6: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/6.jpg)
9/7/2014
6
![Page 7: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/7.jpg)
9/7/2014
7
Gradient of L2 distance Gradient of L1 distance
![Page 8: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/8.jpg)
9/7/2014
8
Generalizing IRLS
Functions for which IRLS works
![Page 9: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/9.jpg)
9/7/2014
9
Resistance to Outliers
Robustness to systematic outliers (e.g. ghost edges)
![Page 10: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/10.jpg)
9/7/2014
10
Written without the squares
Define weights
Minimize weighted cost
Start with initial value x(0)
Define weights w(t) from x(t)
Solve WLS problemx(t+1)= argmin C(x, w(t))
t=0
t = t+1
IRLS Algorithm
![Page 11: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/11.jpg)
9/7/2014
11
Start with initial value x(0)
Define weights w(t) from x(t)
Solve WLS problemx(t+1)= argmin C(x, w(t))
t=0
t = t+1
IRLS Algorithm
Start with initial value x(0)
Define weights w(t) from x(t)
Solve WLS problemx(t+1)= argmin C(x, w(t))
t=0
t = t+1
IRLS Algorithm
![Page 12: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/12.jpg)
9/7/2014
12
Start with initial value x(0)
Define weights w(t) from x(t)
Solve WLS problemx(t+1)= argmin C(x, w(t))
t=0
t = t+1
IRLS Algorithm
Start with initial value x(0)
Define weights w(t) from x(t)
Solve WLS problemx(t+1)= argmin C(x, w(t))
t=0
t = t+1
IRLS Algorithm
![Page 13: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/13.jpg)
9/7/2014
13
No square !!
Robust cost function
Required weights
Sum of distances
![Page 14: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/14.jpg)
9/7/2014
14
Weighted residual sum decreases
Robust residual sum decreases
![Page 15: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/15.jpg)
9/7/2014
15
SupergradientA concave function always has a supergradient
Definition of supergradient
Weighted squared residual decreases
Robust costdecreases
![Page 16: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/16.jpg)
9/7/2014
16
Where gradient descent does not converge to a minimum
![Page 17: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/17.jpg)
9/7/2014
17
L1
0.2 0.4 0.6 0.8 1.0
0.2
0.4
0.6
0.8
1.0
0.2 0.4 0.6 0.8 1.0
0.2
0.4
0.6
0.8
1.0
0.2 0.4 0.6 0.8 1.0
2
4
6
8
10
12Advantage: Robust
Disadvantage: • Function not differentiable• Weights not defined at 0• Can stop at non‐minimum.
![Page 18: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/18.jpg)
9/7/2014
18
Lq
0.2 0.4 0.6 0.8 1.0
0.2
0.4
0.6
0.8
1.0
0.2 0.4 0.6 0.8 1.0
0.2
0.4
0.6
0.8
1.0
0.2 0.4 0.6 0.8 1.0
2
3
4
5
6
7
8
Advantage: • Robust• Cost function differentiable everywhere
Disadvantage: • Weights not defined at 0• Can stop at non‐minimum.
Huber
0.2 0.4 0.6 0.8 1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.2 0.4 0.6 0.8 1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0 0.5 1.0 1.5 2.0 2.5 3.0
0.5
1.0
1.5
2.0
Advantage: • Robust• Cost function differentiable• Weights defined at zero• Convex• Guaranteed to converge (at least to local
minimum)
![Page 19: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/19.jpg)
9/7/2014
19
Pseudo Huber
0.2 0.4 0.6 0.8 1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.2 0.4 0.6 0.8 1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0 0.5 1.0 1.5 2.0 2.5 3.0
0.5
1.0
1.5
2.0 Advantage: • Robust• Cost function differentiable• Weights defined at zero• Convex• Guaranteed to converge (at least to local
minimum)
Cauchy
0.5 1.0 1.5 2.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.2 0.4 0.6 0.8 1.0
0.1
0.2
0.3
0.4
0.0 0.2 0.4 0.6 0.8 1.0
0.5
1.0
1.5
2.0 Advantage: • Robust• Cost function differentiable• Weights defined at zero• Guaranteed to converge (at least to local
minimum)
Disadvantage:• Non‐convex• Increased number of local minima
![Page 20: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/20.jpg)
9/7/2014
20
Blake Zisserman
0.5 1.0 1.5 2.0 2.5 3.0
0.2
0.4
0.6
0.8
0.5 1.0 1.5 2.0 2.5 3.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.5 1.0 1.5 2.0 2.5 3.0
0.2
0.4
0.6
0.8
1.0
Advantage: • Very robust to outliers• Cost function differentiable• Weights defined at zero• Guaranteed to converge (at least to local
minimum)
Disadvantage:• Non‐convex• Increased number of local minima
1 2 3 4 5
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1 2 3 4 5
0.2
0.4
0.6
0.8
1.0
1.2
1 2 3 4 5
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Corrupted Gaussian
Advantage: • Robust• Cost function differentiable• Weights defined at zero• Guaranteed to converge (at least to local
minimum)
Disadvantage:• Non‐convex• Increased number of local minima
![Page 21: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/21.jpg)
9/7/2014
21
Example. L1 Regression
![Page 22: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/22.jpg)
9/7/2014
22
![Page 23: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/23.jpg)
9/7/2014
23
![Page 24: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/24.jpg)
9/7/2014
24
![Page 25: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/25.jpg)
9/7/2014
25
![Page 26: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/26.jpg)
9/7/2014
26
![Page 27: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/27.jpg)
9/7/2014
27
![Page 28: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/28.jpg)
9/7/2014
28
Possible application
![Page 29: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/29.jpg)
9/7/2014
29
Example. Averaging Rotations
![Page 30: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/30.jpg)
9/7/2014
30
![Page 31: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/31.jpg)
9/7/2014
31
Angle between two quaternions is half the angle between the corresponding rotations, defined by
All rotations within a delta‐neighbourhood of a reference rotation form a circle on the quaternion sphere.
Isometry of Rotations and Quaternions
![Page 32: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/32.jpg)
9/7/2014
32
Flatten out the meridians (longitude lines)
Azimuthal Equidistant Projection
Angle‐axis representation of Rotations
Rotations are represented by a ball of radius pi in 3‐Dimensional space.
![Page 33: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/33.jpg)
9/7/2014
33
Relative rotations are computed between some nodes in the graph.
Initialization: Propagate rotations estimates across a tree.
![Page 34: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/34.jpg)
9/7/2014
34
Orientation of each node in turn is recomputed, given the known orientation of its neigbours, and the relative orientation.
This involves “single‐rotation averaging”.
![Page 35: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/35.jpg)
9/7/2014
35
o 569 Imageso 280,000 pointso 42000 pairs of overlapping images (more than 30 points in common)
Task: Find the orientations of all cameras.
Extension: Optimization on Riemannian Manifolds
![Page 36: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/36.jpg)
9/7/2014
36
![Page 37: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/37.jpg)
9/7/2014
37
![Page 38: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/38.jpg)
9/7/2014
38
Image from http://en.wikipedia.org/wiki/File:Tangentialvektor.svg
![Page 39: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/39.jpg)
9/7/2014
39
![Page 40: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/40.jpg)
9/7/2014
40
![Page 41: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/41.jpg)
9/7/2014
41
![Page 42: Iterative Reweighted Least Squares · 2014-09-06 · 9/7/2014 1 Iterative Reweighted Least Squares ECCV, Sept 7, 2014 120 120 120 What point minimizes the distance to the three points](https://reader034.fdocuments.net/reader034/viewer/2022042622/5f8c8936a40f40074146a786/html5/thumbnails/42.jpg)
9/7/2014
42
Hyperbolic (negative curvature) manifold