Theory and Practical Implementation of Harmonic Resonant ...
HARMONIC NETWORKS - cs.ucl.ac. · PDF filescale-space Harmonic Analysis Representation H-Nets...
Transcript of HARMONIC NETWORKS - cs.ucl.ac. · PDF filescale-space Harmonic Analysis Representation H-Nets...
HARMONIC NETWORKS: DEEP TRANSLATION AND ROTATION EQUIVARIANCE
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow
OUR GROUP
OUTLINE
• Brief intro: Invariance and Equivariance
• What has been done for rotations?
• Harmonic Networks and the Fourier Shift Theorem
• Appendix: Further theory and future works
INVARIANCE
• Many tasks invariant to input transformations
Doggie DoggieDoggie
INVARIANCE
e.g. translation
Mapping independent of Tg, for all Tg
f(I) = f(Tg[I])Notational aside: (in our case)
Tg[I](x) = I(Mgx+ bg)
Tg[I](x) = I(x+ bg)
INVARIANCE WARNING: DIFFERENT NOTATION TO PAPER
INVARIANCE: EXAMPLES
Var[p] = Var[Tgp]Variance: translations
Fourier spectrum modulus: translations����Z 1
�1I(x)e�◆!x dx
���� =����Z 1
�1I(x� b)e�◆!x dx
����
Pooling: smooth invertible mappings (typ. diffeomorphisms)Z
x2⌦I(x) dx =
Z
x2⌦I(Tgx) dx max
x2⌦I(x) = max
x2⌦I(Tgx)
spectrum[M] = spectrum[QMQ>] QQ> = Q>Q = I
Eigenspectrum: orthogonal transformations
EQUIVARIANCE“The man swallowed the chicken”
versus “The chicken swallowed the man”
EQUIVARIANCE: GENERAL DEFINITION
Sg[f(I)] = f(Tg[I])Mapping preserves algebraic structure of transformation
Sg = Id
Invariance is special case of equivariance
Different representations of same transformation
EQUIVARIANCE: EXAMPLESFourier spectrum phase: translations
e
�◆!b
Z 1
�1I(x)e�◆!x dx =
Z 1
�1I(x� b)e�◆!x dx
Mean: translationsaEp[x] + b = Ep[ax+ b]
[f ⇤ g](x� b) = [Tb[f ] ⇤ g](x)Convolution (and correlation)
|◆!|nh(!) = d@n
x
h(!)
Fourier transform: differentiation (and pretty much any other linear transformation)
[w ? Tbf ](x) =X
y2X
w(y � x)Tbf(y)
=X
y2X
w(y � x)f(y � b)
=X
y02X
w(y0 + b� x)f(y0)
= [w ? f ](x� b)
TRANSLATION EQUIVARIANCE OF Z-CORRELATION
Boundaries imply local equivariance
Source: https://github.com/vdumoulin/conv_arithmetic
LACK OF ROTATION EQUIVARIANCE:
ROTATION EQUIVARIANCE METHODS
Rotate in network Rotate input
Warped Convolutions (Henriques et al., 2017) Polar Transformer Nets (Esteves et al., 2017)
Exploiting Cyclic Symmetry (Dieleman et al., 2016)
Rotate filters Rotate activations Log-polar trans.Brute-forceRotation-Invariant Neoperceptron (Fasel et al., 2006) TI-Pooling (Laptev et al., 2016)
Roto-Translation Scattering(Oyallon et al., 2014) Group Equivariant CNNs(Cohen et al., 2016) Oriented Response Networks (Zhou et al., 2017) Steerable CNNs (Cohen et al., 2017)
Store full response Store max/mean responseRoto-Translation Invariant Scattering(Sifre et al., 2013) Rotation Equivariant Vector Fields (Marcos et al., 2017)
ALL DISCRETE ROTATIONS
"Do you want the wrong answer to the right question or the right answer to the wrong
question? I think you want the former.”
DAVID BLEI
PART II: HARMONIC NETWORKS
THE FOURIER SHIFT THEOREM
Interestingly this generalises beyond 1D shifts to any transformation if considering groups
Equivariant Invariant
m 2 Z
Constraint on angular component of weights
Z ⇡
�⇡I(�� ✓)e�◆m� d� =
Z ⇡�✓
�⇡�✓I(�0)e�◆m(�0+✓) d�0
= e�◆m✓
Z ⇡
�⇡I(�0)e�◆m�0
d�0
CIRCULAR HARMONICS
Constrained
Example with Gaussian radial component
Live in orthogonal spaces, so don’t interactNote: the sign
flip in the angular
component
Free
Wm(r;�) = R(r)e◆(m�+�)
CIRCULAR HARMONIC WEIGHTS: IN PRACTICAL TERMS
Wm ? I = (WRem ? IRe �W Im
m ? IIm) + ◆(WRem ? IIm +W Im
m ? IRe)
Real Imaginary
CIRCULAR HARMONIC WEIGHTS
K K
CIRCULAR HARMONICS
X-correlation w. circular harmonics returns orientation field
Wm2 ? g([Wm1 ? T✓I])
CIRCULAR HARMONIC ALGEBRASo far we have a mapping
But what about: 1) nonlinearities? 2) subsequent layers?
W? : RM⇥N⇥K ! CM⇥N⇥1
NONLINEARITIESTo maintain linear equivariance apply nonlinearities to invariant magnitude only g(|F |e◆\F ) = g(|F |)e◆\F
�(|F |+ c)ReLU(|F |+ c) LeakyReLU(|F |+ c)
CIRCULAR HARMONIC ALGEBRA
?
CIRCULAR HARMONIC ALGEBRA
Sum orders along chained cross-correlations
Wm2 ? g([Wm1 ? T✓I]) = Y e◆(m1+m2)✓
CIRCULAR HARMONIC ALGEBRA
?
???
?
??
?
?
CIRCULAR HARMONIC ALGEBRAAt a layer, we sum over incoming feature maps, but need to pay special attention to rotation order
CIRCULAR HARMONIC ALGEBRAAt a layer, we sum over incoming feature maps, but need to pay special attention to rotation order
CIRCULAR HARMONIC ALGEBRA
The Equivariance Condition
X
p2Pi
mp = i
Path through i
Rotation order of filterConstant at
feature map i
My conjecture: For Lie Groups, streams identified with basis vectors of Lie Algebra, and harmonic filters act as raising/lowering operators, jumping between basis vectors
CIRCULAR HARMONIC ALGEBRA
CIRCULAR HARMONIC ALGEBRA
SOME EXPERIMENTS
20
20
20
20
20
20
CNN
8
8
16
16
35
35
H-Net
m=0
m=110
8
8
H-DSN
m=0 m=1
16
16
32
32
64
64
64
64
1
1
1
1
1
1
DSN
1
1
1
1
1
1
32
32
64
64
128
128
256
256
256
256
Brute force
Rotation + Scale
ORN-8 [30] 1.37 ~1MTI-Pooling [13] 1.21 ~1MPTN-CNN-B++ 0.89 ~254k
Recent results
SOME EXPERIMENTS
CNNCNN + data aug.H Net (ours)
0.880.900.920.940.960.981.00
2000 4000 6000 8000 10000 12000Training set sizeN
orm
aliz
ed te
st a
ccur
acy
SOME EXPERIMENTS
20
20
20
20
20
20
CNN
8
8
16
16
35
35
H-Net
m=0
m=110
8
8
H-DSN
m=0 m=1
16
16
32
32
64
64
64
64
1
1
1
1
1
1
DSN
1
1
1
1
1
1
32
32
64
64
128
128
256
256
256
256
Dynamic Steerable ResNet 0.732 0.751Recent results
SOME EXPERIMENTS
SOME EXPERIMENTS
SOME EXPERIMENTS
SOME EXPERIMENTS
LIMITATIONS
• Filters too basic
• No stability guarantee for noise/high freq. deformations
• Do we really need higher order streams?
• What about other kinds of transformation?
Working on all of these
REFERENCESRotate filters- Deep Roto-Translation Scattering for Object Classification (Oyallon et al., 2014) - Group Equivariant Convolutional Networks (Cohen et al., 2016) - Rotation Equivariant Vector Field Network (Marcos et al., 2017) - Oriented Response Networks (Zhou et al., 2017) - Steerable CNNs (Cohen et al., 2017) - Roto-Translation Invariant Scattering (Sifre et al., 2013)
Rotate activations
Rotate input- Rotation-Invariant Neoperceptron (Fasel et al., 2006) - TI-POOLING: Transformation-invariant Pooling for Feature Learning in Convolutional
Neural Networks (Laptev et al., 2016) - Warped Convolutions: Efficient Invariance to Spatial Transformations (Henriques et al.,
2017) - Polar Transformer Networks (Esteves et al., 2017)
- Exploiting Cyclic Symmetry in Convolutional Neural Networks (Dieleman et al., 2016)
PART III: THEORY
THEORY: THERE IS A LOT
Group theory
Lie groups
Steerable function spaces
Scale-space theory
Gaussian scale-space
Harmonic Analysis
H-NetsRepresentation theory
THE FOURIER SHIFT THEOREM ON LIE GROUPS
Theory is very involved, so this is just a brief outline
GROUP THEORYA group G=(G, ●) is a set G and a binary
composition operator ●, with
1. Closure
2. Associativity
3. Identity element
4. Unique inverses
(f • g) • h = f • (g • h)
g • h 2 G
g • e = e • g = g
g • g�1 = g�1 • g = e
GROUP THEORY: EXAMPLE
y · z 2 C \ {0}(x · y) · z = x · (y · z)
x · 1 = 1 · x = x
x
�1 =1
|x|e�◆\x
1. Closure 2. Associativity 3. Identity element 4. Unique inverses
(C \ {0}, ·)
2D ROTATIONS FORM A GROUPThese are some different ways to describe rotations
SO(2)
([0, 2⇡),+mod 2⇡)
(C, ·)C = {z 2 C : |z|}C = {z 2 C : |z| = 1}
�{A 2 M2⇥2 : A>A = AA> = I, |A| = 1}, ·
�
In fact this group has a manifold structure: it is a Lie Group (Don’t worry if none of this makes sense—I only understand it every 2nd day)
There are an infinite number of representations to describe essentially the same thing
PATCH IS FUNCTION ON SO(2)
FOURIER TRANSFORMS ON LIE GROUPS
PreviouslyZ
f(�)e�◆m� d� e�◆m�e�◆m✓ = e�◆m(�+✓)
m(✓) m(�) = m(✓ • �)Homomorphism property
NowZ
f(�) m(�) d�
(We lose commutativity)Z
GT✓[f ](�) m(�) d� =
Z
Gf(✓�1�) m(�) d�
=
Z
✓�1Gf(�) m(✓�) d(✓�)
= m(✓)
Z
Gf(�) m(�) d�
Need left-invariant Haar measure
InvariantEquivariant
Extension of Fourier Shift Theorem
INVARIANTS ON LIE GROUPSCan show that the irreducible representations are unitary so
kHk2 = hH|Hi = trace(H>H)
Invariant on group
In such a way we can proceed as before to define a harmonic network over a matrix Lie group
kFm{T✓f}k2 = k m(✓)Fm{f}k2
= h m(✓)Fm{f}| m(✓)Fm{f}i
= trace( ˜Fm{f}> ˜ m(✓)
> m(✓)| {z }
I
Fm{f})
= kFm{f}k2
kFm{T✓f}k2 = k m(✓)Fm{f}k2
= h m(✓)Fm{f}| m(✓)Fm{f}i
= trace( ˜Fm{f}> ˜ m(✓)
> m(✓)| {z }
I
Fm{f})
= kFm{f}k2
kFm{T✓f}k2 = k m(✓)Fm{f}k2
= h m(✓)Fm{f}| m(✓)Fm{f}i
= trace( ˜Fm{f}> ˜ m(✓)
> m(✓)| {z }
I
Fm{f})
= kFm{f}k2
kFm{T✓f}k2 = k m(✓)Fm{f}k2
= h m(✓)Fm{f}| m(✓)Fm{f}i
= trace( ˜Fm{f}> ˜ m(✓)
> m(✓)| {z }
I
Fm{f})
= kFm{f}k2
Fm{f} :=
Z
Gf(�) m(�) d�
WORK IN PROGRESSOpen questions:• How to jump between streams?
• Transformations such as scalings are non-compact, can we approximate them well on an interval?
• Not all transformations form groups. Harmonics are eigenfunctions of the Laplace-Beltrami operator.
• Could we define equivariance on general manifolds or even graphs? • What about eigenfunctions of other operators? c.f. Sturm-Liouville theory
• There are other equivariant transforms such as the Gaussian convolution (affine equivariant). This has a semi-group structure, how does this fit in?
THANK YOU