ForgetMeNot: Memory-Aware Forensic Facial Sketch...
Transcript of ForgetMeNot: Memory-Aware Forensic Facial Sketch...
![Page 1: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/1.jpg)
ForgetMeNot: Memory-Aware Forensic Facial Sketch Matching
Authors: Ouyang, Hospedales, Song, Li
Slides by Josh Kelle
1
![Page 2: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/2.jpg)
Overview
• VIPSL dataset
• experiment goals
• experiment results
• conclusion
2
![Page 3: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/3.jpg)
VIPSL Dataset
• Photographs of 200 faces with neutral expression
• Each photo was sketched by 5 different artists
artist A B C D E
3
![Page 4: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/4.jpg)
Artist Style
Artist A
Artist B
4
![Page 5: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/5.jpg)
Goal: re-sketch in a different style
Gaussian Process
input sketch from artist A
output sketch in the style of artist B
5
![Page 6: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/6.jpg)
output HOG features
HOG representation
Gaussian Process
input sketch invert HOG features
input HOG features
6
![Page 7: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/7.jpg)
Training the GP• Treat each HOG image as a vector in ℝ2560.
• Use PCA to reduce this to ℝ150, although this didn’t produce a noticeable improvement.
• GP: ℝ150 → ℝ150
• Then convert GP output back to ℝ2560 hog space.
X = { }
Y = { }
7
![Page 8: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/8.jpg)
Results for A→B model
input GP prediction ground truth
• The prediction’s gradients look less sharp, which is good. • I was surprised to see more gradients around the outside of the head.
8
![Page 9: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/9.jpg)
Results for A→B model
• It looks like the GP is smoothing too much. • Hypothesis: the GP is putting too much emphasis on the mean face.
input GP prediction ground truth
9
![Page 10: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/10.jpg)
Reverse direction: B to A
B to A
A to BA
B
GPAB
GPBA
A has more gradient
activity than B
A has more gradient
activity than B
10
![Page 11: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/11.jpg)
Quantifying Style Similarity
11
• Measure similarity of sketch style by L2 distance in HOG space.
where xi(A) is the HOG representation of the i-th sketch from artist A
![Page 12: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/12.jpg)
Quantifying Style Similarity
12
Lowest A→B error (err = 91)
A BA BA→B
prediction
Highest A→B error (err = 176)
![Page 13: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/13.jpg)
Which artists have similar style?
• For each pair of artists X→Y, measure average prediction error.
A B C D E
A 0 129.52 119.99 119.27 125.82
B 129.52 0 120.8 121.32 122.95
C 119.99 120.8 0 114.05 121.02
D 119.27 121.32 114.05 0 104.03
E 125.82 122.95 121.02 104.03 0
13
D and E are most similar
A and B are most different
![Page 14: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/14.jpg)
Which artists have similar style?
A B C D E
A 0 129.52 119.99 119.27 125.82
B 129.52 0 120.8 121.32 122.95
C 119.99 120.8 0 114.05 121.02
D 119.27 121.32 114.05 0 104.03
E 125.82 122.95 121.02 104.03 0
A B C D E
14
![Page 15: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/15.jpg)
Chaining
GPAB
input sketch from artist A
reconstructed sketch in the style of artist B
GPBC
reconstructed sketch in the style of artist C
…
15
![Page 16: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/16.jpg)
Chaining
• Does chaining reduce error?
• Average E→C error is 121.
• avg_err(E→D) = 104avg_err(D→C) = 114
• Compare error between E→C vs E→D→C chain.
16
test set index
chaining improved
chaining did not improve
average
![Page 17: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/17.jpg)
Chaining
• Does chaining reduce error?
• Average E→C error is 121.
• avg_err(E→D) = 104avg_err(D→C) = 114
• Compare error between E→C vs E→D→C chain.
17
test set index
chaining improved
chaining did not improve
average
![Page 18: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/18.jpg)
Chaining (best and worst case example)
18
E D C
chaining improved the most
chaining improved the least
![Page 19: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/19.jpg)
Chaining (best test case example)
19
E→D→C E→C
![Page 20: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/20.jpg)
Chaining (worst test case example)
20
E→D→C E→C
![Page 21: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/21.jpg)
Chaining
• Differences are too slight to see a difference in HOG images.
• Error is ~ 100. Difference in error ~3. Most extreme gains and losses are only about 3% different.
• I’m not convinced chaining significantly improves results.
21
![Page 22: ForgetMeNot: Memory-Aware Forensic Facial Sketch Matchingvision.cs.utexas.edu/381V-fall2016/slides/kelle_expt.pdf · 2016. 11. 10. · ForgetMeNot: Memory-Aware Forensic Facial Sketch](https://reader035.fdocuments.net/reader035/viewer/2022062606/5feb3b5cce59923652644af6/html5/thumbnails/22.jpg)
Conclusions
• Gaussian Processes can be used to learn the relation between sketch images.
• It’s not perfect. More data or a different feature space may help.
• The authors’ use of multi-task learning helped alleviate the problem of small data.
22