Colorful Image Colorization - University of Waterlooyboykov/Courses/cs898/Presentations_201… ·...
25
Colorful Image Colorization CS 898 Spring 2019 Presented by: Zhixiang Meng Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European conference on computer vision . Springer, Cham, 2016.
Transcript of Colorful Image Colorization - University of Waterlooyboykov/Courses/cs898/Presentations_201… ·...
Colorful Image ColorizationCS 898 Spring 2019
Presented by: Zhixiang Meng
Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European conference on computer vision. Springer, Cham, 2016.
Agenda
● Problem
● Methods
○ Objective Function
○ Annealed Mean
● Evaluation
○ Perceptual Realism Test
○ Semantic Interpretability
○ Task Generalization
ProblemGiven a grayscale photograph as input,
how to hallucinate a plausible color version of the photograph?
ProblemGiven a grayscale photograph as input,
how to hallucinate a plausible color version of the photograph?
“could potentially fool a human observer”
Method
Grayscale Image: L channel Concatenate (L,ab)
L ab
Objective Function
● Euclidean Loss (Unimodal)
○ L2 Regression
○ Favors grayish, desaturated results
Colors in ab space(Continuous)
Objective Function
● Euclidean Loss (Unimodal)
● Multimodal Classification
Colors in ab space(Discrete)
Example
Objective Function
● Euclidean Loss (Unimodal)
● Multimodal Classification
○ Most of the distribution is at the center of the gamut, where
colors are desaturated and bland
○ The predictions will tend to be desaturated
Empirical Distribution of ab Values(in log scale)
Objective Function
● Euclidean Loss (Unimodal)
● Multimodal Classification
● Class rebalancing to encourage learning of rare colors
Empirical Distribution of ab Values(in log scale)
Class Rebalancing
● Class rebalancing to encourage learning of rare colors
Empirical Distribution of ab Values(in log scale)
Annealed Mean
● Choice A: Take the mode of the predicted distribution for each pixel
○ This provides a vibrant but sometimes spatially inconsistent result
Annealed Mean
● Choice A: Take the mode of the predicted distribution for each pixel
○ This provides a vibrant but sometimes spatially inconsistent result
● Choice B: Taking the mean of the predicted distribution
○ This produces spatially consistent but desaturated results
Annealed Mean
● Choice A: Take the mode of the predicted distribution for each pixel
○ This provides a vibrant but sometimes spatially inconsistent result
● Choice B: Taking the mean of the predicted distribution
○ This produces spatially consistent but desaturated results
● Choice C: Re-adjusting the temperature T of the softmax distribution,
and taking the mean of the result
Annealed Mean
Network Architecture
Evaluation
● Colorization Quality
● Perceptual Realism Test
● Set up a “colorization Turing test”
● They show participants real and
synthesized colors for an image,
and ask them to identify the fake
Evaluation
Evaluation
Evaluation
● Semantic Interpretability
● VGG Classification
● Feed colorized images into VGG
model and test the classification
accuracy
Task Generalization
● Cross-Channel Encoding as
Self-Supervised Feature Learning
● To evaluate the feature representation
● Freeze the weights of the network,
provide semantic labels, and train linear
classifiers on each convolutional layer
Task Generalization
● Does the feature representation transfer to other datasets and tasks?
● Fine-tune the model on PASCAL VOC on classification, detection, and segmentation
Task Generalization
● Does the feature representation transfer to other datasets and tasks?
Results
