Colorful Image ColorizationCS 898 Spring 2019

Presented by: Zhixiang Meng

Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European conference on computer vision. Springer, Cham, 2016.

Agenda

● Problem

● Methods

○ Objective Function

○ Annealed Mean

● Evaluation

○ Perceptual Realism Test

○ Semantic Interpretability

○ Task Generalization

ProblemGiven a grayscale photograph as input,

how to hallucinate a plausible color version of the photograph?

ProblemGiven a grayscale photograph as input,

how to hallucinate a plausible color version of the photograph?

“could potentially fool a human observer”

Method

Grayscale Image: L channel Concatenate (L,ab)

L ab

Objective Function

● Euclidean Loss (Unimodal)

○ L2 Regression

○ Favors grayish, desaturated results

Colors in ab space(Continuous)

Objective Function

● Euclidean Loss (Unimodal)

● Multimodal Classification

Colors in ab space(Discrete)

Example

Objective Function

● Euclidean Loss (Unimodal)

● Multimodal Classification

○ Most of the distribution is at the center of the gamut, where

colors are desaturated and bland

○ The predictions will tend to be desaturated

Empirical Distribution of ab Values(in log scale)

Objective Function

● Euclidean Loss (Unimodal)

● Multimodal Classification

● Class rebalancing to encourage learning of rare colors

Empirical Distribution of ab Values(in log scale)

Class Rebalancing

● Class rebalancing to encourage learning of rare colors

Empirical Distribution of ab Values(in log scale)

Annealed Mean

● Choice A: Take the mode of the predicted distribution for each pixel

○ This provides a vibrant but sometimes spatially inconsistent result

Annealed Mean

● Choice A: Take the mode of the predicted distribution for each pixel

○ This provides a vibrant but sometimes spatially inconsistent result

● Choice B: Taking the mean of the predicted distribution

○ This produces spatially consistent but desaturated results

Annealed Mean

● Choice A: Take the mode of the predicted distribution for each pixel

○ This provides a vibrant but sometimes spatially inconsistent result

● Choice B: Taking the mean of the predicted distribution

○ This produces spatially consistent but desaturated results

● Choice C: Re-adjusting the temperature T of the softmax distribution,

and taking the mean of the result

Annealed Mean

Network Architecture

Evaluation

● Colorization Quality

● Perceptual Realism Test

● Set up a “colorization Turing test”

● They show participants real and

synthesized colors for an image,

and ask them to identify the fake

Evaluation

Evaluation

Evaluation

● Semantic Interpretability

● VGG Classification

● Feed colorized images into VGG

model and test the classification

accuracy

Task Generalization

● Cross-Channel Encoding as

Self-Supervised Feature Learning

● To evaluate the feature representation

● Freeze the weights of the network,

provide semantic labels, and train linear

classifiers on each convolutional layer

Task Generalization

● Does the feature representation transfer to other datasets and tasks?

● Fine-tune the model on PASCAL VOC on classification, detection, and segmentation

Task Generalization

● Does the feature representation transfer to other datasets and tasks?

Results