Efficient Regression for Computational Imaging:

from Color Management to Omnidirectional Superresolution

Maya R. Gupta

Eric Garcia

Raman Arora

Regression

2

Regression

Regression

Linear Regression: fast, not good enough

Problem: Device Dependent Colors Depend on Device

Color Management For each device, characterize the mapping between the native

color space and a device independent color space.

8/5/2009 7

CIELab (Lab)

ICC Profile

ICC Profile

ICC Profile

ICC Profile

Color Management • For each device, characterize the mapping between the native

color space and a device independent color space.

8/5/2009 8

CIELab (Lab)

ICC Profile

ICC Profile

ICC Profile

ICC Profile

CIELab is a widely used device-independent color space that is

perceptually uniform (i.e. Euclidean distance approximates human

judgement of color dissimilarity)

Color Management • For each device, characterize the mapping between the native

color space and a device independent color space.

8/5/2009 9

CIELab (Lab)

ICC Profile

ICC Profile

ICC Profile

ICC Profile

Mapping from RGB -> CIELab and CIELab -> CMYK can be highly

nonlinear

Gamut mapping: linear transforms not adequate

Skin

tones Skin

tones

Original gamut

Extended gamut

Original Gamut Linear regression Nonlinear regression

Creating Custom Color Enhancements

original transformed by artist to “sunset”

2 hrs. work in Photoshop

Ex: simulating illumination effects

Example Convert an image to how it would look in Cinecolor based on 16 sample color pairs

www.widescreenmuseum.org

Original cinecolor

Color management: speed by LUT

8/5/2009 13

Color management: speed by LUT

8/5/2009 14

Color management: speed by LUT

15

Color management: speed by LUT

Color management: speed by LUT

Color management: speed by LUT

Color management: speed by LUT

Linear Interpolation is linear in the outputs

Linear Interpolation is linear in the outputs

Linear Interpolation is linear in the outputs

Lattice Regression Choose the lattice outputs to minimize the post-linear

interpolation empirical risk on the data:

8/5/2009 23

Lattice Regression Choose the lattice outputs to minimize the post-linear

interpolation empirical risk on the data:

8/5/2009 24

Lattice Regression Choose the lattice outputs to minimize the post-linear

interpolation empirical risk on the data:

Effect of Different Lattice Regression Regularizers

8/5/2009 26

Effect of Different Lattice Regression Regularizers

8/5/2009 27

Lattice Regression Closed Form Solution

9/15/2010 28

Sparse: No more than 7dm non-zero entries (of m2) with cubic interpolation.

Example Color Management Results

Example Color Management Results

9/15/2010 31

Omnidirectional Super-resolution:

Omnidirectional Superres Related Work

State of the Art:

Arican and Frossard 2008-2009 (ICPR 2008 Best Paper Award)

• Interpolation with spherical harmonics

• Alignment with an iterative conjugate gradient approach.

Lattice Regression Approach Finding the correct registration of the low-resolution images is

challenging non-convex optimization problem.

Evaluate a candidate registration:

use lattice regression on image subset -> high-res spherical grid

sum interpolation error for all left-out low res image data

Lattice Regression Approach Finding the correct registration of the low-resolution images is

challenging non-convex optimization problem.

Evaluate a candidate registration:

use lattice regression on image subset -> high-res spherical grid

sum interpolation error for all left-out low res image data

Finding the optimal joint registration is a 3(N-1)-d opt. problem

We use FIPS to find the global optimum.

9/15/2010 36

Visual Homing

START

.

. .

HOME

. .

.

.

Lattice Regression Better For Visual Homing

Some Conclusions

Some Conclusions

Some Conclusions

Some Conclusions

For details, see: • “Optimized Regression for Efficient Function Evaluation,” Eric K. Garcia,

Raman Arora, and Maya R. Gupta, (in review – draft upon request).

• “Lattice Regression”, Eric K. Garcia, Maya R. Gupta, Neural Information Processing Systems (NIPS) 2009.

• “Building Accurate and Smooth ICC Profiles by Lattice Regression,” Eric K. Garcia, Maya R. Gupta, 17th IS&T Color Imaging Conference 2009.

• "Adaptive Local Linear Regression with Application to Printer Color Management," Maya R. Gupta, Eric K. Garcia, and Erika Chin, IEEE Trans. on Image Processing , vol. 17, no. 6, 936-945, 2008.

• "Learning Custom Color Transformations with Adaptive Neighborhoods," Maya R. Gupta, Eric K. Garcia, and Andrey Stroilov, Journal of Electronic Imaging, vol. 17, no. 3, 2008.

• "Gamut Expansion for Video and Image Sets," Hyrum Anderson, Eric K. Garcia, and Maya R. Gupta, Computational Color Imaging Workshop, 2007.

8/5/2009 42

Color is an event

light source

human

cones respond:

human

perceives

color

L = long wave = red

M = medium wave = green

S = short wave = blue

reflection

What does it mean to see black?

light source

human

cones respond

???

human

perceives

color

L = long wave = red

M = medium wave = green

S = short wave = blue

What does it mean to see white?

light source

human

cones respond

???

human

perceives

color

L = long wave = red

M = medium wave = green

S = short wave = blue

What does it mean to see white? images from: www.omatrix.com/uscolors.html

You can see “white” given

light made up of 2-spectra

Color Science Crash Course

• What we see can be represented by three primaries.

8/5/2009 47

Stiles-Burch 10° color matching

functions averaged across 37

observers . Adapted from (Wyszecki

& Stiles, 1982) by handprint.com.

monochromatic

light at some

wavelength

match

mixture of three

primary colors

Color Distances

• CIELab • Based on spectral

measurements of color, integrated over CMF envelopes.

• Euclidean distance between two colors approximates the perceptual difference noticed by a human observer.

• Distance metrics created to correct for perceptual non- uniformities in the space:

8/5/2009 48

image source: www.handprint.com

2-D and 3-D Simulation

8/5/2009 49

d=2

d=3

Color printer

8 bit RGB color patch printed

color patch Human eye

Measure CIEL*a*b*

Color management for printers

Goal: Print a given CIEL*a*b* value. Problem: What RGB value to input?

Inverse Device Characterization

8/5/2009 51

CIELab

Step 1 Sample the device

Step 2 Build an inverse look-up-table

Regression

Look-up-table

Output Measure

8/5/2009 52

Gaussian Process Regression • Models data as being drawn from a Gaussian Process

8/5/2009 53

(L large, σ2 small) (L small, σ2 small) (L large, σ2 large)

• A leading method in geostatistics (2-d regression) also known as Kriging.

• Generally considered a state-of-the-art method by machine learning folks

• Parameters: Covariance Function (length scale L), Noise Power σ2.

Gaussian Process Regression • Models data as being drawn from a Gaussian Process

8/5/2009 54

(L large, σ2 small) (L small, σ2 small) (L large, σ2 large)

• A leading method in geostatistics (2-d regression) also known as Kriging.

• Generally considered a state-of-the-art method by machine learning folks

• Parameters: Covariance Function (length scale L), Noise Power σ2.

• Given Covariance form, parameters can be learned by maximizing marginal likelihood. (i.e. automatically from data).

2-D Simulation

8/5/2009 55

Gaussian Process Regression (Direct) Gaussian Process Regression (to nodes of lattice) Lattice Regression (GPR bias) Lattice Regression (Bilinear bias)

50 Training Samples 1000 Training Samples

3-D Simulation

8/5/2009 56

Gaussian Process Regression (Direct) Gaussian Process Regression (to nodes of lattice) Lattice Regression (GPR bias) Lattice Regression (Bilinear bias)

50 Training Samples 1000 Training Samples