DENOISING OF VOLUMETRIC DEPTH CONFIDENCE FOR VIEW RENDERING

Srinivas Parthasarathy, Akul Chopra, Emilie Baudin, Pravin Kumar Rana, and Markus Flierl

School of Electrical Engineering, KTH Royal Institute of Technology, Stockholm, Sweden

1 MOTIVATION

Free-Viewpoint Television

Digital Camera

Virtual Camera

Depth Map Estimation

Stereo

Matching

Stereo

Matching

view ♯1 view ♯2 view ♯3 view ♯4 view ♯5

depth map ♯2 depth map ♯4

Inconsistency of Multiview Depth Imageryview ♯1 view ♯2 view ♯3

(a) Dancer.

(b) Newspaper.

2 APPROACH

Estimated Depth Maps

depth map ♯1 depth map ♯2 depth map ♯n

Volumetric Depth Confidence

PerspectiveProjection

b

b

b

b

b

Camera 1 Camera 2 Camera 3

A(+1)

B(+2)

C(+3)

D(+2)

E(+1)

(0) Occluded Area

-2-2

-2

-2

-2

-2

-2

-2

-2

TransparentMedium

Example: Confidence Assignment to Voxels

•Each mapped depth pixel from a viewpoint con-tributes +1 as confidence for a voxel

•Each occluded pixel from a viewpoint does notchange the confidence for a voxel

•Each ray traversing in a transparent voxel con-tributes -1 as confidence for a voxel

3 3D WAVELET DENOISING

c© Disney

Why wavelet denoising?

• It can create sparse coefficients and spreads outi.i.d. noise equally among all the coefficients

• It can distinguish between signal and noise effi-ciently

Wavelet Denoising Process

VolumeDecomposition

Thresholding

VolumeReconstruction

Wavelet Coefficients

Modified Coefficients

Noisy Volume

DiscreteWaveletTransform(DWT)

DenoisingMethods

InverseDWT

Denoised Volume

Adaptive Thresholding Using SURE Shrink

Goal: Determine thresholds that remove noise efficiently

•SURE (Stein’s Unbiased Risk Estimator) is usedto estimate sub–band adaptive thresholds

•For multivariate normal observations, SURE offersan unbiased estimate of the expected squared er-ror of the mean

•Let {ci : i = 1, . . . , l} be the noisy wavelet coeffi-cients in the j-th sub-band.

SURE(θ; c) = l − 2|{i : |ci| < θ}| +l∑

i

min(|ci|, θ)2,

where θ is a given threshold and | · | denotes thecardinality of a set

•Set the SURE threshold θS by

θS = argminθ

SURE(θ; c)

4 VIEW RENDERING

Virtual

View

Hole

Filling

DenoisedVolume

Perspective Projection

center depth map

virtual center view

left

depth

map

right

depth

map

left view right view

3DWarping

-by checking depth consistency

-by using 3× 3 median filter

5 EXPERIMENTAL RESULTS

Objective Results

Rendered Views with Confidence Range [-3,3]

Noisy

Volume

db1

Th.=0.01

db2

Th.=1.40

db3

Th.=1.20

db3

Adaptive

db4

Th.=1.00

38.87 38.7 38.71 38.72 38.87 38.64

31.8431.33 31.42 31.43

32.08

31.38

PSNR

[dB]

Dancer

Newspaper

Effect of Confidence Range on NewspaperRendered Views

MPEG

Depth Maps

Noisy Volume db3 Adaptive

31.63

31.84

32.0832.04

32.33 32.32

PSNR

[dB]

[−3,+3]

[−7,+7]

Subjective Results

Rendered views of Newspaper with ConfidenceRange [-7,+7]

(a) Original view. (b) With MPEG depth maps.

(c) With noisy depth volume. (d) With denoised volume.

c© Disney

... making important notes!

•Adaptive thresholding in the 3D wavelet domainimproves the quality compared to rendering withMPEG depth maps and noisy volumetric data,respectively

•For synthetic scenes, the improvement is insignif-icant due to consistent description of the geom-etry

•Volumetric depth confidence improves renderingresults even without wavelet denoising when pro-jecting highest confidence voxels into the cameraplane

6 CONCLUSIONS

•Define volumetric depth confidence to handle in-consistent depth estimates

•Use superposition principle to incorporate confi-dence information from multiple camera views

•Denoise volumetric depth confidence by usingadaptive 3D wavelet thresholding

• Improve the visual quality of rendered views byusing the denoised volumetric depth confidence