Light Field Compression Using 2-D Warping and Block Matching
description
Transcript of Light Field Compression Using 2-D Warping and Block Matching
![Page 1: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/1.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
1
Light Field Compression Using 2-D Warping and Block Matching
Shinjini KunduAnand Kamat Tarcar
EE398A Final Project
![Page 2: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/2.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
2
Outline
• Motivation and Goals• Overview of Our Method• Results and Analysis• Summary• Future Work• References
![Page 3: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/3.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
3
Motivation
• Light field images are used in computer graphics to compute new views of a scene without need for scene geometry model1.
• Need to compress large set of images• Exploit inter-view coherence to achieve
compression.
1. M. Levoy and P. Hanrahan, “Light field rendering,” in Computer Graphics (Proceedings SIGGRAPH 96), August 1996, pp. 31-42.
![Page 4: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/4.jpg)
Light Fields• Represents a 3D scene or object from all viewing
positions and directions– 2D array of 2D images– Difficult to Acquire– Very Large
• Perfect representation requires images of the order of the resolution
![Page 5: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/5.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
5
Light Field Views
![Page 6: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/6.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
6
Light Field Data Set8.4 MB uncompressed data setshttp://lightfield.stanford.edu/aperture.swf?lightfield=data/lego_lf/preview.zip&zoom=1
Credit: Andrew Adams
![Page 7: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/7.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
7
Related Work• Intra-frame coding
– Vector quantization, DCT coding, transform coding yield compression ratios of less than 30:1
• Inter-frame coding (compression in the hundreds, thousands)– Disparity compensation– 3D geometry models– Blockwise
Compression ideal: maximally use coherence between two images
![Page 8: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/8.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
8
Our Method: 2-D Warping
• Each consecutive view is a projection of the previous view due to constant predictable movement of camera
• Find this relation between the views by obtaining projection matrix for each pair of views
• Predict the view and encode the residual
![Page 9: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/9.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
9
Our Encoding Scheme
Reconstructed Previous View
Previous Frame 2-D Warped
Lagrangian Cost
Function
Cost=R1+λD1
Cost=R2+λD2
2D Warping Algorithm
2-D DCT for the Residual
Residual and MV
?
Input View
--
Use for Reconstruction
![Page 10: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/10.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
10
Notes• DCT used on 8x8 blocks to encode residual• Laplacian distribution assumed for motion vectors• Projection matrix was encoded by normalizing values with
respect to 10, and assuming Laplacian distribution of bitrate. The min and max values are encoded separately using binary encoding.
• H =
-0.578 0.005 -0.720 -0.003 -0.572 0.007 0.000 0.000 -0.582
![Page 11: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/11.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
11
1. Feature match by correlation2. Projective matrix computedLagrangian Mode Decision using two references3. Clipped edges are interpolated using motion
compensation
![Page 12: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/12.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
12
Getting a predicted projection:Step 1: Feature matching by Correlation
corners detected corners detected
Features detected by Harris corner detection algorithm, and matching points identified by maximum correlation
![Page 13: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/13.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
13
Computing the Homography Matrix
• A homography is an invertible transformation from the real projective plane to the projective plane that maps straight lines to straight lines
![Page 14: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/14.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
14
Results for 2-D Projection Warping
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 131
32
33
34
35
36
37
38
Rate, in bits/pixel
PS
NR
(dB
)
Lego Men
motion compensation + DCT onlyprojection method
![Page 15: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/15.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
15
Results for 2-D Projective Warping
0 0.5 1 1.5 2 2.529.5
30
30.5
31
31.5
32Crystal Ball
Rate, in bits/pixel
PS
NR
(dB
)
data is for crystal ball light fieldmotion compensation + DCT onlyprojection method
![Page 16: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/16.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
16
Results for 2D Projective Warping
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.529
30
31
32
33
34
35
36
37
38Lego Men vs. Crystal
PS
NR
(dB
)
Rate, in bits/pixel
Lego MenCrystal Ball
![Page 17: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/17.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
17
Compression Ratios
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 60
50
100
150
200
250
300
350
400
450Compression Ratios
quantizer step size (log(Q))
com
pres
sion
ratio
projective methodmotion compensation only
![Page 18: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/18.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
18
Conclusion
• Advantages: decreased coding complexity, and increased rate/PSNR as well as compression
• Experimental results demonstrate improved coding efficiency with our 2D warp method when compared with MVC.
![Page 19: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/19.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
19
Future Work Possible
• Optimize the code to give better PSNR values and check performance by introducing extra modes like copy mode
• Explore other methods of using inter-view redundancy in detail like disparity compensation at sub-pel accuracy
• Run for larger data sets and optimize complexity of the algorithm
![Page 20: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/20.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
20
Summary
• Light fields represent a 3D scene using sequence of 2-D images
• Large amounts of data• Can use redundancy between images using 2-
D warping with motion compensated block matching
• Results in a sleek method for compression• Performance wise..
![Page 21: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/21.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
21
Acknowledgement
• Prof. Girod for pointing us in the right direction• Mina Makar for his help• Chuo-Ling Chang for DAPBT code• Huizhong Chen and Derek Pang for their help• Prof. Peter Kovesi for open source matlab
function library• Prof. Levoy’s group and Andrew Adams for
access to light field images
![Page 22: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/22.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
22
Questions?
![Page 23: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/23.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
23
Other Projects
• Use Motion Compensation with Directional Transforms
o Result: Gain in PSNR due to directionality is approximately 0.1dB at high Quantization; almost nil increase seen at low quantization
• So, We adapted the direction of out project to study a new approach of compression presented next.
![Page 24: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/24.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
24
Results with Motion Compensation and DAPBT for Crystal light field
![Page 25: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/25.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
25
Results with Motion Compensation and DAPBT for Lego light field
![Page 26: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/26.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
26
This is how blocking is done and direction selection happens!IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field
![Page 27: Light Field Compression Using 2-D Warping and Block Matching](https://reader036.fdocuments.net/reader036/viewer/2022081507/56815dc7550346895dcbf3ca/html5/thumbnails/27.jpg)
EE398A - Compression of Light Fields using 2-D Warping and Block Matching
27
For Lego light field IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field