Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol...
-
Upload
verity-flowers -
Category
Documents
-
view
214 -
download
0
Transcript of Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol...
![Page 1: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/1.jpg)
![Page 2: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/2.jpg)
Perception-motivated Perception-motivated High Dynamic Range High Dynamic Range
Video Video EncodingEncoding
Perception-motivated Perception-motivated High Dynamic Range High Dynamic Range
Video Video EncodingEncoding
Rafal Mantiuk, Grzegorz Krawczyk,Rafal Mantiuk, Grzegorz Krawczyk,Karol Myszkowski, Hans-Peter SeidelKarol Myszkowski, Hans-Peter SeidelRafal Mantiuk, Grzegorz Krawczyk,Rafal Mantiuk, Grzegorz Krawczyk,
Karol Myszkowski, Hans-Peter SeidelKarol Myszkowski, Hans-Peter Seidel
INFORMATIKINFORMATIK
![Page 3: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/3.jpg)
[log cd/m^2]Luminance
-6 -4 -2 0 2 4 6 8
Bright projector
LCD Monitor
CRT Monitor
Human eye
High Dynamic RangeHigh Dynamic RangeHigh Dynamic RangeHigh Dynamic Range
![Page 4: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/4.jpg)
High vs Low High vs Low Dynamic Range VideoDynamic Range VideoHigh vs Low High vs Low Dynamic Range VideoDynamic Range Video
LDR Video
Intended for
existing displays
Relative pixel brightness
HDR Video
Intended for
the human eye
Photometric or radiometric units [cd/m2, Watt/m2sr]
![Page 5: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/5.jpg)
High Dynamic Range VideoHigh Dynamic Range VideoHigh Dynamic Range VideoHigh Dynamic Range Video
Goal: Efficient encoding of full dynamic range of luminance perceived by the human observer
Luminance-6 -4 -2 0 2 4 6 8
[log cd/m^2]
• 1st demo
![Page 6: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/6.jpg)
OverviewOverviewOverviewOverview
1. HDR Pipeline
2. HDR Video Encoding• Luminance Quantization
• Edge Coding
3. Results • vs. MPEG-4
• vs. OpenEXR
4. Demo & Applications
![Page 7: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/7.jpg)
RelatedRelated Work WorkRelatedRelated Work Work
HDR Pipeline
Acquisition Storage Display
![Page 8: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/8.jpg)
RelatedRelated Work WorkRelatedRelated Work Work
HDR Pipeline
Acquisition Storage Display
• Global Illumination
• HDR Cameras
•HDRC (IMS Chips)
•Lars III (Silicon Vision)
•Autobrite (SMal Camera Technologies)
•LM9628 (National)
•Digital Pixel System (Pixim)
•Technology overview [Nayar2003]
HDRC – IMS Chips
![Page 9: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/9.jpg)
RelatedRelated Work WorkRelatedRelated Work Work
HDR Pipeline
Acquisition Storage Display
• Still images
•Radiance – RGBE [Ward91]
•OpenEXR [Bogart2003]
•logLuv TIFF [Ward98]
•HDR JPEG [Ward2004]
• Video
•No video format
![Page 10: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/10.jpg)
RelatedRelated Work WorkRelatedRelated Work Work
HDR Pipeline
Acquisition Storage Display
• LDR Displays
•But Tone Mapping necessary
• HDR displays start to appear
•University of British Columbia [Seetzen2004]
![Page 11: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/11.jpg)
HDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding Framework
Detail level 1: Input & Output
White: MPEG
Orange: HDR Encoder
HDR
LDR bitstreamVideo encoder
![Page 12: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/12.jpg)
HDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding Framework
Detail level 2: Color Transform
White: MPEG
Orange: HDR Encoder
bitstreamColorTransform Encoder
Video
HDR
LDR YCrCb
L u'v'p
![Page 13: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/13.jpg)
HDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding Framework
Detail level 3: Edge Coding
White: MPEG
Orange: HDR Encoder
ColorTran. Comp.
MotionCodingDCT
Run-length
EdgeCoding
LDR
HDR
bitstream
Variablelength
![Page 14: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/14.jpg)
ColorTran. Comp.
MotionCodingDCT
Run-length
EdgeCoding
LDR RGB
HDR XYZ
bitstream
Variablelength
Encoding of ColorEncoding of ColorEncoding of ColorEncoding of Color
![Page 15: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/15.jpg)
Encoding of ColorEncoding of ColorEncoding of ColorEncoding of Color
How to represent color data?• Floating Points – ineffective compression• Integers – ok, but require quantization
How to quantize color data?• Quantization errors < threshold of perception• Use uniform color space (L*u*v*, L*a*b*) [Ward98]• Find minimum number of bits
Color (u*v*) – 8 bits are enough
![Page 16: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/16.jpg)
Encoding of Encoding of LuminanceLuminanceEncoding of Encoding of LuminanceLuminance
How to quantize luminance?• Gamma correction?• Logarithm?
Integer representation
log
Lu
min
anc
e Y
-4
-2
0
2
4
6
8
log(Y)?
![Page 17: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/17.jpg)
Threshold Versus IntensityThreshold Versus IntensityThreshold Versus IntensityThreshold Versus Intensity
Psychophysical measurements• The smallest perceivable difference Y for a
certain adaptation level YA
• tvi [Ferwerda96, CIE 12/2.1]
YA - Adaptation
Luminance
Ytvi
-2-4 0 2 4 6 8-2
0
2
4
6
log Adaptation Luminance YA
log
Thr
esh
old
Y
![Page 18: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/18.jpg)
Maximum quantization error
Luminance QuantizationLuminance QuantizationLuminance QuantizationLuminance Quantization
-4
-2
0
2
4
6
8
0 2048Integers Lp
log
Lu
min
anc
e Y
f
Ytvi )(
maxe
Just below threshold of perception
![Page 19: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/19.jpg)
Maximum quantization error
Luminance QuantizationLuminance QuantizationLuminance QuantizationLuminance Quantization
-4
-2
0
2
4
6
8
0 2048Integers Lp
log
Lu
min
anc
e Y
f
Ytvi )( ))((2)( 1 ltvif
dl
ld
maxe
decrease threshold
mapping to)(
space in
f
YLl
LLl
P
P
Just below threshold of perception
Capacity function [Ashkihmin02]
Grayscale Standard Display Function [DICOM03]
10 – 11 bits are enough
![Page 20: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/20.jpg)
Luminance QuantizationsLuminance QuantizationsComparisonComparisonLuminance QuantizationsLuminance QuantizationsComparisonComparison
RGBE
-2 0 2 4-4 6 8log
Con
tras
t Thr
esho
ld
log Adapting Luminance
-4
-2
0
2
32-bit LogLuv
cvi11-bit percep. quant.
![Page 21: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/21.jpg)
ColorTran. Comp.
MotionCodingDCT
Run-length
EdgeCoding
LDR RGB
HDR XYZ
bitstream
Variablelength
Edge CodingEdge CodingEdge CodingEdge Coding
![Page 22: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/22.jpg)
Edge CodingEdge Coding: : MotivationMotivationEdge CodingEdge Coding: : MotivationMotivation
HDR video can contain sharp contrast edges• Light sources, shadows
DCT coding of sharp contrast may cause high frequency artifacts
DCT coding Edge coding
![Page 23: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/23.jpg)
Edge CodingEdge Coding: : SolutionSolutionEdge CodingEdge Coding: : SolutionSolution
Solution: Encode sharp edges in spatial domain, the rest in frequency domain
Original Signal
Sharp edge signal
Smoothed signal
Run-length
encoding
DCT
encoding
![Page 24: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/24.jpg)
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
![Page 25: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/25.jpg)
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
![Page 26: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/26.jpg)
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
![Page 27: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/27.jpg)
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
![Page 28: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/28.jpg)
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
![Page 29: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/29.jpg)
ResultsResultsResultsResults
2x size of tone-mapped MPEG-4 video
20-30x saving compared to intra-frame compression (OpenEXR)
27
1
0.55
OpenEXR
HDR Enc.
MPEG-4
Bit-stream Size
![Page 30: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/30.jpg)
Demo & ApplicationsDemo & ApplicationsDemo & ApplicationsDemo & Applications
Display dependent rendering
Choice of tone-mapping
Extended postprocessing
![Page 31: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/31.jpg)
ConclusionsConclusionsConclusionsConclusions
HDR video compression• Modest changes to MPEG-4
• Lpu’v’ color space– Luminance quantization (10-11 bits)
• Edge coding
Applications• On-the-fly tone mapping
• Blooming, motion blur, night vision
• Tuned for display– LDR / HDR Display
![Page 32: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/32.jpg)
AcknowledgmentsAcknowledgmentsAcknowledgmentsAcknowledgments
HDR Images and Sequences• Paul Debevec• SpheronVR• Jozef Zajac• Christian Fuchs• Patrick Reuter
HDR Camera• HDRC(R) VGAx
courtesy of IMS CHIPSwww.hdrc.com
Comments and help• Volker Blanz
• Scott Daly
• Michael Goesele
• Jeffrey Schoner
![Page 33: Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK.](https://reader035.fdocuments.net/reader035/viewer/2022070323/56649dd95503460f94ace918/html5/thumbnails/33.jpg)
Thank youThank youThank youThank you
http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/