C. -Y. Tsai 2006/4/201Communication & Multimedia
Multiview Video CompressionMultiview Video Compression
Student: Chia-Yang TsaiStudent: Chia-Yang TsaiAdvisor: Prof. Hsueh-Ming HangAdvisor: Prof. Hsueh-Ming Hang
Institute of Electronics, NCTUInstitute of Electronics, NCTU
C. -Y. Tsai 2006/4/202Communication & Multimedia
OutlineOutline IntroductionIntroduction Coding methodsCoding methods PerformancePerformance ConclusionsConclusions
C. -Y. Tsai 2006/4/203Communication & Multimedia
ReferencesReferences
A. Smolic, P. Kauff, “Interactive 3-D video representation and coding technologies”, Proceedings of the IEEE, vol. 93, no. 1, pp 98-110, Jan. 2005.
ISO/IEC JTC1/SC29/WG11, “Submissions received in CfP on Multiview Video Coding”, MPEG Document M12969, Bangkok, Thailand, January 2006.
ISO/IEC JTC1/SC29/WG11, “Multiview video compression using V frames”, MPEG Document M12828, Bangkok, Thailand, January 2006.
C. -Y. Tsai 2006/4/204Communication & Multimedia
IntroductionIntroduction
C. -Y. Tsai 2006/4/205Communication & Multimedia
Multivew= Multiple ViewpointsMultivew= Multiple Viewpoints
Applications Applications of multiviewof multiview 3D video 3D video Free Free
viewpoints viewpoints selectionselection
C. -Y. Tsai 2006/4/206Communication & Multimedia
Multivew= Multiple ViewpointsMultivew= Multiple Viewpoints
Reasons for multiview compressionReasons for multiview compression PC is powerful enoughPC is powerful enough Increasing network bandwidthIncreasing network bandwidth Future 3D videoFuture 3D video
C. -Y. Tsai 2006/4/207Communication & Multimedia
MPEG StandarizationMPEG Standarization
Call for proposal (N7567, Oct. 2005)Call for proposal (N7567, Oct. 2005) Proposal competition (M12969, Jan. 2006)Proposal competition (M12969, Jan. 2006)
NTT and Nagoya UniversityNTT and Nagoya University Thomson and University of Southern CaliforniaThomson and University of Southern California KDDI Corp.KDDI Corp. ETRI and Sejong University (=M12871)ETRI and Sejong University (=M12871) MERL (=M12828)MERL (=M12828) KBS and Yonsei University (=M12874)KBS and Yonsei University (=M12874) Fraunhofer-HHI (=M12945)Fraunhofer-HHI (=M12945) Technical University of BerlinTechnical University of Berlin
C. -Y. Tsai 2006/4/208Communication & Multimedia
Coding MethodsCoding Methods
Disparity compensated view prediction (DCVP)Disparity compensated view prediction (DCVP)View synthesis prediction (VSP)View synthesis prediction (VSP)
Hierarchical B framesHierarchical B frames
C. -Y. Tsai 2006/4/209Communication & Multimedia
Multiview Frame StructureMultiview Frame Structure
1 2 3 4 5 6 7
.
.
.
…..
time
view
C. -Y. Tsai 2006/4/2010Communication & Multimedia
Block diagramBlock diagram
Predictions based on H.264/AVC JM95Predictions based on H.264/AVC JM95
View i picture
Motion Compensation
Motion Estimation
Entropy Coding
Bitstream+ Quantization
IQuantization
+
Transform
ITransform
Deblocking Filter
Reference Picture
Store for View i
Reference Picture
Store for Other Views
Disparity/Illumination
Compensation
Disparity/Illumination Estimation
Intra Prediction
-
+
+
+
Mode Decision
C. -Y. Tsai 2006/4/2011Communication & Multimedia
Block diagramBlock diagram DecoderDecoder
C. -Y. Tsai 2006/4/2012Communication & Multimedia
DCVPDCVP DCVP= Disparity Compensated View DCVP= Disparity Compensated View
PredictionPrediction ProblemsProblems
High spatial correlations between High spatial correlations between different viewpointsdifferent viewpoints
SolutionSolution Prediction between viewpointsPrediction between viewpoints
C. -Y. Tsai 2006/4/2013Communication & Multimedia
DCVPDCVP DCVP= Disparity Compensated View PredictionDCVP= Disparity Compensated View Prediction
…..
I B B P B B I B B P B B I
I B B P B B I B B P B B I
I B B P B B I B B P B B I
I B B P B B I B B P B B I
I B B P B B I B B P B B I
P
B
B
C. -Y. Tsai 2006/4/2014Communication & Multimedia
VSPVSP
VSP= View Synthesis PredictionVSP= View Synthesis Prediction ProblemsProblems
Different viewpoints have different Different viewpoints have different depthdepth Rotation, translation speedRotation, translation speed
SolutionSolution Synthesis virtual images before real Synthesis virtual images before real
predictionprediction
C. -Y. Tsai 2006/4/2015Communication & Multimedia
VSPVSPtime
view
View Synthesis ViaView Interpolation
View Synthesis ViaView Warping
)(],,,[]1,,[)()(],,[ 1 cTyxtcDyxcAcRwvu
)}'(],,{[)'()'(]',','[ 1 cTwvucRcAzyx
R: Rotation matrixR: Rotation matrix
D: Depth informationD: Depth information
T: Translation matrixT: Translation matrix
A: Intrinsic matrixA: Intrinsic matrix
C
C’
C. -Y. Tsai 2006/4/2016Communication & Multimedia
How to get depths?How to get depths?
From camera recordFrom camera record From well-known computer vision algorithmsFrom well-known computer vision algorithms Block-based depth searchBlock-based depth search
||]',',,'[],,,[||),,,( min}
max...2
min,
min,
min{
yxtcIyxtcIyxtcD
Dstep
DDstep
DDDd
where || I [c,t,x,y] – I [c’,t,x’,y’] || denotes the average error between the block at (x,y) in camera c at time t
C. -Y. Tsai 2006/4/2017Communication & Multimedia
How to get depths?How to get depths?
Depths map:Depths map: Left: computer vision algorithmLeft: computer vision algorithm Right: block based depth searchRight: block based depth search
Compression result:Compression result: Depth information: 5-10% total bitratesDepth information: 5-10% total bitrates Left and right have equal performanceLeft and right have equal performance
C. -Y. Tsai 2006/4/2018Communication & Multimedia
Prediction structurePrediction structure
C. -Y. Tsai 2006/4/2019Communication & Multimedia
Hierarchical B picturesHierarchical B pictures
Hierarchical B picturesHierarchical B pictures Fully compatible to AVC Main profileFully compatible to AVC Main profile Non-dyadic decomposition is availableNon-dyadic decomposition is available
A B3 B2B1 AB3 B3 B3B2
L3 H1 H2H3H2 L3H1 H1 H1
GOP boundaries
AVC Main Profilecompatible base layer
MCTF enhancementlayer
C. -Y. Tsai 2006/4/2020Communication & Multimedia
Hierarchical B picturesHierarchical B pictures
C. -Y. Tsai 2006/4/2021Communication & Multimedia
PerformancePerformance
C. -Y. Tsai 2006/4/2022Communication & Multimedia
ExperimentsExperiments
C. -Y. Tsai 2006/4/2023Communication & Multimedia
ExperimentsExperiments
C. -Y. Tsai 2006/4/2024Communication & Multimedia
ConclussionConclussion
C. -Y. Tsai 2006/4/2025Communication & Multimedia
ConclussionConclussion
DCVP & VSP can reduce the correlation DCVP & VSP can reduce the correlation between viewpointsbetween viewpoints
Future workFuture work Depth search algorithmsDepth search algorithms Motion synthesisMotion synthesis
MCTFMCTF Correlation between temporal and viewpoints axisCorrelation between temporal and viewpoints axis
Rate-controlRate-control ScalabilityScalability Error protectionError protection Low delay issueLow delay issue
C. -Y. Tsai 2006/4/2026Communication & Multimedia
Thanks for your attention!Thanks for your attention!Any questions?Any questions?
Top Related