Post on 27-Dec-2015
ENEE631 Digital Image Processing (Spring'04)
Basics on Video Communications andBasics on Video Communications and
Other Video Coding Approaches/StandardsOther Video Coding Approaches/Standards
Spring ’04 Instructor: Min Wu
ECE Department, Univ. of Maryland, College Park
www.ajconline.umd.edu (select ENEE631 S’04) minwu@eng.umd.edu
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
04
)
Based on ENEE631 Based on ENEE631 Spring’04Spring’04Section 13Section 13
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [2]
Quick Review – A Few Basics on VideoQuick Review – A Few Basics on Video
Acquisition, Display, Analog & Digital FormatsAcquisition, Display, Analog & Digital Formats
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
04
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [3]
Video CameraVideo Camera
Frame-by-frame capturing
CCD sensors (Charge-Coupled Devices)
– 2-D array of solid-state sensors– Each sensor corresponding to a pixel– Store in a buffer and sequentially read out– Widely used
small and light
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [4]
Video DisplayVideo Display
CRT (Cathode Ray Tube)– Large dynamic range– Bulky for large display
CRT physical depth has to be similar to screen width
LCD Flat-panel display– Use electrical field to change the optical properties hence the
brightness/color of liquid crystal– Generating the electrical field
by an array of transistors: active-matrix thin-film transistors by plasma
“Active-matrix display” (also known as TFT) has a transistor located at each pixel, allowing display be switched more frequently and less current to control pixel luminance. Passive matrix LCD has a grid of conductors with pixels located at the grid intersections
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [5]
Composite vs. Component VideoComposite vs. Component Video
Component video– Three separate signals for tristimulus color representation or luminance-
chrominance representation – Pro: higher quality– Con: need high bandwidth and synchronization
Composite video– Multiplex into a signal signal– Historical reason for transmitting color TV through monochrome
channel– Pro: save bandwidth– Con: cross talk
S-video: luminance sig. + single multiplexed chrominance sig.
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [6]
Analog Video RasterAnalog Video Raster
Line-by-line “Raster Scan”– Represent line-by-line image frame with 1-D analog
waveform– Synchronization signal for horizontal and vertical retrace
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [7]
Forming Picture on TV Tube (Monochrome)Forming Picture on TV Tube (Monochrome)
How many lines?
From B.Liu EE330S’01 Princeton
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [8]
How Many TV Lines?How Many TV Lines?
Determined by spatial freq. response of HVS
dot
dot
Cannot resolve if
distance > 2000 x separation
(~ 0.03 degree viewing angle)
From B.Liu EE330S’01 Princeton
N = 500 for D=4H
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [9]
Review: Progressive vs. Interlaced scanReview: Progressive vs. Interlaced scanFrom B.Liu EE330S’01 Princeton
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [10]
Analog Color TV SystemsAnalog Color TV Systems
Historical notes – Color TV system had to be compatible with earlier monochrome TV system
3 formats– NTSC ~ North American + Japan/Taiwan – PAL ~ Western Europe + Asia(China) + Middle East– SECAM ~ Eastern Europe + France– What format in your home country?
From Wang’s Preprint Fig.1.5
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [11]
Comparison of Three Analog TV SystemsComparison of Three Analog TV Systems
– Spatial and temporal resolution– Color coordinate– Signal bandwidth– Multiplexing of luminance, chrominance, and audio
(From Wang’s Preprint)
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [12]
NTSCNTSC
4:3 aspect ratio (width:height)
525 lines/frame, 2:1 interlace at field rate 59.94Hz– 483 active lines per frame; vertical retrace takes time of 9 lines– rest for broadcaster’s info. like closed caption
YIQ color coordinate for transmission– RGB primary slightly different from PAL– Orthogonal chrominance
I ~ orange-to-cyan; Q ~ green-to-purple (need less bandwidth)
Multiplexing over 6M Hz total bandwidth– Artifacts due to cross talk between luminance and chrominance
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [13]
NTSC 6MHz Bandwidth NTSC 6MHz Bandwidth From Wang’s Preprint Fig.1.6(b)
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [14]
Analog Video RecordingAnalog Video Recording
Comparison of common formats
From Wang’s Preprint Table 1.2
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [15]
Digital Video FormatsDigital Video Formats ITU-R BT.601 recommendation
Downsampled chrominance– Y Cb Cr coordinate and four subsampling formats
Inter. Telecomm. Union – Radio sector
Wang’sPreprint Fig.1.8
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [16]
From Wang’sPreprint
Table 1.3
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
01
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [17]
ResourceResource
Background and Motivation on Background and Motivation on Multimedia Coding / CommunicationsMultimedia Coding / Communications
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
04
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [18]
Generations of Video CodingGenerations of Video Coding
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
From R.Liu Seminar Course ’00 @ UMCP
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [19]
Channel Bandwidth Channel Bandwidth
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
From R.Liu Seminar Course ’00 @ UMCP
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [20]
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
Storage CapacityStorage Capacity
UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)
From R.Liu Seminar Course ’00 @ UMCP
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [21]
Source Video FormatsSource Video Formats
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
From R.Liu Seminar Course ’00 @ UMCP
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [22]
Application RequirementsApplication Requirements
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
From R.Liu Seminar Course ’00 @ UMCP
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [24]
Other Standard and Considerations for Other Standard and Considerations for
Digital Video Coding Digital Video Coding
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
04
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [25]
Performance Tradeoff for Video CodingPerformance Tradeoff for Video Coding
From R.Liu’s Handbook Fig.1.2:
“mos” ~ 5-pt mean opinion scale of bad, poor, fair, good, excellent
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [26]
H.26x for Video TelephonyH.26x for Video Telephony Remote face-to-face communication: A dream for years
H.26x – Video coding targeted low bit rate– Through ISDN or regular analog telephone line ~ on the order of 64kbps – Need roughly symmetric complexity on encoder and decoder
H.261 (early 1990s)– Similar to simplified MPEG-1 ~ block-based DCT/MC hybrid coder– Integer-pel motion compensation with I/P frame only ~ no B frames– Restricted picture size/fps format and M.V. range
H.263 (mid 1990s) and H.263+/H.263++ (late 1990s)– Support half-pel motion compensation & many options for improvement
H.264 (latest, 2001-): also known as H.26L / JVT / MPEG4 part10
– Hybrid coding framework with many advanced techniques– Focusing on greatly improving compression ratio at a cost of complexity
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [27]
MPEG-2MPEG-2
Extend from MPEG-1
Target at high-resolution high-bit-rate applications
– Digital video broadcasting, HDTV, …– Also used for DVD
Support scalability
Support interlaced video
– Frame pictures vs. Field pictures– New prediction modes for motion compensation related to interlaced
video Use previously encoded fields to do M.E.-M.C.U
MC
P E
NE
E4
08
G S
lide
s (c
rea
ted
by
M.W
u &
R.L
iu ©
20
02
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [30]
Scalability in Video CodecsScalability in Video Codecs
Scalability: provide different quality in a single stream– Stack up more bits on base layer to provide improved quality
Possible ways for achieving scalabilities– SNR Scalability ~ Multiple–quality video services
Basic vs. premium quality
– Spatial Scalability ~ Multiple-dimension displays Display on PDA vs. PC vs. Super-resolution display
– Temporal Scalability ~ Multiple frame rates– Frequency Scalability ~ Blurred version to sharp, detailed version
Layered coding concept facilitates:– Unequal error protection – Efficient use of resources– Different needs from customers – Multiple services
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [31]
SNR ScalabilitySNR Scalability
Two layers with same spatio-temporal resolution but different qualities
base-layerencoder
base-layerdecoder
enhancement-layerencoder
mul
tipl
exer+ -
Video inBase-layerbitsteam
Enhancement-layerbitsteam
Outputbitsteam
From R.Liu Seminar Course @ UMCP
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [36]
MPEG-4MPEG-4
Many functionalities targeting a variety of applications
Introduced object-based coding strategy– For better support of interactive applications & graphics/animation video– Require encoder to perform object segmentation
difficult for general applications
Introduced error resilient coding techniques– “Streaming video profile” for wireless multimedia applications
Part-10 is converged into H.264– Focused on improving compression ratio and error resilience– Stick with Hybrid coding frameworkU
MC
P E
NE
E4
08
G S
lide
s (c
rea
ted
by
M.W
u &
R.L
iu ©
20
02
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [37]
Object-based Coding in MPEG-4Object-based Coding in MPEG-4
Interactive functionalities Higher compression
efficiency by separately handling – Moving objects– Unchanged background– New regions– M.C.-failure regions=> “Sprite” encoding
Object segmentationneeded (not easy )– Based on color, motion,
edge, texture, etc.– Possible for targeted
applications
Revised from R.Liu Seminar Course @ UMCPU
MC
P E
NE
E4
08
G S
lide
s (c
rea
ted
by
M.W
u &
R.L
iu ©
20
02
)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [39]
Analysis-Synthesis Coding: A General FrameworkAnalysis-Synthesis Coding: A General Framework
From R.Liu Seminar Course @ UMCP
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [40]
MPEG-7MPEG-7
“Multimedia Content Description Interface”
– Not a video coding/compression standard like previous MPEG– Emphasize on how to describe the video content for efficient
indexing, search, and retrieval
Standardize the description mechanism of content
– Descriptor, Description Scheme, Description Definition Languages– Example of MPEG-7 visual descriptor: Color, Texture, Shape, …
Figure from MPEG-7 Document N4031 (March 2001)
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [44]