MPEG-4 AVC / H.264 Coding Standard AVC / H.264 ...vca.ele.tue.nl/sites/default/files/5LSE0 Mod 11...

1

1

Fac. EE SPS-VCA

PdW / 2018 Multimedia Video Coding & A / 5LSE0

Module 11 MPEG-4 AVC – H.264

Multimedia Video Coding & Architectures

(5LSE0), Module 11

MPEG-4 AVC / H.264 Coding

Peter H.N. de With

( [email protected] )

2

Fac. EE SPS-VCA



H.264 (MPEG-4 AVC) Standard / Contents

Part 1: General H.264 system properties and outline∗ Introduction to standard and history∗ Compression capabilities & complexity (vs. MPEG-2) ∗ Block diagram and some key properties

Part 2: Video Coding Layer∗ Video data partitioning and macroblocks∗ Motion Compensation and Prediction Modes∗ Integer Transform and Deblocking Filter

Part 3: VLC Coding Techniques and Compr. Performance∗ VLC Techniques: CAVLC and CABAC∗ Performance comparison under various conditions∗ Implementation aspects

3

Fac. EE SPS-VCA



MPEG-4 AVC / H.264 Coding Standard

Part 1: Introduction and overview

Peter H.N. de WithEindhoven Univ. of Technology, The Netherlands

[email protected] )

TU Eindhoven

Faculty EE, SPS-VCA

Flux 5.092

PO Box 513,

5600MB Eindhoven,

The Netherlands

4

Fac. EE SPS-VCA



AVC / H.264 - Preliminaries

∗ Advanced Video Coding definition• Algorithm, Standard • Profiles and Levels for various applications

∗ Rapid acceptance of standard, because … – Emerged from two communities simultaneously: both

IEC MPEG and ITU H.26L standards committees

– Joint Video Team shared development in 1999-2003

– Offers significant improvement in compression

– One major application immediately avalaible: the next generation DVD (2006): Blu-ray Disc (BD) and HD-DVD

2

5

Fac. EE SPS-VCA



AVC / H.264 Intro - Applications

∗ Entertainment Video (1-8+ Mbit/s, higher latency)– Broadcast / Satellite / Cable / DVD / VoD / FS-VDSL / …– DVB / ATSC / SCTE, DVD Forum, DSL Forum

∗ Conversational Services (usually < 1 Mbit/s, low latency)– H.320 Conferencing /Conversational - circuit-switched– 3GPP Conferencing /Conversational H.324/M - circuit-switched– H.323 Conversational Internet / best effort IP/RTP - packet-switch‘d– 3GPP Conversational IP/RTP/SIP - packet-switch‘d

∗ Streaming Services (typ. lower bit rate, higher latency) – 3GPP Streaming IP/RTP/RTSP– Streaming IP/RTP/RTSP (without TCP fallback)

∗ Other Services– 3GPP Multimedia Messaging Services

6

Fac. EE SPS-VCA



AVC / H.264 Intro –

Relation to other standards

∗ Identical specifications have been approved in ITU-T VCEG and

ISO/IEC MPEG

∗ In ITU-T / VCEG this is a new & separate standard

– ITU-T Recommendation H.264

– ITU-T Systems (H.32x) will be modified to support it

∗ In ISO/IEC MPEG this is a new „part“ in the MPEG-4 suite

– Separate codec design from prior MPEG-4 visual

– New Part 10 called Advanced Video Coding (AVC), similar to „AAC“ position in

MPEG-2 as separate codec

– MPEG-4 Systems / File format has been modified to support it

– H.222.0 | MPEG-2 Systems also modified to support it

∗ IETF finalizing RTP payload packetization

7

Fac. EE SPS-VCA



AVC / H.264 Overview /

Evolution Video Codec Technology..1990 2000

ITU-T

ISO / IEC

Same as High level of commonality

CE Industry Consortium

Proprietary

SMPTE

JVT

8

Fac. EE SPS-VCA



Evolution of AVC / H.264

De-quantizer / Inv. Transform

Motion-Compensated

Predictor

0

Intra/Inter

CoderControl

Decoder

MotionEstimator

Transform/Quantizer-

16*1616*88*16

8*88*44*8

4*4

intra, predictive and bi-predictive coding using

multiple reference

pictures

4*4 Integer Transform

UVLC,CABAC

AVC (H.264)

MB

1, 1/2,1/4, 1/8 pel

QuantizedTransf. coeff’s

EntropyCoding

Motiondata

Controldata

3

9

Fac. EE SPS-VCA



H.264 / Benefits of coding with AVC

∗ AVC (H.264 / MPEG-4 part 10)

– About twice as efficient than MPEG-2

• Same content at lower datarate

• Higher resolution content at same datarate

– Architecture comparable to MPEG-2

• More sophisticated tools

• Important differences in many details (4x4 instead of 8x8 DCT)

– Truly open industry standard

– Different profiles allow for usage from mobile applications

to full High Definition TV

10

Fac. EE SPS-VCA



Coding Efficiency (MPEG-2 vs. AVC)Average bitrate at SD resolution (720 * 576/480)

6

5

4

3

2

1

Mb/s

1995 2000 2005

AVC

MPEG-2Improvements due to:• improved encoding algorithm

• technology progress (further integration)• use of variable bitrate instead of constant bitrate

• improved pre-processing (e.g noise reduction)

AVC about 2 x more efficientFor HD possibly less (~ 1.5)

11

Fac. EE SPS-VCA



H.264 / Detailed view on bit-rate efficiency

MPEG-2

H.263

MPEG-4

H.26L H.264

12

Fac. EE SPS-VCA



Complexity of MPEG-2 vs. AVCIC area

1995 2000 2005

AVCMPEG-2

New IC technologies

reduces IC area

Reduced to about ¼ of initial value

AVC about 4 x more complex than MPEG-2

Required IC area now

about the same as for

MPEG-2 at introduction

4

13

Fac. EE SPS-VCA



AVC / H.264 New Features∗ Multi-mode, multi-refer. Motion Compensation

∗ Motion vector can point out of image border

∗ 1/4-, 1/8-pixel motion vector precision

∗ B-frame prediction weighting

∗ 4×4 integer transform

∗ Multi-mode intra-prediction

∗ In-loop de-blocking filter

∗ UVLC (Uniform Variable Length Coding)

∗ NAL (Network Abstraction Layer)

∗ SP-slices

14

Fac. EE SPS-VCA



AVC / H.264 Profiles and Levels – (1)

∗ Many standards contain different configurations of

capabilities – often based in “profiles” & “levels”

– Profile: usually a set of algorithmic features, the so-called

subset of the syntax

– Level: usually a degree of capability, e.g. resolution, speed,

thus the limitations to coding parameters

∗ H.264 / AVC has three profiles

– Baseline - lower capability plus error resilience

– Main - high compression quality

– Extended - added features for efficient streaming

15

Fac. EE SPS-VCA



AVC / H.264 Profiles and Levels – (2)

∗ Profiles applications: Baseline, Main, and Extended (X)

– Baseline: Progressive low-rate video,

Videoconferencing & Wireless communication

– Main: especially for Broadcast applications, Video Recording

– Extended: Mobile networking

∗ Baseline profile is the minimum implementation

– No use of: CABAC, 1/8 MC, B-frame, SP-slices

∗ 15 Levels

– Resolution, capability, bit rate, buffer, reference #

– Built to match popular int. production and transmission formats

– Ranges from QCIF to D-Cinema

16

Fac. EE SPS-VCA



AVC / H.264 Layer Structure∗ Layer structure has extended networking capabilities

– Extended interfacing to many standards

– Good concept with Network Abstraction Layer (NAL)

5

17

Fac. EE SPS-VCA



H.264 / AVC Video Data

Part 2 Video Coding Layer:

A. Data partitioning & Macroblocks

B. Motion Compens. & Prediction Modes

18

Fac. EE SPS-VCA



H.264 High-level Video summary

∗ Video coding layer is based on hybrid video coding

– Similar spirit as other MPEG standards, like DCT + Motion compensation

– But with important differences

∗ New key aspects are

– Enhanced motion compensation

– Small blocks and partly variable block size for transform coding

– Improved deblocking filter

– Enhanced entropy coding

∗ Performance: substantial bit-rate savings relative to other

standards for the same quality

– On top of the above key features: smart control of those features

19

Fac. EE SPS-VCA



AVC / H.264 – Picture partitioning

∗ Picture divided in slices and MBs

∗ Slices

– Picture has 1 or more slices

– Slices are self-contained

– Slices are a sequence of macroblocks

∗ Macroblocks

– Basic syntax & processing unit

– Contains 16x16 Y (luma) samples and

2x ( 8x8 ) Color (chroma) samples

– Macroblocks within a slice depend on

each other

– Macroblocks can be further partitioned

20

Fac. EE SPS-VCA



AVC / H.264 –

Flexible Macroblock Ordering

∗ Slice Group

– Pattern of macroblocks defined by a macroblock allocation map

– Slice group may contain 1 to several slices

∗ Macroblock allocation map types

– Interleaved slices

– Dispersed macroblock allocation

– Explicitly assign a slice group to each macroblock location in raster scan order

– One or more „foreground“ slice groups and „leftover“ slice group

∗ Remember: Picture quality …!

6

21

Fac. EE SPS-VCA



H.264-Basic MB-Based Coding Structure

EntropyCoding

Scaling & Inv. Transform

Motion-Compensation

ControlData

Quant.Transf. coeffs

MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation

Transform/Scal./Quant.-

InputVideoSignal

Split intoMacroblocks

16x16 pixels

Intra-frame Prediction

De-blockingFilter

OutputVideoSignal

Scheme supports MB

field-based processing

for interlacing also, like

in MPEG-2

22

Fac. EE SPS-VCA



H.264 /AVC - Data Encoding Structure

23

Fac. EE SPS-VCA



H.264 / AVC - Motion Compensation - (1)

EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal


16x16 pixels


De-blockingFilter

OutputVideoSignal

Various block sizes and shapes

8x8

0

4x8

0 10 1

2 3

4x48x4

1

08x8

Types

0

16x16

0 1

8x16

MBTypes

8x8

0 1

2 3

16x8

1

0

24

Fac. EE SPS-VCA



Various block sizes and shapes

8x8

0

4x8

0 10 1

2 3

4x48x4

1

08x8

Types

0

16x16

0 1

8x16

MBTypes

8x8

0 1

2 3

16x8

1

0

H.264 / AVC –

Results Motion Compensation - (2)

7

25

Fac. EE SPS-VCA



AVC / H.264 –

Quarter-sample Y interpolation

∗ Half-sample positions are

obtained by applying a 6-tap

filter with tap values (1, -5,

20,20, -5, 1)

∗ Quarter-sample positions are

obtained by averaging

samples at integer and half-

sample positions

26

Fac. EE SPS-VCA



H.264 / AVC - Multiple Reference Frames

EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal


16x16 pixels


De-blockingFilter

OutputVideoSignal

Multiple Reference Frames for

Motion Compensation

27

Fac. EE SPS-VCA



AVC / H.264 – Multiple Reference Frames &

Generalized Bi-Predictive Frames

1. Extend motion vectors by

reference picture index D

2. Provide reference pictures at

the decoder side

3. In case of bi-predictive

pictures: decode 2 sets of

motion parameters

Can jointly exploit scene cuts, aliasing, uncovered

background and other effects with one approach!

28

Fac. EE SPS-VCA



AVC / H.264 –

New types Temporal Referencing

∗ Known dependencies (MPEG-1 and MPEG-2, etc.)

∗ New types of dependencies

– Referencing order and display order are decoupled

– Referencing ability and picture type are decoupled

8

29

Fac. EE SPS-VCA



AVC / H.264 – Weighted Prediction

∗ In addition to shifting in spatial position, and selecting from

multiple ref. pictures, each region‘s prediction values can be

– Multiplied by a weight

– Given an additive offset

∗ Some key issues

– Improved efficiency for B coding, e.g.,

• Accelerating motion,

• Multiple non-reference B temporally between reference pictures

– Excels at representation of fades, such as fade-in, fade-out, cross-fades

between two scenes

∗ Encoder can apply this to both P and B prediction types

30

Fac. EE SPS-VCA



H.264 /AVC B-frame Prediction Weighting

∗ Playback order: I0 B1 B2 B3 P4 B5 B6 ……...

∗ Bitstream order: I0 P4 B1 B3 B2 P8 B5 ……...

I0 B1 B2 B3 P4 B5 B6

Time

31

Fac. EE SPS-VCA



AVC / H.264 – Spatial Prediction

using surrounding „available“ samples ∗ Available samples are …

– Previously reconstructed within the same slice at the decoder

– Inside the same slice

∗ Luma intra prediction either

– Single prediction for entire 16x16 macroblock

• 4 modes (vertical, horizontal, DC, planar)

– 16 individual predictions of 4x4 blocks

• 9 modes (DC, 8-directional)

∗ Chroma intra prediction

– Single prediction type for both 8x8 regions

• 4 modes (vertical, horizontal, DC,planar)

32

Fac. EE SPS-VCA



H.264 / AVC - Intra-prediction Modes

EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal


16x16 pixels


De-blockingFilter

OutputVideoSignal

Directional spatial prediction

(9 types for luma, 1 chroma)

• e.g., Mode 3:

diagonal down/right prediction

a, f, k, p are predicted by

(A + 2Q + I + 2) >> 2

Q A B C D E F G H

I a b c d

J e f g h

K i j k l

L m n o p

M

N

O

P

1

2

3456

7

8

2- DC

0

9

33

Fac. EE SPS-VCA



AVC / H.264 –

16x16 Intra Prediction Directions ∗ Mode 0 – Vertical and Mode 1 - Horizontal

34

Fac. EE SPS-VCA



AVC / H.264 –

4x4 Intra Prediction Directions -(1)

35

Fac. EE SPS-VCA



AVC / H.264 –

4x4 Intra Prediction Directions -(2)

36

Fac. EE SPS-VCA



H.264 / AVC Part 2 Video Coding Layer:

Transform Coding

C. Integer Transform,

Deblocking Filter and

Performance comparison

10

37

Fac. EE SPS-VCA



H.264 / AVC - 4×4 Integer Transform – (1)

EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal


16x16 pixels


De-blockingFilter

OutputVideoSignal

4x4 Block Integer Transform

Main Profile: Adaptive Block Size

Transform (8x4,4x8,8x8)

Repeated transform of DC coeffs

for 8x8 chroma and 16x16 Intra

luma blocks

1 1 1 1

2 1 1 2

1 1 1 1

1 2 2 1

− − = − −

− −

H

38

Fac. EE SPS-VCA



AVC / H.264 – Integer Transform –(2)

∗ Separable transform of a block B4x4 of size 4x4

∗ Th , Tv : horizontal and vertical transform matrix

∗ 4x4 transform matrix:

– Easy implementation

– Different norms for even and odd rows of the matrix

39

Fac. EE SPS-VCA



AVC / H.264 – Deblocking filter

∗ Improves subjective visual & objective quality of decoded picture

∗ Significantly superior to post filtering

∗ Filtering affects the edges of the 4x4 block structure

∗ Highly content-adaptive filtering procedure mainly removes

blocking artifacts and only scarcely blurs the visual content

– At the slice level, the global filtering strength can be adjusted to the individual

characteristics of the video sequence

– At the edge level, filtering strength is made dependent on inter/intra, motion, and

coded residuals

– At the sample level, quantizer-dependent thresholds can turn off filtering for every

individual sample

– At the tiling level, a specially strong filter for macroblocks with very flat

characteristics almost removes „tiling artifacts“

40

Fac. EE SPS-VCA



H.264 – Principle of Deblocking Filter

∗ One-dimensional visualization of an edge position

– Filtering of p0 and q0 only occurs if

where B(QP) is << than α(QP)

– Filtering of p1 and q1 ioccurs if also

∗ (QP is the MPEG quantization parameter)

11

41

Fac. EE SPS-VCA



AVC / H.264 – Order of Filtering

∗ Filtering can be done on macroblock basis: immediate after MB decode

∗ First the vertical edges are filtered, then the horizontal edges

∗ Bottom row and right column of a macroblock are filtered when

decoding the corresponding adjacent macroblocks

42

Fac. EE SPS-VCA



H.264 / AVC –

Subjective Result Deblocking Filter (Inter)

Without filter with H.264/AVC Deblocking

∗ Highly compressed decoded inter picture

∗ Significantly reduces pred. residuals (0.28bpp)

43

Fac. EE SPS-VCA



AVC / H.264 –

Performance Comparison – (1) ∗ Test of different standards (IEEE Tr. CSVT, July 2003)

– Using same rate-distortion optimization techniques for all coders

– Streaming test: high latency (B frames included)

– Real-time conversion test: no B frames

– Entertainment-quality application test: SD & HD resolution

– Several video sequences per test

– Comparison of 4 codecs:

• MPEG-2 (Main Profile high-latency/streaming test only)

• H.263 (High-latency profile, Conversational High-Compression profile,

Baseline profile)

• MPEG-4 Visual (Simple Profile and Adv. Simple Profile with & without B pict.

• H.264/AVC (Main Profile and Baseline Profile)

44

Fac. EE SPS-VCA



AVC / H.264 – Performance – (3)

Test Results Streaming Application

12

45

Fac. EE SPS-VCA




Example Results Streaming

46

Fac. EE SPS-VCA



H.264 / AVC –

Performance -(6)

Subjective

Comparison

47

Fac. EE SPS-VCA




Result Entertainment QualityConclusion:

H.264 / AVC

saves about 50%

bit rate

compared to

MPEG-2 ..!

48

Fac. EE SPS-VCA



H.264 / AVC – Conclusions – (1)

∗ Video coding: also hybrid video coding, but many differences!

– similar ‘in spirit’ to other standards but with important differences

∗ New key features are:

– Enhanced motion compensation (multi-references)

– Small blocks for transform coding (4x4, 4x8, etc)

– Improved deblocking filter

– Enhanced entropy coding (CAVLC and CABAC)

∗ Substantial bit-rate savings (up to 50%) and complexity

– relative to most other standards for same perceptual quality

– Encoders 3-4x more complex than MPEG-2, decoders 2-3x complex

– Standard within both ITU-T VCEG and ISO/IEC MPEG!

MPEG-4 AVC / H.264 Coding Standard AVC / H.264 ...vca.ele.tue.nl/sites/default/files/5LSE0 Mod 11...

Documents

Transcript of MPEG-4 AVC / H.264 Coding Standard AVC / H.264 ...vca.ele.tue.nl/sites/default/files/5LSE0 Mod 11...