h264 Overview

0

H.264/AVC Video Coding StandardH.264/AVC Video Coding Standard! Standardization, History, Goals, and Applications! Codec Overview! Video Coding Layer (VCL)

• Picture Partitioning and Interlace Processing• Codec Structure• Motion-Compensated Prediction• Intra Prediction• Prediction Residual Coding• Deblocking Filter• Encoder Test Model

! Performance! Network Abstraction Layer (NAL)

• NAL Units and Types• RTP Carriage and Byte Stream Format

1

The The JVT JVT ProjectProject! ITU-T SG16 H.26P and H.26L plans in 1993 (H.26P became H.263)

! ITU-T Q.6/SG16 (VCEG - Video Coding Experts Group) formed for ITU-T standardization activity for video compression since 1997

! August 1999: 1st test model (TML-1) of H.26L

! December 2001: Formation of the Joint Video Team (JVT)Joint Video Team (JVT)between VCEG and ISO/IEC JTC 1/SC 29/WG 11 (MPEG - Moving Pictures Experts Group) to establish a joint standard project - H.264 / MPEG4H.264 / MPEG4--AVCAVC(similar to H.262 / MPEG-2 Video);

! JVT Chairs: G. J. Sullivan, A. Luthra, and T. Wiegand

! ITU-T Approval: May 2003 – ITU-T SG16 Final Standard Approved

! ISO/IEC Approval: March 2003 - Final Draft International Standard –currently balloting

! Extensions Project: Professional Extensions until April 2004

2

! Improved Coding Efficiency

• Average bit rate reduction of 50% given fixed fidelity

compared to any other standard

• Complexity vs. coding efficiency scalability

! Improved Network Friendliness

• Issues examined in H.263 and MPEG-4 are further improved

• Anticipate error-prone transport over mobile networks and the

wired and wireless Internet

! Simple syntax specification

• Targeting simple and clean solutions

• Avoiding any excessive quantity of optional features or profile

configurations

GoalsGoals

3

ApplicationsApplications

circuit-switched

packet-switched

! Entertainment Video (1-8+ Mbps, higher latency)

• Broadcast / Satellite / Cable / DVD / VoD / FS-VDSL / …

• DVB/ATSC/SCTE, DVD Forum, DSL Forum

! Conversational Services (usu. <1Mbps, low latency)

• H.320 Conversational

• 3GPP Conversational H.324/M

• H.323 Conversational Internet/best effort IP/RTP

• 3GPP Conversational IP/RTP/SIP

! Streaming Services (usu. lower bit rate, higher latency)

• 3GPP Streaming IP/RTP/RTSP

• Streaming IP/RTP/RTSP (without TCP fallback)

! Other Services

• 3GPP Multimedia Messaging Services

4

! Identical specifications have been approved in both ITU-T / VCEG and ISO/IEC / MPEG

! In ITU-T / VCEG this is a new & separate standard• ITU-T Recommendation H.264

• ITU-T Systems (H.32x) will be modified to support it

! In ISO/IEC / MPEG this is a new “part” in the MPEG-4 suite• Separate codec design from prior MPEG-4 visual• New part 10 called “Advanced Video Coding” (AVC – similar to

“AAC” position in MPEG-2 as separate codec)

• MPEG-4 Systems / File Format has been modified to support it

• H.222.0 | MPEG-2 Systems also modified to support it! IETF finalizing RTP payload packetization

Relationship to Other StandardsRelationship to Other Standards

5

The The ScopeScope of Picture and Video of Picture and Video Coding StandardizationCoding Standardization

Only Restrictions on the Bitstream, Syntax, and Decoder are standardized:

• Permits optimization beyond the obvious

• Permits complexity reduction for implementability• Provides no guarantees of quality

Pre-Processing EncodingSource

DestinationPost-Processing& Error Recovery

Decoding

Scope of Standard

6

! Many standards contain different configurations of capabilities – often based in “profiles” & “levels”• A profile is usually a set of algorithmic features• A level is usually a degree of capability

(e.g. resolution or speed of decoding)

! H.264/AVC has three profiles• Baseline (lower capability plus error resilience, e.g.,

videoconferencing, mobile video)• Main (high compression quality, e.g., broadcast)• Extended (added features for efficient streaming)

Profiles & Levels ConceptsProfiles & Levels Concepts

7

H.264|AVC Layer StructureH.264|AVC Layer Structure

Video Coding Layer

Data Partitioning

Network Abstraction Layer

H.320 MP4FF H.323/IP MPEG-2 etc.

Con

trol

Dat

a

Coded Macroblock

Coded Slice/Partition

8

HighHigh--Level VCL SummaryLevel VCL Summary

! Video coding layer is based on hybrid video coding and similar in spirit to other standards but with important differences

! Some new key aspects are:• Enhanced motion compensation• Small blocks for transform coding• Improved de-blocking filter• Enhanced entropy coding

! Substantial bit-rate savings relative to other standards for the same quality

9

Input Video SignalInput Video Signal

• Progressive and interlaced frames can be coded as one unit

• Progressive vs. interlace frame is signaled but has no impact on decoding

• Each field can be coded separately

• Dangling fields

ProgressiveFrame

TopField

BottomField

Interlaced Frame (Top Field First)

. t

10

Partitioning of the PicturePartitioning of the Picture

! Slices: • A picture is split into 1 or several slices

• Slices are self-contained• Slices are a sequence of macroblocks

! Macroblocks:• Basic syntax & processing unit• Contains 16x16 luma samples and 2 x 8x8 chroma samples

• Macroblocks within a slice depend on each other

• Macroblocks can be further partitioned

0 1 2 …

Slice #0

Slice #1

Slice #2

Macroblock #40

11

Flexible Macroblock Ordering (FMO)Flexible Macroblock Ordering (FMO)

Slice Group #0

Slice Group #1

Slice Group #2

! Slice Group: • Pattern of macroblocks defined by a Macroblock allocation map

• A slice group may contain 1 to several slices

! Macroblock allocation map types:• Interleaved slices• Dispersed macroblock allocation• Explicitly assign a slice group to each macroblock location inraster scan order

• One or more “foreground” slice groups and a “leftover” slicegroup

Slice Group #0

Slice Group #1

Slice Group #0

Slice Group #1

Slice Group #2

12

Interlaced ProcessingInterlaced Processing

! Field coding: each field is coded as a separate picture using fields for motion compensation

! Frame coding:

• Type 1: the complete frame is coded as a separate picture

• Type 2: the frame is scanned as macroblock pairs, for each macroblock pair: switch between frame and field coding

Macroblock Pair

0 2

1 3

4

5

36

37

…

…

13

MacroblockMacroblock--Based Frame/Field Adaptive CodingBased Frame/Field Adaptive Coding

A Pair of Macroblocks in Frame Mode

Top/Bottom Macroblocks in Field Mode

14

Scanning of a MacroblockScanning of a Macroblock

0 1

2 3

Coded Block Pattern for Luma in 8x8 block order:signals which of the 8x8 blocks contains at least one 4x4 block with non-

zero transform coefficients Luma 4x4 block order for 4x4 intra prediction and

4x4 residual coding

Chroma 4x4 block order for 4x4 residual coding, shown as

16-25, and intra 4x4 prediction, shown as 18-21

and 22-25

10 4 5

2 3 6 7

8 9 12 13

10 11 14 15

2x2 DC

AC

Cb Cr16 17

-1

...

Intra_16x16 macroblock type only: Luma 4x4 DC

18 19

20 21

22 23

24 25

15

Basic Coding StructureBasic Coding Structure

EntropyCoding

Scaling & Inv. Transform

Motion-Compensation

ControlData

Quant.Transf. coeffs

MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation

Transform/Scal./Quant.-

InputVideoSignal

Split intoMacroblocks16x16 pixels

Intra-frame Prediction

DeblockingFilter

OutputVideoSignal

16

Basic Coding StructureBasic Coding Structure

EntropyCoding


MotionCompensation

ControlData


Intra Prediction

Data

Intra/InterMB select

CoderControl

MotionEstimation


InputVideoSignal



DeblockingFilter

OutputVideoSignal

Intra-frame Estimation

MotionData

17

Common Elements with other StandardsCommon Elements with other Standards

! Macroblocks: 16x16 luma + 2 x 8x8 chroma samples! Input: Association of luma and chroma and

conventional sub-sampling of chroma (4:2:0)! Block motion displacement! Motion vectors over picture boundaries! Variable block-size motion! Block transforms! Scalar quantization! I, P, and B coding types

18

EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

Motion Compensation AccuracyMotion Compensation Accuracy

Motion vector accuracy 1/4 (6-tap filter)

8x8

0

4x8

0 10 1

2 3

4x48x4

1

08x8Types

0

16x16

0 1

8x16MB

Types

8x80 1

2 3

16x8

1

0

19

Quarter Sample Luma InterpolationQuarter Sample Luma Interpolation

bb

a cE F I JG

hd

n

H

m

A

C

B

D

R

T

S

U

M s NK L P Q

fe gji kqp r

aa

b

cc dd ee ff

hh

gg

full sample reference positionsfractional sample positions

! Half sample positions are obtained by applying a 6-tap filter with tap values: (1, -5, 20, 20, -5, 1)

! Quarter sample positions are obtained by averaging samples at integer and half sample positions

20

Chroma Sample InterpolationChroma Sample Interpolation

Chroma interpolation is 1/8-sample accurate since luma motion is 1/4-sample accurate

Fractional chroma sample positions are obtained using the equation:

22 /)2/)()())((( ssDddCddsBdsdAdsdsv yxyxyxyx ++−+−+−−=

BA

C D

dxdy

s-dx

s-dy

21

EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

MotionData

OutputVideoSignal

Multiple Reference FramesMultiple Reference Frames

! Multiple Reference Frames! Generalized B Frames! Weighted Prediction

22

Multiple Reference Frames and Multiple Reference Frames and Generalized BiGeneralized Bi--Predictive FramesPredictive Frames

CurrentPicture

4 Prior Decoded Picturesas Reference

. = 1

1. Extend motion vector by reference picture index .

2. Provide reference pictures at decoder side

Can jointly exploit scene cuts, aliasing, uncovered background and other effects with one approach

. = 3

. = 0

3. In case of bi-predictive pictures: decode 2 sets of motion parameters

23

New Types of Temporal ReferencingNew Types of Temporal Referencing

I P P P P

B B B B B B B B

I P B B P

B B B B B P B B

! Known dependencies (MPEG-1, MPEG-2, etc.)

! New types of dependencies:• Referencing order and display order are decoupled

• Referencing ability and picture type are decoupled

24

EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

IntraIntra PredictionPrediction! Directional spatial prediction

(9 types for luma, 1 chroma)

• e.g., Mode 3: diagonal down/right predictiona, f, k, p are predicted by (A + 2Q + I + 2) >> 2

1

2

3456

7

8

0

Q A B C D E F G HI a b c dJ e f g hK i j k lL m n o p

25

! In addition to shifting in spatial position, and selecting from among multiple reference pictures, each region’s prediction sample values can be• multiplied by a weight, and• given an additive offset

! Some key uses:• Improved efficiency for B coding, e.g.,

– accelerating motion,– multiple non-reference B temporally between reference pics

• Excels at representation of fades:– fade-in– fade-out– cross-fade from scene-to-scene

! Encoder can apply this to both P and B prediction types

WeightedWeighted PredictionPrediction

26

Spatial prediction using surrounding “available” samplesSpatial prediction using surrounding “available” samples

! Available samples are…• Previously reconstructed within the same slice at the

decoder• Inside the same slice

! Luma intra prediction either:• Single prediction for entire 16x16 macroblock

– 4 modes (vertical, horizontal, DC, planar)

• 16 individual predictions of 4x4 blocks– 9 modes (DC, 8 directional)

! Chroma intra prediction:• Single prediction type for both 8x8 regions

– 4 modes (vertical, horizontal, DC, planar)

27

16x16 16x16 IntraIntra PredictionPrediction DirectionsDirectionsMode 0 - Vertical Mode 1 - Horizontal

Pred(x, y) = x, y = 0,…,15 (above and left available)

Pred(x, y) = x, y = 0,…,15 (only left available)

Pred(x, y) = x, y = 0,…,15 (only above available)

∑ ∑= =

>>+−+−15

0'

15

0'

5]16)',1()1,'([x y

yPxP

4]8)',1([15

0'

>>+−∑=y

yP

4]8)1,'([15

0'

>>+−∑=x

xP

28

4x4 4x4 IntraIntra PredictionPrediction DirectionsDirectionsMode 0 - Vertical Mode 1 - Horizontal

Mode 3 – Diagonal Down/Left Mode 4 – Diagonal Down/Right

Mode 2 - DC

+

++

+ + ++

29

4x4 4x4 IntraIntra PredictionPrediction DirectionsDirections

Mode 7 – Vertical-Left Mode 8 – Horizontal-Up

Mode 5 – Vertical-Right Mode 6 – Horizontal-Down

30

7

5410

632

1298

1110

4x4 Boundary Conditions4x4 Boundary Conditions

A B C DIJKL

E F G HQ

EFGH not available since this 4x4block is outside the macroblock –replace EFGH with value of D

14 15

31

EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

TransformTransform CodingCoding

! 4x4 Block Integer Transform

! Repeated transform of DC coeffsfor 8x8 chroma and some 16x16 Intra luma blocks

1 1 1 1

2 1 1 2

1 1 1 1

1 2 2 1

− − = − − − −

H

32

! Separable transform of a block B4x4 of size 4x4

! Th, Tv: horizontal and vertical transform matrix

! 4x4 transform matrix:• Easy implementation (adds and shifts)• Different norms for even and odd rows of the matrix

Integer Transforms (1)Integer Transforms (1)

Thxvx TBTC 4444 ••=

−−−−

−−==

1221

1111

2112

1111

TT hv

33

Quantization of Transform CoefficientsQuantization of Transform Coefficients! Logarithmic step size control! Smaller step size for chroma (per H.263 Annex T)! Extended range of step sizes! Can change to any step size at macroblock level! Quantization reconstruction is one multiply, one add,

one shift

34

Deblocking FilterDeblocking Filter! Improves subjective visual and objective quality of the decoded

picture! Significantly superior to post filtering! Filtering affects the edges of the 4x4 block structure! Highly content adaptive filtering procedure mainly removes

blocking artifacts and does not unnecessarily blur the visual content• On slice level, the global filtering strength can be adjusted to

the individual characteristics of the video sequence• On edge level, filtering strength is made dependent on

inter/intra, motion, and coded residuals• On sample level, quantizer dependent thresholds can turn

off filtering for every individual sample• Specially strong filter for macroblocks with very flat

characteristics almost removes “tiling artifacts”

35

PrinciplePrinciple of Deblocking Filterof Deblocking Filter

One dimensional visualization of an edge position

Filtering of p0 and q0 only takes place if:

1. |p0 - q0| < α(QP)

2. |p1 - p0| < β(QP)

3. |q1 - q0| < β(QP)

Where β(QP) is considerably smaller than α(QP)

Filtering of p1 or q1 takes place if additionally :

1. |p2 - p0| < β(QP) or |q2 - q0| < β(QP)

(QP = quantization parameter)

4x4 Block Edge

p0

q0

p1

p2

q1

q2

36

Order of Order of FilteringFiltering! Filtering can be done on a macroblock basis that is, immediately

after a macroblock is decoded! First, the vertical edges are filtered then the horizontal edges! The bottom row and right column of a macroblock are filtered

when decoding the corresponding adjacent macroblocks

37

Deblocking: Deblocking: SubjectiveSubjective ResultResult forfor IntraIntra

1) Without Filter 2) with H264/AVC Deblocking

Highly compressed first decoded intra pictureat a data rate of 0.28 bit/sample

38

Deblocking: Subjective Result for InterDeblocking: Subjective Result for Inter

1) Without Filter 2) with H264/AVC Deblocking

Highly compressed decoded inter picture

39

Entropy CodingEntropy Coding

EntropyCoding

Inv. Scal. & Transform

Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

40

Variable Variable LengthLength CodingCoding! Exp-Golomb code is used universally for

almost all symbols except for transform coefficients

! Context adaptive VLCs for coding of transform coefficients

• No end-of-block, but number of coefficients is decoded

• Coefficients are scanned backwards

• Contexts are built dependent on transform coefficients

41

Context Adaptive VLC (CAVLC)Context Adaptive VLC (CAVLC)

! Transform coefficients are coded with the following elements:• Number of non-zero coefficients.• Levels and signs for all non-zero coefficients.• Total number of zeros before last non-zero

coefficient.• Run before each non-zero coefficient

42

Number of Coefficients/Trailing ”1s”Number of Coefficients/Trailing ”1s”! Typically the last non-zero coefficients have |Level | = 1

! The number of non-zero coefficients (example: N=6) and number of ”Trailing 1s” (T1s=2) are coded in a combined symbol

• In this way typically > 50% of the coefficients are signalled as T1s and no other level information than sign is then needed for these coefficients.

! The VLC table to use is adaptively chosen based on the number ofcoefficients in neighboring blocks.

C o e f f

0

1

2

3

4

5

6

0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2

43

Reverse Scanning and Level CodingReverse Scanning and Level Coding

! In a forward scan coefficients levels typically start with high values and decrease towards 1 (Trailing ”1s”)

! Therefore the value of the last nonzero coefficient is more accurately predictable than for the first one.

! Efficient adaptation is obtained by

• Start with a default VLC table for the first coefficient in the reverse scan

• The table to use for the next coefficient is then selected based on the context as adapted by previously coded levels in the reverse scan.

• To adapt to a wide variety of input statistics there are 7 structured VLC tables to choose between.

44

Run Information: TotalZeros and RunBeforeRun Information: TotalZeros and RunBefore

! TotalZeros

• This is the total number of zeros before the last nonzero coefficient in a forward scan.

• Since the number of non-zero coefficients (N) is already known, the maximum value of TotalZeros is: 16 – N, and a VLC of appropriate length can be used.

! RunBefore

• Finally, in a reverse scan order, the run before each non-zero coefficient is coded.

• Since this run can take on only a certain set of values, depending on TotalZeros and runs coded so far, a VLC with optimal length and statistics can always be used.

45

BitBit--Rate Rate SavingsSavings forfor CAVLCCAVLC

Bit-rate ReductionRelative to Run-Level UVLC [%] Inter-Picture Coding

0

2

4

6

8

10

12

14

16

18

20

4 8 12 16 20 24 28

Q

Tempete Mobile

Paris Foreman

Silent News

Container

46

ContextContext--based Adaptive Binary based Adaptive Binary Arithmetic Codes (CABAC)Arithmetic Codes (CABAC)

! Usage of adaptive probability models for most symbols

! Exploiting symbol correlations by using contexts

! Restriction to binary arithmetic coding• Simple and fast adaptation mechanism• Fast binary arithmetic codec based on

table look-ups and shifts only

47

CABAC: Technical OverviewCABAC: Technical Overview

Context modeling

Binarization Probability estimation

Coding engine

update probability estimation

Adaptive binary arithmetic coder

Chooses a model conditioned on

past observations

Maps non-binary symbols to a

binary sequence

Uses the provided model for the actual encodingand updates the model

48

Probability Estimation Probability Estimation ! Probability estimation is realized via table look-up! Table contains states and transition rules upon receipt of MPB or LPB

8

9

7

6

5

4

3

2

1

0

1718

1920

2122

2324

25 2627

2829

16

15

30

10

11

1213

14

31 32333435 36

3738

3940

4142

4344

4546474849 5150 52 535455 5657 5859 60 6162 630,00

0,05

0,10

0,15

0,20

0,25

0,30

0,35

0,40

0,45

0,50

0 5 10 15 20 25 30 35 40 45 50 55 60

States = Table Entries

Probabilityof LPB

MPB

LPBState k

State k+1

49

BinarizationBinarization

Mapping to a binary sequence,e.g., using the unary code tree:• Applies to all non-binary syntax

elements except for macroblock type

• Ease of implementation• Discriminate between binary

decisions (bins) by their positionin the binary sequence

⇒ Usage of different models fordifferent bin_num in the table-based arithmetic coder

1 2 3 4 5 6 7 ...Bin_num

....

0 0 0 0 0 0 16

0 0 0 0 0 15

0 0 0 0 14

0 0 0 13

0 0 12

0 11

10

BinarizationSymbol

50

Exploitation of inter-symbol dependencies:Neighboring motion vector components A and B used for conditioning of current symbol C

0 0 1 0 0Binarization:

C=3Current symbol:(motion vector component)

(bit, model_no): (0,1?) (0,2) (1,3)(0,5)(0,5)

C

B

A

1 , if | | | | 2ctx_no(C)

1 , else

a A B

b

+ <=

ctx_no(C)= 11bb|A|=2, |B|=3

Context Modeling Example: Coding of MVContext Modeling Example: Coding of MV

Probability Estimation

Coding Engine

Update Probability Estimation

Adaptive binary arithmetic coder

BinaryEvents

Channel11bb

51

BitBit--Rate Rate SavingsSavings forfor CABAC CABAC

Average Bit-Rate Savings CABAC vs. VLC/CAVLC for SD interlace sequences

0

2

4

6

8

10

12

14

16

18

20

20 24 28 32 36 40QP

bit

-rat

e re

du

ctio

n [

%] Canoe

F1

RugbyFootball

Mobile

52

Coder ControlCoder Control

! Coder control is a non-normative part of H.264/AVC! Goal within standardization process: demonstrate H.264/AVC

performance and make design decisions using common conditions

! Choose coding parameters at encoder side„What part of the video signal should be coded using what method and parameter settings?“

! Constrained problem:

! Unconstrained Lagrangian formulation:

with λ controlling the rate-distortion trade-off

D - DistortionR - RateRT - Target ratep - Parameter Vector

TRRD =)p(s.t.)p(minp

{ })p()p(minargpp

RDopt •.+=

53

RateRate--Constrained Mode DecisionConstrained Mode Decision! For given values of Q and λM, minimize

M - Evaluated macroblock mode out of a set of possible modesQ - Value of quantizer control parameter for transform coefficientsλM - Lagrange parameter for mode decisionD2 - Sum of squared differences (luma & chroma)R - Number of bits associated with header, motion, transform coefficients

! Set of possible macroblock modes• Dependent on frame type (e.g. I, P, B)

• For instance, P frame in H.264|AVC:M. {SKIP, INTER_16x16, INTER_16x8, INTER_8x16,

INTER_8x8, INTRA_4x4, INTRA_16x16}! Prior to macroblock mode decision: sub macroblock (8x8) mode

decision

)|()|(2 QMRQMD M •.+

54

! Integer-pixel motion search as well as fractional sample search is performed by minimizing

m - Motion vector containing spatial displacement and picture reference parameter ∆

pm - Predictor for motion vectorλD - Lagrange parameter for motion estimationD1 - Sum of absolute differences (luminance)R - Number of bits associated with motion information

RateRate--Constrained Motion EstimationConstrained Motion Estimation

)p|m()m(1 mD RD •.+

55

Relationship between Relationship between .. and and QPQP! Experiment:

• Fix Lagrangian multiplier and

• Add modes with quantizer changing (DQUANT)

• Perform rate-constrained mode decision

• See [Wiegand and Girod, ICIP 2001]

M. MD .. =

56

! H.264/AVC:

Relationship between Relationship between .. and and QPQP

! H.263 / MPEG-4p2:2

263.85.0 HM QP•=.

3/)12(285.0 −•= QPM.

6/)12(263. 2 − QP

HQP

MD .. =

MD .. =

⇒

57

! Test of different standards (Trans. on Circuits and Systems for Video Technology, July 2003, Wiegand et al)

! Using same rate-distortion optimization techniques for all codecs

! “Streaming” test: High-latency (included B frames)! “Real-time conversation” test: No B frames! “Entertainment-quality application“ test: SD & HD resolutions! Several video sequences for each test! Compare four codecs:

• MPEG-2 (in Main profile high-latency/streaming test only)• H.263 (High-Latency profile, Conversational High-

Compression profile, Baseline profile)• MPEG-4 Visual (Simple profile and Advanced Simple

profile with & without B pictures)• H.264/AVC (Main profile and Baseline profile)

A A ComparisonComparison of Performanceof Performance

58

! Theoretical performance versus actual implementation quality is a serious consideration

! Need tests on larger body of material for strong statistical significance

! PSNR analysis and perceptual quality can differ

CautionCaution: : YourYour MileageMileage Will Will VaryVary

59

Test Set Test Set forfor StreamingStreaming ApplicationsApplications

Camera zoom; spatial detail; fast random motion8.67 sec.CIFTempete

Slow panning and zooming; complex motion; high spatial and color detail

8.33 sec.CIFMobile & Calendar

Slow and steady camera panning over landscape; spatial and color detail

8.33 sec.CIFFlower Garden

Fast translational motion and camera panning; moderate spatial detail

5 sec.CIFBus

Camera zoom; spatial detail; fast random motion8.67 sec.QCIFTempete

Still camera on human subjects with synthetic background

10 sec.QCIFNews

Still camera on slow moving scene10 sec.QCIFContainer Ship

Fast camera and content motion with pan at the end

10 sec.QCIFForeman

CharacteristicsDurationResolutionName

60

Test Test ResultsResults forfor StreamingStreaming ApplicationApplication

30.61%--H.263 HLP

42.95%16.65%-MPEG-4 ASP

63.57%47.58%37.44%H.264/AVC MP

MPEG-2H.263 HLPMPEG-4 ASPCoder

Average bit-rate savings relative to:

61

ExampleExample StreamingStreaming Test Test ResultResult

Tem pete CIF 15Hz

242526272829303132333435363738

0 256 512 768 1024 1280 1536 1792

Bit-rate [kbit/s]

Y-P

SN

R [

dB

]

MPEG-2 H.263 HLPMPEG-4 ASPH.264/AVC MPTest Points

62

ExampleExample StreamingStreaming Test Test ResultResult

Tempete CIF 15Hz

0%

10%

20%

30%

40%

50%

60%

70%

80%

26 28 30 32 34 36 38

Y-PSNR [dB]

Rat

e sa

vin

g r

elat

ive

to M

PE

G-2

H.263 HLP

H.264/AVC MP

MPEG-4 ASP

63

ComparisonComparison to MPEGto MPEG--4 ASP4 ASPTempete CIF 30Hz

2526272829303132333435363738

0 500 1000 1500 2000 2500 3000 3500

Bit-rate [kbit/s]

QualityY-PSNR [dB]

MPEG-4H.264|AVC

??????--hand hand sideside

??????--hand hand sideside

64

ComparisonComparison to MPEGto MPEG--2, H.263, MPEG2, H.263, MPEG--44Tempete CIF 30Hz

2526272829303132333435363738

0 500 1000 1500 2000 2500 3000 3500

Bit-rate [kbit/s]

QualityY-PSNR [dB]

MPEG-4H.264|AVC

LeftLeft--handhand sideside

RightRight--hand hand sideside

65

Test Set Test Set forfor RealReal--TimeTime ConversationConversation

Still camera on human subject with synthetic background

10 sec.CIFSean

Still camera on human subjects; typical videoconferencing content

10 sec.CIFParis


10 sec.CIFForeman

Fast camera and content motion with landscape passing

10 sec.CIFCarphone

Still camera on human subjects10 sec.QCIFMother & Daughter

Still camera but fast moving subject10 sec.QCIFSilent


10 sec.QCIFForeman

Still camera on human subject with synthetic background

10 sec.QCIFAkiyo


66

Test Test ResultsResults forfor RealReal--TimeTime ConversationConversation

15.69%--MPEG-4 SP

17.63%2.04%-H.263 CHC

40.59%29.37%27.69%H.264/AVC BP

H.263 BaseMPEG-4 SPH.263 CHCCoder


67

ExampleExample RealReal--TimeTime ConversationConversation ResultResult

Paris CIF 15Hz

24252627282930313233343536373839

0 128 256 384 512 640 768

Bit-rate [kbit/s]

Y-P

SN

R [

dB

]

H.263-Base H.263 CHCMPEG-4 SP H.264/AVC BPTest Points

68

ExampleExample RealReal--TimeTime Test Test ResultResult

Paris CIF 15Hz

0%

10%

20%

30%

40%

50%

24 26 28 30 32 34 36 38

Y-PSNR [dB]

Rat

e sa

vin

g r

elat

ive

toH

.263

-Bas

elin

e

H.264/AVC BP

H.263 CHC

MPEG-4 SP

69

ComparisonComparison to MPEGto MPEG--2, H.263, MPEG2, H.263, MPEG--44

2728

2930

31

3233

34

3536

3738

39

0 50 100 150 200 250

Bit-rate [kbit/s]

Foreman QCIF 10Hz

LeftLeft--handhand sideside

RightRight--hand hand sideside

QualityY-PSNR [dB]

MPEG-4H.264|AVC

70

Test Set Test Set forfor EntertainmentEntertainment--QualityQuality ApplicationsApplications

Camera zoom, highly complex motion, high spatial detail

10 sec.1280⋅ 720pPreakness

Static camera, fast complex motion7.67 sec.1280⋅ 720pNight

Translational and random motion; high spatial detail

10 sec.1280⋅ 720pSailormen

Jiggling camera, low contrast, lighting change10 sec.1280⋅ 720pShuttle Start

Scene cut between slow and fast moving scene10 sec.720⋅ 576iNews

Camera and content motion; spatial detail10 sec.720⋅ 576iEntertainment

Fast camera and content motion; high spatial detail

9.92 sec.720⋅ 576iBasketball

Fast camera zoom; local motion8.8 sec.720⋅ 576iHarp & Piano


71

Test Test ResultsResults EntertainmentEntertainment--QualityQuality ApplicationsApplications

45%H.264/AVC MP

MPEG-2Coder


72

ExampleExample EntertainmentEntertainment--QualityQuality ApplicationsApplications ResultResult

Entertainm ent SD (720x576i) 25Hz

24252627282930313233343536373839

0 1 2 3 4 5 6 7 8 9 10

Bit-rate [Mbit/s]

Y-P

SN

R [

dB

]

MPEG-2

H.264/AVC MP

73

ExampleExample EntertainmentEntertainment--QualityQuality ApplicationsApplications ResultResult

Entertainm ent SD (720x576i) 25Hz

0%

10%

20%

30%

40%

50%

60%

26 28 30 32 34 36 38

Y-PSNR [dB]

Rat

e sa

vin

g r

elat

ive

to M

PE

G-2

H.264/AVC MP

74

The various standard decoders together with bit-streams of all test cases presented in this paper can be down-loaded at

ftp://ftp.hhi.de/ieee-tcsvt/

MoreMore ResultsResults ??

T. Wiegand, H. Schwarz, A. Joch, F. Kossentini, and G. J. Sullivan: “Rate-Constrained Coder Control and Comparison of Video Coding Standards,” inIEEE Transactions on Circuits and Systems for Video Technology, July 2003.

MoreMore Details ?Details ?

75

H.264/AVC Layer StructureH.264/AVC Layer Structure

Video Coding Layer

Data Partitioning

Network Abstraction Layer

H.320 H.324 H.323/IP MPEG-2 etc.

Con

trol

Dat

a

Macroblock

Slice/Partition

76

Networks and ApplicationsNetworks and Applications

! Broadcast over cable, satellite, DSL, terrestrial, etc.! Interactive or serial storage on optical and magnetic

devices, DVD, etc. ! Conversational services over ISDN, Ethernet, LAN, DSL

Wireless Networks, modems, etc. or a mixture of several.! Video-on-demand or multimedia streaming services over

ISDN, DSL Ethernet, LAN, Wireless Networks, etc.! Multimedia Messaging Services (MMS) over ISDN, DSL,

Ethernet, LAN, Wireless Network, etc.! New applications over existing and future networks!

How to handle this variety of applications and networks?

77

Network Abstraction LayerNetwork Abstraction LayerMapping of H.264/AVC video to transport layers like! RTP/IP for any kind of real-time wireline and wireless Internet

services (conversational and streaming)! File formats, e.g. ISO MP4 for storage and MMS! H.32X for wireline and wireless conversational services! MPEG-2 systems for broadcasting services, etc.Outside the scope the H.264/AVC standardization, but awareness!

Provision of appropriate mechanisms and interfaces! Provide mapping to network and to facilitate gateway design! Key Concepts: Parameter Sets, Network Abstraction Layer

(NAL) Units, NAL unit and byte-stream formatsCompletely within the scope of H.264/AVC standardization

78

Network Abstraction Layer (NAL) UnitsNetwork Abstraction Layer (NAL) Units

Constraints• Many relevant networks are packet switched networks• Mapping packets to streams is easier than vice versa• Undetected bit-errors practically do not exist on the

application layerArchitecture: NAL units as the transport entity

• NAL units may be mapped into a bit stream…• … or forwarded directly by a packet network• NAL units are self-contained (independently decodable) • The decoding process assumes NAL units in decoding

order• The integrity of NAL units is signaled by the correct size

(conveyed externally) and the forbidden_bit set to 0.

79

Access UnitsAccess Units

access unit delimiter

SEI

primary coded picture

redundant coded picture

end of sequence

end of stream

start

end

80

NAL Unit Format and TypesNAL Unit Format and Types

NAL unit payloadNAL unit header

NAL unit header: 1 byte consisting of! forbidden_bit (1 bit): may be used to signal that a NAL unit is

corrupt (useful e.g. for decoders capable to handle bit errors)! nal_storage_idc (2 bit): signals relative importance, and if the

picture is stored in the reference picture buffer! nal_unit_type (5 bit): signals 1 of 10 different NAL unit types

• Coded slice (regular VCL data),• Coded data partition A, B, C (DPA, DPB, DPC),• Instantaneous decoder refresh (IDR),• Supplemental enhancement information (SEI),• Sequence and picture parameter set (SPS, PPS),• Picture delimiter (PD) and filler data (FD).

NAL unit payload: an emulation prevented sequence of bytes.

81

RTP Payload Format for H.264/AVCRTP Payload Format for H.264/AVC

! The specification of an RTP payload format is on the way within the IETF AVT

! The draft also follows the goals “back-to-basic” and simple syntax specification

! RTP payload specification expects that NAL units are transmitted directly as the RTP payload

! Additional concept of aggregation packets is introduced to aggregate more than one NAL unit into a single RTP packet (helpful for gateway designs between networks with different MTU size requirements)

! RTP time stamp matches presentation time stamp using a fixed 90 kHz clock

! Open Issue: media unaware fragmentation

82

ByteByte--stream Format for H.264/AVCstream Format for H.264/AVC

! Not all transport protocols are packet-based, e.g. MPEG-2 systems over S/C/T, H.320 over ISDN

! H.264/AVC standard defines a byte-stream format to transmit a sequence of NAL units as an ordered stream of bytes

! NAL unit boundaries need to be identified to obtain NAL units with correct size to guarantee integrity

! A byte-oriented HDLC-like framing including start codes (1or 2 bytes) and emulation prevention is specified

! For simplified gateway operation, the emulation prevention on byte basis is applied to all raw byte sequence payloads (RBSPs).

83

Byte AlignmentByte Alignment, Emulation Prevention and FramingEmulation Prevention and Framing

010001000000000000000011101010101010

Sequence of binary video data10100101010101010

Emulation Prevention

1000

Byte Alignment ⇒ Sequence of raw byte sequence payloads

010001000000000000000011101010101010 10100101010101010

010001000000000000000011101010101010 101001010101010000000

0x44 0x00 0x03 0xAA 0xA8 0xA5 0x55 0x00 0x00 0x02

0x44 0x00 0x03 0x03 0xAA 0xA8 0xA5 0x55 0x00 0x03 0x00 0x03 0x02

Slice Boundary

⇒ NAL unit+ NAL unit header

Framing only for Byte Stream Format according to Annex B0x44 0x00 0x03 0x03 0xAA 0xA8 0x00 0x01 0x21 0xA5 0x55 0x00 0x03 0x00 0x03 0x02

0x21

84

Access Unit DelimiterAccess Unit Delimiter! Observation: No Picture Header and no Picture Type

• No need for either in many applications• Their existence harms the performance in some

applications! But: some applications need a picture type

• Primarily Storage Applications, for trick modes! Hence: Introduction of the access unit delimiter

• Optional tool• Signals the picture type and whether the picture is

stored in the reference frame buffer• Inserted before the first NAL unit of a picture in

decoding order, hence signals implicitly the boundary between pictures

85

Data Partitioning NAL Units 1/2Data Partitioning NAL Units 1/2! H.264 | AVC contains Data Partitioning w/ 3 Partitions

• Data partition A (DPA) contains header info– Slice header– All macroblock header information – Motion vectors

• Data partition B (DPB) contains intra texture info– Intra CBPs– Intra coefficients

• Data partition C (DPC) contains inter texture info– Inter CBPs– Inter Coefficients

! When DP is used, all partitions are in separate NAL units

86

Data Partitioning NAL Units 2/2Data Partitioning NAL Units 2/2! Properties of the Partition Types

• DPA is (perceptually) more important than DPB• DPB cleans up error propagation, DPC does not

! Transport DPA w/ higher QoS than DPB, DPC• In lossy transmission environments typically leads to

overall higher reconstructed picture quality at the same bit rate

• Most packet networks contain some prioritization– Sub-Transport and Transport level, e.g. in 3GPP

networks or when using DiffServ in IP– Application Layer protection

– Packet Duplication– Packet-based FEC

87

Parameter Set ConceptParameter Set Concept

NAL unit with VCL Data encodedwith PS #3 (address in Slice Header)

1 2 12

JVT Encoder JVT Decoder

3 3

Parameter Set #3:• Video format PAL• Entr. Code CABAC• ...

Reliable Parameter Set Exchange

! Sequence, random access, picture headers can get lost! Solutions in previous standards: duplication of headers! H.264/AVC coding applies a new concept: parameter sets

88

Parameter Set DiscussionParameter Set Discussion

! Parameter Set: Information relevant to more than one slice• Information traditionally found in sequence / picture

header• Most of this information is static, hence transmission

of a reference is sufficient• Problem: picture-dynamic info, namely timing (TR)• Solution: picture-dynamic info in every slice

– Overhead is smaller than one would expect! Parameter Sets are conveyed out-of-band and reliable

• No corruption/synchronization problems• Aligned with closed control application• Need in-band transmission mechanism for broadcast

89

Nested Parameter SetsNested Parameter Sets! Each slice references a picture parameter set (PPS) to be

used for decoding its VCL data:• PPS selected by short variable length codeword

transported in slice header• Contains, e.g. entropy coding mode, FMO parameters,

quantization initialization, weighted prediction indications, etc.

• PPS reference can change between pictures! Each PPS references a sequence parameter set (SPS)

• SPS is referenced only in the PPS• Contains, e.g. profile/level indication, display

parameters, timing concept issues, etc.• SPS reference can change only on IDR pictures

90

Establishment and Updates of Parameter SetsEstablishment and Updates of Parameter Sets

! If possible, SPS and PPS should be established and updated reliably and out-of-band• Typically established during capability exchange (SIP,

SDP, H.245) or in session announcement,• Updates also possible by control protocols,• SPS and PPS could be pre-defined, e.g. in multicast or

broadcast applications! Special NAL unit types are specified to setup and change

SPS and PPS in-band• Intended ONLY for those applications where no control

protocol is available• Allows to have self-contained byte-streams• Use of in-band and out-of-band Parameter Set

transmission mutually exclusive (to avoid sync problems)

91

Supplemental Enhancement Information (SEI)Supplemental Enhancement Information (SEI)

! Supplemental Enhancement information NAL unit contains synchronously delivered information that is not necessary to decode VCL data correctly

! SEI is helpful for practical decoding or presentation purpose

! An SEI message is associated with the next slice or data partitioning RBSP in decoding order

! Examples are• Display information, absolute timing, etc.• Scene transition information (fades, dissolve, etc.)• Control info for videoconferencing (e.g. FPR)• Error resilience issues, e.g. repetition of reference

picture buffer management information• Arbitrary user data, etc.

92

Summarizing NALSummarizing NAL

! In H.264/AVC, the transport of video has been taken into account from the very beginning

! Flexibility for integration to different transport protocols is provided

! Common structure based on NAL units and parameter sets is maintained for simple gateway operations

! Mapping to MPEG-2 transport stream is provided via byte-stream format

! On the way are payload specification to different transport protocols, e.g. to RTP/IP

93

! Three profiles now: Baseline, Main, and Extended! Baseline (e.g., Videoconferencing & Wireless)

• I and P picture types (not B)• In-loop deblocking filter

• 1/4-sample motion compensation

• Tree-structured motion segmentation down to 4x4 block size

• VLC-based entropy coding (CAVLC)

• Some enhanced error resilience features– Flexible macroblock ordering/arbitrary slice ordering

– Redundant slices

• Note: No support for interlaced video in Baseline

GroupingGrouping of of CapabilitiesCapabilities intointo ProfilesProfiles

94

! Main Profile (esp. Broadcast/Entertainment)• All Baseline features except enhanced error resilience features• B pictures• Adaptive weighting for B and P picture prediction • Picture and MB-level frame/field switching• CABAC• Note: Main is not exactly a superset of Baseline

! Extended Profile (esp. Streaming/Internet)• All Baseline features• B pictures• Adaptive weighting for B and P picture prediction• Picture and MB-level frame/field switching• More error resilience: Data partitioning• SP/SI switching pictures• Note: Extended is a superset of Baseline (but not of Main)

NonNon--BaselineBaseline ProfilesProfiles

95

! Codec design includes relaxation of traditional bounds on complexity (memory & computation) – rough guess 3x decoding power relative to MPEG-2, 4x encoding

! Problem areas:

• Smaller block sizes for motion compensation (cache access issues)

• Longer filters for motion compensation (more memory access)

• Multi-frame motion compensation (more memory for reference frame storage)

• More segmentations of macroblock to choose from (more searching in the encoder)

• More methods of predicting intra data (more searching)

• Arithmetic coding (adaptivity, computation on output bits)

ComplexityComplexity of of CodecCodec DesignDesign

96

! UB Video (JVT-C148) CIF resolution on 800 MHz laptop• Encode: 49 fps• Decode: 137 fps• Encode+Decode: 36 fps• Better quality than R-D optimized H.263+ Profile 3 (IJKT) while using

25% higher rate and low-delay rate control ! Videolocus/LSI (JVT-D023) SDTV resolution

• 30 fps encode on P4 2 GHz with hardware assist• Decode on P3 1 GHz laptop (no hardware assist)• No B frames, no CABAC (approx baseline)

! Tandberg Videoconferencing • All Tandberg end-points ship with H.264/AVC since July 14, ‘03

! Reference software (super slow)! Others: HHI, Deutsche Telekom, Broadcom, Nokia, Motorola, &c! Caution: These are preliminary implementation reports only – mostly

involving incomplete implementations of non-final draft designs

ImplementationsImplementations: : TheThe EarlyEarly ReportsReports

97

CompaniesCompanies PubliclyPublicly KnownKnown to to bebe DoingDoingPreliminaryPreliminary ImplementationImplementation WorkWork

! Amphion! British Telecom! Broadcom (chip)! Conexant (chipset for STB)! DemoGraFX (with bit precision extension)! Deutsche Telekom! Envivio! Equator! Harmonic (filtering and motion estimation)! HHI (PC & DSP encode & decode; demos)! iVast! LSI Logic (chip, plus Videolocus acquisition

demoing real-time FPGA+P4 encode, P4 dec)! Mainconcept! Mobile Video Imaging! Modulus Video! Moonlight Cordless! Motorola ! Nokia! PixelTools! PixSil Technology! Polycom (videoconferencing & MCUs)

! Sand Video (demoed 2 Xilinx FPGA decoder, encode/decode & decode-only chips to fab in ’03)

! Sony (encode & decode, software & hardware, including PlayStation Portable 2004 & videoconferencing systems)

! ST Micro (decoder chip in ‘03)! Tandberg (videoconferencing – shipping in all end

points and as software upgrade)! Thomson! TI (DSP partner with UBV for one of two UBV real-

time implemenations)! Toshiba! UB Video (demoed real-time encode and decode,

software and DSP implementations)! Vanguard Software Solutions (s/w, enc/dec)! VCON

CAUTION: All such information should be considered preliminary and should not be considered to be product announcements – only preliminary implementation work. It will be awhile before robust interoperable conforming implementations exist.

98

! Mainconcept http://www.mainconcept.com/h264.shtml! Mobile Video Imaging http://www.digitalwebcast.com/2003/03_mar/news/dlmvi32703.htm! Modulus Video http://www.modulusvideo.com/! Moonlight Cordless http://www.prweb.com/releases/2003/3/prweb59692.php! PixelTools http://www.pixeltools.com/experth264.html! PixSil Tech http://www.pixsiltech.com/products.htm! Polycom (videoconferencing & MCUs) http://www.polycom.com/investor_relations/0,1406,pw-2573,FF.html! Sand Video http://www.sandvideo.com/pressroom.html! Sony http://www.eetimes.com/issue/mn/OEG20030801S0024 & http://news.sel.sony.com/pressrelease/3691! ST Microelectronics http://www.eetuk.com/tech/news/OEG20021113S0026! Tandberg http://tandberg.net/tb.asp?s=pagesimple&aid={8395730F-6D6F-4101-812F-B10A37412E16}! UB Video http://www.eetimes.com/semi/news/OEG20021202S0048! Vanguard Software Solutions (software encode & decode) http://www.vsofts.com/codec/h264.html! VCON http://www.vcon.com/press_room/english/2003/03031102.shtml

99

ConclusionsConclusions! Video coding layer is based on hybrid video coding and

similar in spirit to other standards but with important differences

! New key features are:••••

! Bit her sta pecially for hig es)

! Sta MPEG

Enhanced motion compensationSmall blocks for transform codingImproved deblocking filterEnhanced entropy coding

-rate savings around 50 % against any otndard for the same perceptual quality (esher-latency applications allowing B picturndard of both ITU-T VCEG and ISO/IEC

h264 Overview

Documents

Transcript of h264 Overview