CMPT-884

Jan 18, 2010

Error Concealment

Presented by: Cameron Harvey

CMPT 820

October 19 2010

Internet Basics

The internet is not reliable as a communication channel Packets can become corrupted Packets can get lost Packets can have high latency Packets can arrive out of order

Encoder/decoder

MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 16 FIGURE 2.4

Motion Compensated Prediction

Standard compression techniques use MCP to estimate a macroblock based on previous or future frames

Packet loss will affect not just the frame being decoded, but many other frames as well Errors are propagated temporally until a new I-

frame is encountered

Slices

Slices A slice is composed of a series of macroblocks Slices are non-overlapping Slices are structure in such a way that motion

vectors are all contained within the slice

Slice Coding Schemes

Constant MBs/slice Maximum bytes/slice

Macroblock Allocation Maps

FMO – Flexible Macro-block Ordering New to the H.264/AVC standard is the ability to

define slice groups Groups can be further divided into separate slices

Flexible Macroblock Ordering

FMO provides a useful tool for error concealment

We can use FMO to arrange for slice groups in such a way that macroblocks are not adjacent to macroblocks from the same slice group

Slices are packetized separately and transmitted

Some types of FMO patterns

Y Dhondt, P Lambert, Flexible Macroblock Ordering – an error resilience tool in H.264/AVC, Gent, Belgium, Dec 2004

FMO – Type 1

Slices are interleaved in such a way that a missing slice will be surrounded by many samples which are decoded correctly

MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 31 FIGURE 2.12

Example

Suppose the packet containing macroblocks of slice group #0 is lost

Example (2)

The missing macroblock can be interpolated using the data from the surrounding macroblocks

This is type of concealment is called Spatial Concealment

T

B

RL

Spatial Error Concealment

The key assumption behind Spatial Error Concealment is that image content is continuous and has smooth texture

Spatial Error Concealment

The pixel, Xij, is calculated as the weighted average of four pixel values on the edge of the macroblock

MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 33 FIGURE 2.13

Spatial Error Concealment (2)

The calculation for pixel (i,j) is

The values of determine the impact the border pixels will have and are generally related to the distance from the pixel under consideration

Spatial Error Concealment (3)

This technique will result in blurred areas as the reconstructed macroblock is basically an average of the surrounding macroblocks

Refinements of this technique can be used to recover some texturing information

Spatial Error Concealment (4)

Error concealment without FMO Error concealment using FMO

Picture 11 from a GOP – 5% packet loss

Y Dhondt, P Lambert, Flexible Macroblock Ordering – an error resilience tool in H.264/AVC, Figure 2, Gent, Belgium, Dec 2004

Temporal Error Concealment

Temporal concealment operates with the assumption that a video sequence is continuous

In its simplest form, one can simply copy a missing macroblock from the corresponding macroblock in the previous frame. This is called Previous Frame Concealment (PFC) Useful if the scene is static Problematic if the scene changes rapidly as this

creates many artifacts

Temporal Error Concealment (2)

We can improve on PFC with an assumption of uniform motion. With this assumption, one can examine the motion vectors of the adjacent macroblocks to infer the motion of the lost macroblock

Many possibilities exist. A boundary matching technique can be used to select from the many possibilities

Boundary Matching Technique

The sum of the squares of the differences between border pixels of Top, Right, and Bottom is determined

The motion vector which minimizes this sum is a good candidate

MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 35 FIGURE 2.15

Other Temporal Concealment Refinements

Overlapped Block Motion The average of three 16X16 pixel regions is used

Use Median Motion Vector of T, R, and B Sum of Absolute Difference (instead of

square) Estimate motion with the Top macroblock

Use PFC if top motion vector is unavailabel Multi-hypothesis error concealment uses a

weighted average of multiple frames

Example

Click to see an example of Spatial-temporal error concealment http://w3.antd.nist.gov/public_safety/concealment/

Project_conceal_files/ecdemo.shtml

Temporal Concealment over a Scene Change

Packet loss over a scene change violates the assumption of temporal continuity Scene change detection can be used at the cost

of additional complexity Macroblock information over a scene change

exhibit distinct characteristics

Abrupt Scene Changes

S. Pei and Y Chou, Novel Error Concealment Method With Adaptive Prediction to the Abrupt and Gradual Scene Changes, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 3, Figure 2(a) Apr 2002

Abrupt Scene Changes (2)

Use Spatial Concealment on the P-frame (Why?)

Use Temporal Error Concealment on the B frames Use the P or I frame from the same scene as

reference For IBBP, all four frames must be processed

before concealment can occur

Gradual Scene Changes

S. Pei and Y Chou, Novel Error Concealment Method With Adaptive Prediction to the Abrupt and Gradual Scene Changes, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 3, Figure 2(b) Apr 2002

Gradual Scene Changes (2)

In the first sub-GOP (IBBP or PBBP), temporal concealment can be used for the final P-frame (Why?)

B frames are calculated as a weighted average of the first P or I frame and the final P-frame

Across P frame in different sub-GOP, interpolation is used aided by the large number of intracoded macroblocks

Questions

He must be very ignorant for he answers every question he is asked.

Voltaire