Multimedia Video Coding & Architectures 5LSE0), …vca.ele.tue.nl/sites/default/files/5LSE0 Mod 01...

1

1

PdW / 2017

Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 /

Module 01 Introduction coding

Multimedia Video Coding & Architectures

(5LSE0), Module 01

Introduction to coding aspects

Peter H.N. de With([email protected] )

slides version 1.0

2

PdW / 2017

Fac. EE SPS-VCA



Lecture Information

∗ LecturerProf.dr.ir. Peter H.N. de WithFaculty Electrical Engineering, University Technology EindhovenSPS – Video Coding & Architectures (VCA)Address: Flux 5.092, Tel: 040-247.25.40, Email: [email protected]

∗ Information: http://vca.ele.tue.nl/courses/5LSE0– Manuscript– Lecture slides– Software Practical Exercises

∗ Examination: Oral Examn (70%) + Practical exercises (30%)

3

PdW / 2017

Fac. EE SPS-VCA



Mod 01 Part 1:

The Need for Compression

4

PdW / 2017

Fac. EE SPS-VCA



What is Digital Signal Coding?

∗ Digital signals: audio, image, video (multimedia)

– Will not discuss document compression

– No detailed lectures on still picture coding (not yet)

∗ Coding/Compression = compact representation:

– Less space required on storage media (hard disks, CD, DVD)

– Smaller transmission bandwidth required (Internet, terrestrial broadcast, satellite links, mobile phones)

– Allows for faster uploading/downloading of multimedia files (Internet, authoring environment)

2

5

PdW / 2017

Fac. EE SPS-VCA



Why Digital Signal Coding? – (1)

(40% of original size)

1200 lines x 1600 pixels per line

RGB, 24 bit (3 bytes) per color pixel

Total uncompressed (raw, bmp) size is 5.8 Mbyte

• 36 photo’s film: 200 Mbyte (about 1/3 of a CD-ROM)

• Download time: 1.5 hours (GSM); 12 minutes

(ISDN/56k); 46 seconds (ADSL); 5 seconds (slow

IntraNet); <1 second (fast IntraNet)

Digital camera technology 2002

(approx. 400 Euro)

6

PdW / 2017

Fac. EE SPS-VCA



Why Digital Signal Coding? – (2)

576 lines x 720 pixels per line

YUV, 16 bit (2 bytes) per color pixel

Total uncompressed (raw, bmp) size per frame is 830

kByte

• 1 hour of video 75 GByte

• Download time: 16 hours (slow IntraNet); 2 hours

(fast IntraNet)

Digital (MiniDV) camcorder technology

1995 - 2010 (1000 Euro)

7

PdW / 2017

Fac. EE SPS-VCA



What is Digital Signal Coding? – (1)

∗ Digital signals: audio, image, video (multimedia)– Will not discuss document compression in detail

∗ Coding/Compression = compact representation:– Less space required on storage media

– Smaller transmission bandwidth required

– Allows for faster uploading/downloading of multimedia files / communication

∗ Examples of “products” using compression:

– Digital audio: MP3

– Digital image and video cameras: JPEG, MPEG

– Internet movie downloading/streaming DivX, Qtime, Real players

– Speech compression for mobile phones: GSM

8

PdW / 2017

Fac. EE SPS-VCA



What is Digital Signal Coding? – (2)

∗ Objectives of this course:

– Fundamental limitations of compression: Information Theory

– Techniques by which compression can be achieved: Signal

Processing, the key steps

– What is inside the MPEG standard, DVB, DV etc.

– Provide an introduction to architectural trade-offs

– Practical experiments with compression factor and quality:

Software package

3

9

PdW / 2017

Fac. EE SPS-VCA



Compression is now an embedded system

∗ “Coding” includes Compression, Security, and Error Control

techniques

∗ This course focuses on Compression

Compress

Packetization and

Multiplexing

A/D

Channel

symbols

Baseband

modulation

Signal to be coded x(t), x(t,s)

Secure

RF/IF

modulate

FEC

10

PdW / 2017

Fac. EE SPS-VCA



Lossless vs. Lossy Compression – (1)

001010010110001011010101

Compressed photo

Identical

Compress Decompress),( jix ),( jiy

Lossless or Reversible Compressiony(i,j) = x(i,j)

Not very effective for multimediaUsed for document compression (Zip)

11

PdW / 2017

Fac. EE SPS-VCA



Lossless vs. Lossy Compression – (2)

001010010110001011010101

Compressed photo

Not Identical

Compress Decompress

Lossy or Non-reversible Compression

y(i,j) ≠ x(i,j)

Examples: JPEG, MPEG, MP3, DivXCan trade compression for amount of difference (quality)

),( jix),( jiy

12

PdW / 2017

Fac. EE SPS-VCA



Some Ballpark Figures – (1)∗ MP3 Audio/Music

» CD: 44.1 kHz at 16 bits/sample = 700 kbit/sec (mono);

Stereo 1.44 Mbit/sec

» MP3: 128 kbit/sec

» Bit rate : 1.5 bit/sample (bps)

» Compression factor : 11

∗ GSM Speech

» 8 kHz at 8 bit/sample = 64 kbit/sec

» GSM: approx. 13 kbit/sec

» Bit rate : 1.6 bit/sample (bps)

» Compression factor : 4.9

4

13

PdW / 2017

Fac. EE SPS-VCA



Some Ballpark Figures – (2)∗ JPEG Images

» Digital camera: 1600x1200 pixels at 24 bit/color pixel = 46

Mbit/file = 5.8 MByte/file

» “Good quality” JPEG: 600 kByte

» Bit rate : 2.5 bit / color pixel (bpp)


∗ MPEG Video

» Broadcast TV: 720x576 at 16 bit/color pixel at 25

pictures/second = 166 Mbit/sec = 20 MByte/sec

» “Commercial tv station” MPEG: 4.5 MBit/sec

» Bit rate : 0.43 bit / pixel (bpp)


14

PdW / 2017

Fac. EE SPS-VCA



Compression is not “for free” – (1)

“lossy”

15

PdW / 2017

Fac. EE SPS-VCA


Module 01 Introduction coding0.6 bit per pixel (Cf=13)

Compression is not “for free” – (2)16

PdW / 2017

Fac. EE SPS-VCA




5

17

PdW / 2017

Fac. EE SPS-VCA



Compression is not “for free” – (4)18

PdW / 2017

Fac. EE SPS-VCA




∗ Same holds for video

High quality MPEG Low quality MPEG

∗ And Audio

High quality MP3 Low quality MP3

19

PdW / 2017

Fac. EE SPS-VCA



Quality of Compressed Signals – (1)

∗ Visual evaluation (always important)

∗ Numerical evaluation: PSNR

Coder Decoder

Visual evaluationPSNR

Original signal

X YRbit rate

Compressed signal

20

PdW / 2017

Fac. EE SPS-VCA



Quality of Compressed Signals – (2)

Number Score Quality Scale Impairment Scale

5 Excellent Imperceptible

4 Good (Just) perceptible but

not annoying

3 Fair (Perceptible and)

slightly annoying

2 Poor Annoying (but not

objectionable)

1 Unsatisfactory (Bad) Very annoying

(Objectionable)

6

21

PdW / 2017

Fac. EE SPS-VCA



Quality of Compressed Signals

decibels) = (dB )],(),([

255log10

Ratio Noise-to-SignalPeak

2

2

10

−

==

jiYjiX

PSNR

15

20

25

30

35

40

45

50

55

7 6 5 4 3 2 1

Number of bits per pixel

PS

NR

(d

B)

Excellent-good

Poor–very bad

Good-Moderate-Poor

Visually:

22

PdW / 2017

Fac. EE SPS-VCA



Mod 01 Part 2:

Obvious Ways to Compress Signals

23

PdW / 2017

Fac. EE SPS-VCA



PCM: Simple Compression

Two obvious ways to compress signals

1. Reduce the sampling rate (subsampling)

– Fewer audio samples per second

– Fewer pixels per picture line and fewer lines (images)

– Fewer pictures per second (video)

2. Reduce the No. of amplitude levels per signal sample:

– Pulse coded modulation or PCM

– Obviously we will develop far more advanced techniques in

this course

24

PdW / 2017

Fac. EE SPS-VCA



Reduce the #Amplitudes – (1)

24 bits (16777200 different colors)

7

25

PdW / 2017

Fac. EE SPS-VCA



8 bits (256 different colors)Compression factor 3

Reduce the #Amplitudes – (2) 26

PdW / 2017

Fac. EE SPS-VCA





27

PdW / 2017

Fac. EE SPS-VCA





PdW / 2017

Fac. EE SPS-VCA



1 bit (2 different colors)Compression factor 24


8

29

PdW / 2017

Fac. EE SPS-VCA



8 bits (256 different gray values)Compression factor 3


PdW / 2017

Fac. EE SPS-VCA





31

PdW / 2017

Fac. EE SPS-VCA





PdW / 2017

Fac. EE SPS-VCA





9

33

PdW / 2017

Fac. EE SPS-VCA



JPEGCompression factor 15

But real coding is always better…! 34

PdW / 2017

Fac. EE SPS-VCA



Mod 01 Part 3:

Roadmap Compression

35

PdW / 2017

Fac. EE SPS-VCA



Questions to be Answered

Three questions will play a central role:

1. Why can signals be compressed?

2. How much can signals be compressed?

3. Which signal processing/information theory algorithms

are most efficient in reaching the maximum

compression?

(For lossless and lossy compression)

36

PdW / 2017

Fac. EE SPS-VCA



Questions to be Answered

Three questions will play a central role:

1. Why can signals be compressed?

2. How much can signals be compressed?

3. Which signal processing/information theory algorithms

are most efficient in reaching the maximum

compression?

(For lossless and lossy compression)

10

37

PdW / 2017

Fac. EE SPS-VCA



Why can Signals be Compressed? – (1)

Because signal amplitudes are statistically redundant

Example:

– Signal has integer amplitudes in range [0,7]

– What is the required number of bits per signal sample (the

bit rate) to represent the signal amplitudes?

38

PdW / 2017

Fac. EE SPS-VCA





Example:– Signal has integer amplitudes in range [0,7]– 3 bit per signal sample (3 bps or bpp)

Signal value Codeword0 0001 0012 0103 0114 1005 1016 1107 111

39

PdW / 2017

Fac. EE SPS-VCA





Example:

– Signal has integer amplitudes in range [0,7]

– Signal amplitudes have following probabilities of occurrence

0

0,1

0,2

0,3

0,4

0,5

0,6

0 1 2 3 4 5 6 7

40

PdW / 2017

Fac. EE SPS-VCA





Example:– Signal has integer amplitudes in range [0,7]– 3 bit per signal sample (3 bps or bpp)

Signal value Probability Codeword0 0.125 0001 0 0012 0.5 0103 0 0114 0.125 1005 0.125 1016 0 1107 0.125 111

11

41

PdW / 2017

Fac. EE SPS-VCA




Because signal amplitudes are statistically redundantEntropy Coding or Variable Length Coding

Example:– Signal has integer amplitudes in range [0,7]– 2 bits per signal sample

Signal value Probability Codeword0 0.125 1001 02 0.5 03 04 0.125 1015 0.125 1106 07 0.125 111

42

PdW / 2017

Fac. EE SPS-VCA





Question 1:

What is the shortest average codeword length that

one can achieve for a given signal (or “source”)?(Shannon Information Theory)

Question 2:

How did you design/derive those code words?(Construction Recipes)

43

PdW / 2017

Fac. EE SPS-VCA




Because infinite accuracy of signal amplitudes is

(perceptually) irrelevant

24 bit (16777200 different colors)

8 bit (256 different colors)

Compression factor 3

The difference between these two pictures is perceptually irrelevant

because our visual system is insensitive for such small differences.

But the files differ a factor of 3 in size!

44

PdW / 2017

Fac. EE SPS-VCA






Represent amplitudes

coarser (“PCM”)

)(ix )(iy

-

Numerical difference: SNRPerceptual difference

12

45

PdW / 2017

Fac. EE SPS-VCA






RZix or )( ∈

)(iy

Typically implemented by a Quantizer (intelligent “rounding”)

Q

)(ix )(iy

46

PdW / 2017

Fac. EE SPS-VCA



Example of 8-Level Quantizer

Q)(ix )(iy

)(ix

)(iy

0 255

255

47

PdW / 2017

Fac. EE SPS-VCA






Usually in combination with variable length encoding

)(ix

)(iy

VLCQ)(ix )(iy 010010

0

110

111010

48

PdW / 2017

Fac. EE SPS-VCA






By choosing the number of representation levels, we

trade off rate versus distortion


Distortion =

Difference y(i)-x(i)

Number of

representation levels

~ average codeword

length L

13

49

PdW / 2017

Fac. EE SPS-VCA




Because infinite accuracy of signal amplitudes is (perceptually) irrelevant

Question 1:What is the best possible trade-off between required bit rate and resulting distortion?

(Rate-Distortion Theory)

Question 2:How do we implement a system that gives us that best possible trade-off?

(Scalar and Vector Quantization Theory)

50

PdW / 2017

Fac. EE SPS-VCA




Because signal amplitudes are mutually dependent

0 2000 4000 6000 8000 1 104

50

100

150

200

250

108000 j

Any signal that carries “information” contains a certain structure or stochastic dependencies

51

PdW / 2017

Fac. EE SPS-VCA



Why can Signals be Compressed? – (14)Because signal amplitudes are mutually dependent

0 500 1000 1500 2000 2500 3000 3500 4000 450050

100

150

200

250

“S”

0 1000 2000 3000 4000 5000100

150

200

“O”

52

PdW / 2017

Fac. EE SPS-VCA





one image line

14

53

PdW / 2017

Fac. EE SPS-VCA




Example of signal with no dependencies between

successive amplitudes (Gaussian uncorrelated noise)

Indeed “noise” compresses badly

54

PdW / 2017

Fac. EE SPS-VCA





Exploit the stochastic dependencies (~ correlation):

– Predicting the next value of the basis of the current one

– Calculate the amplitude of prediction difference

– This difference has (on the average) a much smaller amplitude

range (~smaller variance). Quantize prediction difference in few

representation levels

),1(),(ˆ jixjix −=

),1(),(),(ˆ),(),( jixjixjixjixjix −−=−=∆

55

PdW / 2017

Fac. EE SPS-VCA




56

PdW / 2017

Fac. EE SPS-VCA





Zero!

Positive value

Negative value


T

15

57

PdW / 2017

Fac. EE SPS-VCA




Dependency is commonly expressed by the autocorrelation function:

=

−=

−=

N

n

X

X

knXnXN

kR

knXnXEkR

0

)()(1

)(

)]()([)(

k

)(kRX

58

PdW / 2017

Fac. EE SPS-VCA





For images: 2-D autocorrelation function

)],(),([),( lmknXmnXElkRX −−=

k

l

),( lkRX

59

PdW / 2017

Fac. EE SPS-VCA





Question 1:

What is the best possible exploitation of the correlation

(dependencies) in natural signals? (Rate-Distortion Theory)

Question 2:

How do we implement a system that exploits the

correlation in natural signals? (Compression algorithms: - DPCM

- Subband/wavelet- Transform/DCT- Motion compensation)

60

PdW / 2017

Fac. EE SPS-VCA



Summarizing …

Sampling Fine Quantization

Transform Quantizer VLC encoding

A/D CONVERTER

COMPRESSION/SOURCE CODING

CHANNEL CODING

Error protect.Encipher

PCM encoded or “raw” signal

(“wav”, “bmp”, …)

Compressed

bit stream

(mp3, jpg, …)

Channel bit stream

Analog

capturing

device

(camera,

microphone)

Multimedia Video Coding & Architectures 5LSE0), …vca.ele.tue.nl/sites/default/files/5LSE0 Mod 01...

Documents

Transcript of Multimedia Video Coding & Architectures 5LSE0), …vca.ele.tue.nl/sites/default/files/5LSE0 Mod 01...