Download - Mm presentation bkk

MODELING THE QOE OF

RATE CHANGES IN

SKYPE/SILK VOIP CALLS

CHIEN-NAN CHEN CING-YU CHU

SU-LING YEH

HAO-HUA CHU

POLLY HUANG

UNIVERSITY OF

ILLINOIS, URBANA-

CHAMPAIGN

NATIONAL TAIWAN

UNIVERSITY

1

OUTLINE

• Motivation

• Preliminary Experiment

• Proposed Model

• Large-Scale Experiment

• Evaluation

• Conclusion

2

VOICE OVER IP

Internet

Bandwidth FluctuationPacket LossDelay Jitter

MotivationPre. Exp.Proposed ModelLarge Exp.EvaluationConclusion

3

RATE ADAPTATION

• Available bandwidth

Ramping up the sending rate

• Available bandwidth

Tuning down the sending rate

Is the quality improved proportionally?

Rate change Disturbing users?


4

GOAL

• Investigating the relationship of

Sending rate vs. Perceived quality

• To explore the influence of

Rate change magnitude/frequency

• Methodology

• Synthesized VoIP calls

• User study experiments


5

CONTRIBUTION

• Sending bitrate vs. user perception

Logarithmic Relationship

• Frequency of rate change

Logarithmic Relationship

• Magnitude of rate change

Complicated, but Interesting

• Closed-form models to predict user

perception under bandwidth fluctuation


6

PRELIMINARY EXPERIMENT

• To confirm the influence of

• sending bitrate

• rate change magnitude

• rate change frequency

• 5-level MOS (Mean Opinion Score)

• 14 participants


7

AUDIO TRACK PRODUCTION

• Skype/SILK audio codec

• 30s audio track

• sentences without contextual connection

• Fixed-rate tracks

• Variable-rate tracks Bitrate (kbps)

5.6 9.5 13.3 17.2 21.1 25.0 28.9 32.8 36.6 40.6


8

RESULT

FIXED-RATE

• MOS vs. sending bitrate

User Variation

Logarithmic

Trend


9

RESULT

VARIABLE-RATE

• MOS - ΔT plot

Rate change

matters!


10

EFFECT OF RATE CHANGE

FREQUENCY

• When ΔT varies…

Logarithmic

Trend


11


MAGNITUDE

• When sharing the same average bitrate…

Magnitude

MOS


12


MAGNITUDE

• However, with the same magnitude…

Higher (hr + lr)

Lower (hr + lr)


13

SHORT SUMMARY

• Fixed-rate

• MOS – bitrate logarithmic

• Variable-rate

• MOS – ΔT logarithmic

• MOS – (hr, lr)

• hr - lr up MOS down

• hr + lr up MOS up


14

PROPOSED MODELS

• Fixed-rate model

• Variable-rate model

Massive Data Numerical Fitting


15

LARGE-SCALE EXPERIMENT

• Same methodology

• 127 participants

• Each track is scored by 30 participants

• Rate selection

Bitrate (kbps)40.627.719.414.1

5.6 6.1 7.1 8.5

10.7

r1r2r3r4r5r9 r8 r7 r6


16

SIGNIFICANCE OF FACTORS

• ANOVA tests

• MOS – sending bitrate

Significant

• Interaction between ΔT and (hr, lr)

Significant

• MOS - ΔT

Test p-value Test p-value Test p-value

r1r2 .31 r6r7 .31 r7r8 .26

r3r4 .42 r6r8 .11 r7r9 .34

r4r5 .31 r6r9 .09 r8r9 .32


17

MODEL SPECIFICS

FIXED-RATE MODEL

• α=4.091, β=1.515, and γ=1.000

• with R-square = 0.96

Lower bound of user perception (?)

close to the lowest bitrate of SILK


18

MODEL SPECIFICS

VARIABLE-RATE MODEL

• Logarithmic regression on each (hr, lr)

pair

(r1, r2): p12 x ln(ΔT) + q12

(r1, r3): p13 x ln(ΔT) + q13

(r1, r4): p14 x ln(ΔT) + q14

(r1, r5): p15 x ln(ΔT) + q15

:

:

:

SCALE() SHIFT()


19

MODEL SPECIFICS

SCALE()

• Polynomial regression

• x = hr – lr , y = hr + lr


20

MODEL SPECIFICS

SHIFT()

• Independent to ΔT

• Basic idea

• ΔT approaches the track duration

• Fluctuation diminishes


21


EVALUATION

GOODNESS OF FIT

• Training data

• R-square = 0.86


22

EVALUATION

ACCURACY OF PREDICTION

• 2 dataset independent to training data

• Dataset I: Preliminary experiment

• Dataset II: Additional (New) experiment


23

PESQ

• Perceptual Evaluation of Speech Quality

• Limited spectrum

• Narrow-band: 8k Hz

• Wide-band: 16k Hz

(SILK: 8k, 12k, 16k and 24 k Hz)

• Requires both original and degraded

audio files

24


COMPARISON WITH

PESQ – FIXED RATE

25


model Proposed PESQ

R-square 0.9601 0.7841

model Proposed PESQ

Avg. Err. Ratio 3.68% 14.59%

COMPARISON WITH

PESQ – VARIABLE RATE

26


model Proposed PESQ

R-square 0.2512 -0.3491

model Proposed PESQ

Avg. Err. Ratio 8.03% 12.60%

COMPARISON ON

AMR-WB

• AMR-WB audio codec

• Older Codec

• Widely used in 3G network

• 9 difference coding bitrates

• User study experiment

• Same methodology

• 14 participants

27


COMPARISON ON

AMR-WB

28


model Proposed PESQ

R-square 0.7878 0.6289

model Proposed PESQ

Avg. Err. Ratio 2.18% 2.86%

Proposed

PESQ

Proposed

PESQ

CONCLUSION

• The logarithmic relationship (Weber-Fechner

Law) is observed in the MOS-bitrate relation-

ship of Skype/SILK

• Rate change frequency (W-F Law) and

magnitude (complicated) have significant

influence on perceived quality

• We have established both fixed- (SIGCOMM’12

W-MUST) and variable-rate models

• User-centric rate adaptation for VoIP

applications (coming next)


29

Q&A

30

MODEL SPECIFICS

SHIFT()

• Dominant quality

• Expected quality when fluctuation diminishes

• ΔT approaches the track lengthdominant

quality

ΔT (second) 31

MODEL SPECIFICS

SHIFT()

• Dominant quality: D()

hr = 14.1 kbps

D() = MOSh

32

MODEL SPECIFICS

SHIFT()

• hr > 14.1 kbps

• normalized y-axis

33

MODEL SPECIFICS

SHIFT()

• Linear to the MOS difference (hr > 14.1 kbps)

34