IMAGE QUALITY ASSESSMENT

H. R. Sheikh, A. C. Bovik, “Image Information and Visual Quality,” IEEE Trans. Image

Process., vol. 15, no. 2, pp. 430-444, Feb. 2006Lab for Image and Video Engi., Dept. of ECE

Univ. of Texas at Austin

Outline Introduction of Image Quality

Assessment Visual Information Fidelity Experiments and Results Conclusion

Quality Assessment (QA) For testing, optimizing, bench-marking,

and monitoring applications.

Quality?

Three Broad QA Categories Full-Reference (FR) QA Methods Non-Reference (NR) QA Methods Reduced-Reference (RR) QA Methods

Reference Image

Distorted ImageFR QA Quality

PSNR Simple but not close to human visual

quality

Contrast enhancement

Blurred

JPEG compressed

VIF = 1.10VIF = 0.07

VIF = 0.10

Prior Arts Image Quality Assessment based on

Error Sensitivity

CSF: Contrast Sensitivity FunctionChannel Decomposition: DCT or Wavelet TransformError Normalization: Convert the Error into Units of Just Noticeable Difference (JND)Error Pooling:

1

,,

l kklkl eeE

Problems of Error-Sensitivity Approaches The Quality Definition Problem The Suprathreshold Problem The Natural Image Complexity Problem The Decorrelation Problem The Cognitive Interaction Problem

Visual Information Fidelity

Natural Image Source

Channel(Distortion) HVS

HVS

C DF

E

ReferenceImage

TestImage

Human Visual

System

Reference Image Information

Human Visual

System

Test Image Information

Definition of VIF

subbandsj

jNjNjNsubbandsj

jNjNjN

sECI

sFCIVIF

,,,

,,,

;

;

Natural Image Source

Channel(Distortion) HVS

HVS

C D F

E

Source Model The natural images are modeled in the

wavelet domain using Gaussian scale mixtures (GSMs).

tscorfficien bubband toingcorrespond vectorsldimensiona- are and

: , : and

indices spatial ofset thedenodes where)C covariance andmean -(zero

:

U2

MMUC

IiUUIiSS

IS

IiUSUSC

ii

ii

i

ii

The subband coefficients are partitioned into nonoverlapping blocks of M coefficients each

Gaussian scale mixture (GSM) Wavelet coefficient => non-Gaussian

The variance is proportional to the squared magnitudes of coefficients at spatial positions.

UzX

V. Strela, J. Portilla, and E. Simoncelli, “Image denoising using a local Gaussian scale mixture model in the wavelet domain,” Proc. SPIE, vol. 4119, pp. 363–371, 2000.

Implementation Issues Assumption about the source model:

N

i

TiiU

iUTi

i

CCN

C

MCCCs

1

12

1ˆ

ˆ

V. Strela, J. Portilla, and E. Simoncelli, “Image denoising using a localGaussian scale mixture model in the wavelet domain,” Proc. SPIE, vol.4119, pp. 363–371, 2000.

Distortion Model

IC

IiVV

IigGIiVCgVGCD

vV

i

i

iii

2 ance with variRF noise

Gaussian mean -zreo additive stationary a is :

fieldgain scalar icdeterminst a is : where:

Distorted Images

Distorted Images Synthesized versions

Distorted Images

Human Visual System (HVS)Model

Natural Image Source

Channel(Distortion) HVS

HVS

C D F

E

ICC

C

IiNNIiNN

NDFNCE

nNN

i

ii

2'

aslity dimensiona same the

ithGaussian w temultivaria eduncorrelatmean -zreo are:'' and : where

image)(test 'image) (reference

Visual Information Fidelity Criterion (IFC)

Natural Image Source

Channel(Distortion) HVS

HVS

C D F

E

C E

Mutual InformationI(C;E)

Mutual Information Assuming that G, and are known2

v 2n

N

i n

nUi

N

iiiiii

N

iiji

Njiji

N

j

N

i

NNN

I

ICs

sNhsNCh

sECI

sECECIsECI

12

22

2

1

1

11

1 1

log21

;

,,;;

C E

Mutual InformationI(C;E)

Mutual Information

N

i

M

k nv

kiiNNN

N

i

M

k n

ki

N

i n

nUiNNN

k

TUU

sgsFCI

s

I

ICssECI

QQCC

1 122

22

2

1 12

2

2

12

22

2

1log21;

1log21

log21;

seigenvalue ofmatrix diagonal a is symmetric, is Since

Implementation Issues Assumption about the source model:

N

i

TiiU

iUTi

i

CCN

C

MCCCs

1

12

1ˆ

ˆ

V. Strela, J. Portilla, and E. Simoncelli, “Image denoising using a localGaussian scale mixture model in the wavelet domain,” Proc. SPIE, vol.4119, pp. 363–371, 2000.

Implementation Issues Assumption about the distortion model:

use B x B window centered at coefficient i to estimate and at i

Assumption about the HVS model:Hand-optimize the value of

),(ˆ),(ˆ

),(),(ˆ2,

1

DCCovgDDCov

CCCovDCCovg

iiv

i

ig 2ˆv

2ˆn

(by linear regression)

Definition of VIF

subbandsj

jNjNjNsubbandsj

jNjNjN

sECI

sFCIVIF

,,,

,,,

;

;

Natural Image Source

Channel(Distortion) HVS

HVS

C D F

E

Experiments Twenty-nine high-resolution(768x512) 24-bits/pixel

RGB color images Five distortion types: JPEG 2000, JPEG, white noise

in RGB components, Gaussian blur, and transmission errors

20-25 human observers Perception of quality: “Bad,” “Poor,” “Fair,” “Good,”

and “Excellent” Scale to 1-100 range and obtain the difference mean

opinion score (DMOS) for each distorted image Data base:

http://live.ece.utexas.edu/research/quality/

Scatter Plots for Four Objective Quality Criteria

(x) JPEG2000, (+) JPEG,(o) white noise in RGB space, (box) Gaussian blur, and (diamond) transmission Errors in JPEG2000 stream over fast-fading Rayleigh channel

Scatter Plots for the Quality Prediction

Validation Scores

THE VALIDATION CRITERIA ARE: CORRELATION COEFFICIENT (CC), MEAN ABSOLUTE ERROR (MAE), ROOT MEAN-SQUARED ERROR (RMS), OUTLIER RATIO(OR), AND SPEARMAN RANK-ORDER CORRELATION COEFFICIENT (SROCC)

Two version of VIF:VIF using the finest resolution at all orientations andUsing the horizontal and vertical orientations only

Cross-Distortion Performance

Cross-Distortion Performance

(dark solid) JPEG2000, (dashed) JPEG, (dotted) white noise, (dash-dot) Gaussian blur, and (light solid) transmission errors in JPEG2000 stream over fast-fading Rayleigh channel

Dependence on the HVS Parameter

Dependence of VIF performance on the parameter.(solid) VIF, (dashed) PSNR,(dash-dot) Sarnoff JNDMetrix 8.0, and (dotted) MSSIM.

2ˆn

Conclusion A VIF criterion for full-reference image QA

is presented. The VIF was demonstrated to be better

than a state-of-the-art HVS-based method, the Sarnoff’s JND-Metrix, as well as a state-of-the-art structural fidelity criterion, the SSIM index

The VIF provides the ability to predict the enhanced image quality by contrast enhancement operation.

Reference1. H. R. Sheikh, A. C. Bovik, “Image Information and Visual

Quality,” IEEE Trans. Image Process., vol. 15, no. 2, pp. 430-444, Feb. 2006.

2. H. R. Sheikh, A. C. Bovik, and G. de Veciana, “An information fidelity criterion for image quality assessment using natural scene statistics,” IEEE Trans. Image Process., vol. 14, no. 12, pp. 2117–2128, Dec. 2005.

3. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error measurement to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.

4. V. Strela, J. Portilla, and E. Simoncelli, “Image denoising using a local Gaussian scale mixture model in the wavelet domain,” Proc. SPIE, vol. 4119, pp. 363–371, 2000.