Privacy Protection of Multimedia Information - Vis...

1

Privacy Protection of Multimedia

Information

Sen-ching Samson Cheung 張先正

張先正

張先正

張先正

Center for Visualization & Virtual Environments

Department of Electrical & Computer Engineering

University of Kentucky

http://www.vis.uky.edu/mialab

2

�Smart video

surveillance

�Biometric signals

�Mobile-m

edia

processing

�RFID tracking

Multimedia privacy concerns

3

Challenges from Multimedia

�What to protect?

�Identify selective semantic objects for

protection

�How to protect it?

�Reliable protection without sacrificing

perceptual utility, processing speed and

bandwidth

�How to control it?

�Flexible control and secure authentication of

privacy data

4

Privacy Protected Video Surveillance

Ob

ject

S

egm

enta

tio

nan

d T

rack

ing

Ob

fusc

atio

n

Pri

vacy

Dat

a P

rese

rvat

ion

Su

rvei

llan

ceV

ideo

Dat

abas

e

Su

bje

ct

Iden

tifi

cati

on

Mo

du

le

Sec

ure

Cam

era

Sys

tem

Pri

vacy

Dat

a M

anag

emen

t S

yste

m

5

Summary of contributions

Tasks

Our contributions

CameraPlacement

Optimal visual sensor design for localizing

subjects

Object Segmentation

and Tracking

Visible-light and therm

al camera fusion for

better background modeling

Obfuscation

Object-basedVideo In-painting

Privacy Data

Preservation

Rate-distortion optimal data hiding

Anonymous Subject

Identification

Homomorphicencryption based biometric

access control

Applications

VIBE: Video Interface

Behavioral Evaluation

6

Visual Tagging for Subject Identification

Visual Tagging: use of visual features to locate objects

�Pure visual, no special hardware

�Do not need subject cooperation

�Self and other occlusions

7

Multi-camera Localization

Use epipolar lines

from different

cameras to localize

occluded objects

8

Camera Network Planning

�Multi-camera localization needs each

object to be visible by at least two

cameras

�For a surveillance environment:

�How many cameras?

�Where should we put those cameras?

�What is the expected performance?

�Need a proper model to capture all known

and unknown parameters

9

Optimal Sensor Planning

�Art Gallery or Illumination Problems

�K-guarding problem -at most n-1 cameras for any

planar n-sided polygon with no holes [Belleville et al. 94]

�Optimal solution: NP-complete problem !?

�Continuous-domain approach

�Hill-clim

bing [Bodor et al 07], simulated annealing [Mittal,

Davis 08], evolutionary approach [Dunn, Olague, Lutton 06]

�Restrictive modeling, computational intensive, local

minima

�Proposed Discrete-domain approach

�Integer-programming

�Resource constraint problem

10

Statistical Visibility Model

Random Parameters, P: model F(P)

�position of a tag

�orientation of a tag

�mutual occlusion

Fixed Parameters, K: user-specified

�room topology

�cameras’ intrinsic parameters

�dimensions (lengths) of a tag

Design Parameters, C: controllable

�number of cameras

�position of each camera

�orientation of each camera

General framework:

�Computable Visibility

function I

(P|K

,C) for a tag

at P

�Performance of

C:

�Optimization:

∫)

()

,|

(P

dFC

KP

I

∫)

()

,|

(m

axP

dFC

KP

IC

11

Visibility Function I(P|K,C)

�Environment K

�Camera C=(x

C,yC,φ,ρ)

�Tag P =(x

P,yP,zP,θ,λ)

�I(P|K,C) = 1 if the tag image is large enough

θ

(xP,yP,zP)

(xC,yC)

(φ,ρ)

λ

12

Variable Definitions

�Discretize both tag and camera space into

lattice points

�Output Variable: camera placement

�Measurement Variable: Tag visibility

xTag at Pivisible at 2 or more cameras

Otherwise

Camera present at Ci

Otherwise

13

A camera cannot see two direction at the same time

Binary Integer Program

�Cost function

�Constraints:

1.

2.

3.

ibm

axExpected volume visible by 2 or more cameras

At each tag grid P

i, define xibased on the visibility function

Camera constraint

Linear cost & constraints

-Solved using lpsolve, c-plex

-NP-hard (greedy search)

∑=P

N jj

jx1ρ

()

∑=

<+

−C

N ij

ci

ji

xN

CK

PI

b1

11

),

|(

∑=

≥−

CN i

ji

ji

xC

KP

Ib

10

2)

,|

(

∑=

≤C

N ii

mb

1

1y)

(x,

at

All

≤∑ ib

ib

14

Simulated Perform

ance

Comparison with other schemes:

Use of traffic modeling:

15

Experimental Results

Optimal

Visibility = 0.53

Uniform

Visibility = 0.38

Zhao, J., S.-C. Cheung and T. Nguyen. 2008. Optimal Camera

Network Configurations for Visual Tagging. In IEEE Journal

on Selected Topics in Signal Processing, Volume 2,

Number 4, August 2008, pp. 464-479.

16


Tasks

Our contributions

CameraPlacement


subjects

Object Segmentation

and Tracking




Obfuscation


Privacy Data

Preservation


Anonymous Subject

Identification


access control

Applications



Background Subtraction

�Shadows and highlights

�Illumination changes

�Non-static background

�Color similarity

Fusion of thermal and visible-light

�Therm

al Im

aging (PV320 digital therm

al camera)

�Uncooled focal plane array of ferroelectric

sensors (-20 –500oC)

�Challenges

�Registration

�Data Fusion

�Existing Approaches:

�Optical Fusion [Volfson06], [Wu08]

�Im

age W

arping [Davis05], [Kumar06][Han07]

Cameras Registration and Blob alignment

1.Calibration to obtain fundamental and rectification

matrices

2.Estimate a homography for each foreground blob based

on disparity -assume each person is of constant depth

(a) Infrared (b) visible light

Blob Extraction and Data Fusion

�Individual tracker

tracks object in each

camera view

�Combined tracker

estimates

homographies

�Second tier adjusts

parameters and

updates the states

using fused data

Zhao, J. and S.-C. Cheung. 2009. Human Segmentation by

Fusing Visible-light and Therm

al Im

aginary. Submitted to the

Ninth IEEE International Workshop on Visual

Surveillance (VS2009).

Results versus image warping

Fusion Results

Video Results

23

24


Tasks

Our contributions

CameraPlacement


subjects

Object Segmentation

and Tracking




Obfuscation


Privacy Data

Preservation


Anonymous Subject

Identification


access control

Applications



25

Video Obfuscation

Original

Pixelation/

Blurring

Black

Out

In-painted

26

Challenges of Video Inpainting

27

Dynamic Object In-painting

�Basic idea: Using object template extracted form

other time instant to complete a conceptually

consistent sequence.

�Steps:

1. Similarity based on optimal alignment

2. Motion continuity

3. Positioning of templates

?

28

Motion Continuity

??

29

Object-based Video In-painting

�Better motion in-painting by better registration and task

separation

�Capable to in-paint partially and completely occluded

objects

�Im

proved computational performance (Matlab)

Num

ber

of fr

ames

with

com

plet

e oc

clus

ion

Num

ber

of fr

ames

with

par

tial o

cclu

sion

30

Public-domain Sequences

31

Multi-people sequence

32

Complex Sequences

33


Tasks

Our contributions

CameraPlacement


subjects

Object Segmentation

and Tracking




Obfuscation


Privacy Data

Preservation


Anonymous Subject

Identification


access control

Applications



34

Keeping sensitive inform

ation

Medium

Method

Pro

Con

Separate

File

Encryption +

Cryptographic

signature

�Standard Technology

�Storage efficiency

�Pervious to attacker

�Difficult to distribute with

the modified video

�Separate authentication for

modified video

Meta-data

Encryption +

Cryptographic

signature

�Standard Technology

�Storage efficiency

�Less pervious to attacker

�Depend on format

Data

hiding

Encrypted

watermark

�Im

pervious to attacker

�Inseparable from data

�Joint authentication

�May need more storage

�May affect visual quality

35

Privacy Data Preservation

36

Data Hiding

�Data hiding/Stenography/W

atermarking

�Active research in the past fifteen years

�Typical applications include authentication, copy

detection, monitoring

�Challenges in our application:

�Picture-in-picture: large embedding capacity

�Compatibility with existing compression scheme

�Minimal visual distortion

37

Optimal Data Hiding

Psy

cho-

visu

alM

odel

ing

Blo

ck-b

ased

Rat

e-D

isto

rtio

nC

alcu

latio

n

Dis

cret

eO

ptim

izat

ion

Sol

ve c

onst

rain

ed o

ptim

izat

ion

020

4060

8010

012

014

016

018

020

00

0.2

0.4

0.6

0.81

1.2

1.4

Rat

e

Distortion

020

4060

8010

012

014

016

018

020

00

0.2

0.4

0.6

0.81

1.2

1.4

Rat

e

Distortion

Com

bine

d ra

te-

dist

ortio

n co

st C

(x) #

embe

dded

bits

38

R-D framework

�Target cost function:

�Ri= Increase in Bandwidth of Block i

�Di= Perceptual Distortion in Block i

�δ= Relative Weight

�Greedy embedding of P data bits in Block i:

�Lagrangian optimization: determine the optimal Piand λto

embed the target number of data bits:

39

Proposed Data Hiding

Mot

ion

Com

pens

atio

nD

CT

Ent

ropy

Cod

ing

•D

CT

Dom

ain

•F

requ

ency

, co

ntra

st a

nd

lum

inan

ce m

aski

ng [W

atso

n]

H.2

63H

.263

Enc

rypt

ed f

oreg

roun

dvi

deo

bit-

stre

am

DC

TP

erce

ptua

lM

ask

Par

ityE

mbe

ddin

g

R-D

O

ptim

izat

ion

Pos

ition

s of

th

e “o

ptim

al’

DC

T c

oeff

for

embe

ddin

g

DC

T(i,

j) =

wat

erm

ark_

bit+

2*ro

und(

DC

T(i,

j)/2)

Priv

acy

prot

ecte

dvi

deo

Last

dec

oded

fram

e

J. Paruchuri & S.-C. Cheung “Rate-

Distortion Optimized Data Hiding

for Privacy Protection” submitted to

ISCAS 2008

Block embedding strategies compared

40

Bit Allocation strategies compared

41

Overall results

42

43

Examples 1/2

119kbps

No data

Distortion

637 kbps

81 kbps data

Rate &

Distortion

562 kbps

81 kbps data

Rate only

370 kbps

81 kbps data

44

Examples 2/2

406.3kbps

No data

Distortion

743 kbps

81 kbps data

Rate &

Distortion

678 kbps

81 kbps data

Rate only

610 kbps

81 kbps data

45


Tasks

Our contributions

CameraPlacement


subjects

Object Segmentation

and Tracking




Obfuscation


Privacy Data

Preservation


Anonymous Subject

Identification


access control

Applications



46

�What you have:

�RFID [Wickramasuriya04], Hard hat

[Schiff07], Colored Marker [Zhou09]

�Tags are vulnerable or easy to

forge

Subject Identification

�Who you are:

�Biometric like fingerprint, iris, face

and gait

�Worse privacy as system can

associate video to identity

How do you perform

biometric

verification anonymously?

How do you perform

biometric

verification anonymously?

47

�Anonymous Biometric Access Control [Luo09] [Yi09]

�A secure multi-party protocol that guarantees:

�Bob knows if ∃y∈DBwhere y matches q but does

not know which y.

�Alice knows the result but knows nothing about DB

ABAC

ABAC

Alice and her biometric q

Bob & his DB

ABAC Protocol

48

Query

Encryption

Distance

Computation

Distance Bit

Extraction

Threshold

Comparison

Multiplication

(Accumulation)

Enc p

k(y+r)

with rand r

Decryption

Hash(y+r)

Hash(r)

Equal?

Enc p

k(y) where y=0 (success)

or function of matching

Alice can’t

cheat!

Using ABAC in Surveillance

49

50

Problem: ABAC is not scalable

�Use Paillier homomorphic encryption and

interactive protocols to implement

similarity search [Luo09], [Yi09]

�Match one 9600-bit iris code [Masek03]:

�Initialization (one-time)

290 ms

�Hamming distance

98 ms

�Bit extraction & compare

4120 ms

�For a DB with 10,000 iris codes:

�~11.5 hours and 120MB data exchanged

�Tradeoff complexity with privacy by providing

extra information to Bob (server):

�Idea behind kAQ:

�Alice tells Bob the cell C to which her query belongs

�Bob runs ABAC within C only ⇒

achieves k-anonymity

�Design of the cell or quantization structure:

1.

Must minimize leakage of Alice’s privacy

2.

Must ensure the correctness of similarity search

k-Anonymous Quantization

51

Bob knows a subset (cell) C that contains at least k

xi∈DBsuch that all q with d(q,xi)<εare in C.

Minimize privacy leakage

�Which structure leaks less information?

�Hypothesis: Quantization 1 as the entries within

are “maximally dissimilar”

52

Biometric signals in gallery

Four cells

Quantization 1

Quantization 2

Probe q

�Demonstrated in fingerprints [Jain02] and palmprints

[Kong06]

X := distance between iris-codes from twins

Y := distance between iris-codes from non-twins

Null hypothesis

H0:

µ X= µ

Y

Alternative hypothesis

H1:

µ X< µ

Y

�Data: 1118 iris from 100 pairs of twins [CASIA05]

�Distribution-free W

ilcoxonRank-Sum Test produces

a one-side P-value of 6.17 x 10-75

⇒Strongly favors the adoption of the alternative

hypothesis

Kinship ⇒

Similar biometrics ?

53

Given a kAQ Γ, its privacy-protecting capability

can be measured by this utility function:

To design a good kAQ:

�Maximizing utility is not enough!

�To ensure the correctness of kAQ, we need to

ensure that if y matches x in C, y itself must also

be in C ⇒

Neighborhood Structure

Utility of kAQ

54

2

,)

,(

min∑

∩∈

Γ∈

CD

Bx

xj

ij

iC

xx

d

�Typical choice of Neighborhoods:

�Bounding Balls or Boxes

�Good Neighborhood:

�Should capture the variability of all biometrics

�Should minimize the overlap of neighborhoods

between different individuals

Neighborhoods

55

Mary’s Mom’s

Mary’s

C1

C2

�Data-driven design –based on capturing

many biometrics from each person

�1-2 patterns for testing, rest for training –

correct recognition depends on generalization

�Neighborhood candidates:

Different Neighborhoods

56

εε

ε

Actual

Bounding

Box

Actual

bounding

Ball

Maximum

radius

Average+1stdev

radius

Greedy kAQ algorithm

1.

Embed hamming space into low-dimensional

Euclidean space

2.

Uniform quantization to form bins

3.

# of cells, N = floor(|DB|/k)

4.

Randomly assign a neighborhood to each cell

5.

Select the neighborhood-cell pair that maximize the

gain in utility

6.

Repeat step 5 until all neighborhoods are exhausted

7.

If a cell has less than k neighborhoods, N := N-1 and

back to step 4.

Note: overlapping neighborhoods in different cells will

cause multiple cells to be used in later stage.

57

Results

For a database 10,000 entries

�full encrypted processing takes ~ 12 hours to run.

�k-anonymous quantization with k=50 takes 650 seconds

58

privacy

S. Yee, Y. Lou, J. Zhao and S.-C. Cheung. 2009. Anonymous Biometric Access Control. Accepted to

EURASIP Journal on Information Security

59

The Experiment Result of Different

Neighborhood Structures

ε-ball with a statistical radius is the best choice.

60


Tasks

Our contributions

CameraPlacement


subjects

Object Segmentation

and Tracking




Obfuscation


Privacy Data

Preservation


Anonymous Subject

Identification


access control

Applications



Tantrums: disruptive behaviors in children

�80% of children ages 1 to 4, with 20% of 2-year olds and

10% of 4-year-olds have daily tantrums.

�Prolonged, frequent, and age-inappropriate tantrums

�may indicate underlying mental illness

�may predict later antisocial behavior

�are at a higher risk of abuse

�affect proper functioning of families and schools

�demoralize caregivers as a reflection of poor parenting

61

Existing Approaches

�Need careful assessments on

�Events that trigger and ameliorate tantrums

�Exact tantrum behaviors

�Consequence

�Clinicians and behavior therapists rely on

�Real-time observation during clinic’s visits

�Caregivers’ account of Events

�Limitations

�Many children do not engage in disruptive behaviors in the

clinician’s office due to absence of social triggers or

expectation of acceptable behaviors in the clinical setting.

�Caregivers’ account is often biased, incomplete and

selective

62

VIBE: Video Interface Behavioral Evaluation

�What is it?

�A networked video monitoring system to

catalogue children’s disruptive behavior

�Advantages

�Unobtrusive recording of behaviors and social

interaction in the child’s natural environments

(school, home or car)

�Privacy enhancement technologies to filter

sensitive contents

�Event Recognition technologies to allow clinicians

in rapidly identify relevant episodes

63

Current Status

�Two-year study in Lexington and Beijing to

�Aim 1: To examine attitudes toward VIBE among caregivers of

children with disruptive behaviors in different cultures

�Aim 2: To identify factors associated with negative attitudes

toward VIBE and determ

ine whether these constitute a barrier

�Aim 3: To evaluate and compare the clinician-reported value

about children’s disruptive behaviors collected using retrospective

written accounts, caregiver-recorded video and VIBE.

�Partners

�Dr. Neelkamal Soares, Pediatrics, University of Kentucky

�Dr. Brea Perry, Sociology, University of Kentucky

�Dr. Xiaoyi Yu, Peking University

64

Privacy Protection of Multimedia Information - Vis...

Documents

Transcript of Privacy Protection of Multimedia Information - Vis...