Instance-privacy Preserving Crowdsourcing (HCOMP2014)
-
Upload
kanojikajino -
Category
Data & Analytics
-
view
176 -
download
2
Transcript of Instance-privacy Preserving Crowdsourcing (HCOMP2014)
Instance-privacy Preserving Crowdsourcing
Hiroshi Kajino1, Yukino Baba2, Hisashi Kashima3"
1: The University of Tokyo, JSPS Research Fellow DC2"2: National Institute of Informatics, " JST ERATO, Kawarabayashi Large Graph Project 3: Kyoto University
Nov. 4, 2014 1 HCOMP-2014
15 min
Overview
■ Instance Privacy"□ Sensitive information in a task instance leaks"
■ Instance Clipping Protocol"□ Exploit the locality of (task, privacy)"
■ Proposed Evaluation Method"□ Define the task result quality & the amount of privacy invasion"
■ Experiment"□ Evaluate the performance on a task to detect faces in images"
Nov. 4, 2014 HCOMP-2014 2
A privacy issue in submitting tasks in crowdsourcing
Overview
■ Instance Privacy"□ Sensitive information in a task instance leaks"
■ Instance Clipping Protocol"□ Exploit the locality of (task, privacy)"
■ Proposed Evaluation Method"□ Define the task result quality & the amount of privacy invasion"
■ Experiment"□ Evaluate the performance on a task to detect faces in images"
Nov. 4, 2014 HCOMP-2014 3
A privacy issue in submitting tasks in crowdsourcing
Example of Privacy Invasion
■ Case Study: Task to hide faces in an image"
Nov. 4, 2014 HCOMP-2014 4
Sensitive information in a task instance leaks"
Instance Result Worker Instruction"
Process the instance following the instruction
Example of Privacy Invasion
■ Case Study: Task to hide faces in an image"
Nov. 4, 2014 HCOMP-2014 5
Task instance can be used for other purposes"
Instance Result Worker Instruction"
Sensitive information"of Mr. A
Linking between PIs (ex. face)"& his properties!
(place where he is, friendship, action)
Mr. A
Research Questions
■ Question 1: Design of a Privacy-Preserving protocol"□ General guideline to design a PP protocol"
■ Question 2: Performance measures for PP protocols"□ Quality of a result & privacy leakage:"
• PP protocol must involve some operation on instance"• Quality can be degraded by the operation"
■ Assumptions"□ Instance: Multi-way array (image, text, audio, video)"□ Result & sensitive info: Label"
Nov. 4, 2014 HCOMP-2014 6
Our concerns are a PP protocol and a performance metric for it
Overview
■ Instance Privacy"□ Sensitive information in a task instance leaks"
■ Instance Clipping Protocol"□ Exploit the locality of (task, privacy)"
■ Proposed Evaluation Method"□ Define the task result quality & the amount of privacy invasion"
■ Experiment"□ Evaluate the performance on a task to detect faces in images"
Nov. 4, 2014 HCOMP-2014 7
A privacy issue in submitting tasks in crowdsourcing
Case Study: Instance Clipping Protocol
■ Instance Clipping Protocol"□ Moving C × C pixels window by C/2 pixels □ Clip the instance to generate sub-instances"□ Submit a task with the sub-instances"
Nov. 4, 2014 HCOMP-2014 8
Illustrate a guideline by the IC protocol
C
C/2 Clipping window
sub-instances"
Key Insight: Locality of Task
■ Locality Property of Task"□ Local task: Local area has sufficient info for task execution"
ex) Face detection, name & license number transcription"
□ Global task: Global area is necessary for task execution"ex) abstract of a meeting (audio/text)" place, friendship, action detection (image)"
Nov. 4, 2014 HCOMP-2014 9
Task execution is classified by its dependency on instance
Apple tries to buy a fingerprint security company
Apple tries to buy a fingerprint security company
Key Insight: Locality of Privacy
■ Locality Property of Privacy"□ Local privacy: Local area has sufficient info for privacy invasion"
ex) Face, name, license number"
□ Global privacy: Global area is necessary for privacy invasion"ex) abstract of a meeting (audio/text)" place, friendship, action (image)"
Nov. 4, 2014 HCOMP-2014 10
Privacy invasion is classified by its dependency on instance
Apple tries to buy a fingerprint security company
Apple tries to buy a fingerprint security company
Basic Idea: Instance Transformation
■ Local-info-preserving transformation"□ Preserve local info, mash global info (e.g., Clipping)"□ Suitable for (local task, global privacy)"
Nov. 4, 2014 HCOMP-2014 11
Privacy preservation using opposite properties of task & privacy
Basic Idea: Instance Transformation
■ Local-info-preserving transformation"□ Preserve local info, mash global info (e.g., Clipping)"□ Suitable for (local task, global privacy)"
■ Global-info-preserving transformation"□ Preserve global info, mash local info (e.g., Blurring)"□ Suitable for (global task, local privacy)"
Nov. 4, 2014 HCOMP-2014 12
Privacy preservation using opposite properties of task & privacy
Overview
■ Instance Privacy"□ Sensitive information in a task instance leaks"
■ Instance Clipping Protocol"□ Exploit the locality of (task, privacy)"
■ Proposed Evaluation Method"□ Define the task result quality & the amount of privacy invasion"
■ Experiment"□ Evaluate the performance on a task to detect faces in images"
Nov. 4, 2014 HCOMP-2014 13
A privacy issue in submitting tasks in crowdsourcing
Evaluation: Motivation
■ Proposed Evaluation Method"□ Two metrics:!
• Quality of task result"• Privacy invasion"
□ Requirements:!• Privacy: Should penalize even if wrong info leaks"
– Suffer from a wrong rumor"– Checking answers with ground truths is not appropriate"
• Task: Have the same unit with the privacy measure"Nov. 4, 2014 HCOMP-2014 14
Trade-off between task result quality and privacy leakage
Task result quality ☺"Privacy invasion "
Task result quality #"Privacy invasion ☺
Trade-off
Evaluation: Proposed Metrics
Nov. 4, 2014 HCOMP-2014 15
Task execution process Privacy invasion process
Model
Worker samples task result of instance In from p(R | In) R: Result or sensitive info" I: Instance" p(R | I): Instruction"
Measures
EI[ KL(p(R | I) || p’(R | I)) ]
■ Difference from the original"■ Smaller, better
(Mutual info between R & I )" = EI[ KL(p’(R | I) || p’(R)) ]
■ Info leakage of R from I ■ Even wrong info should not leak from I ■ Smaller, better
w/o PP w/ PP
Design measures based on a worker model
0
1
face/non-face
Evaluation: Modeling
□ Instance: I □ Result: R □ Worker = Sampling from p(R | I)
Nov. 4, 2014 HCOMP-2014 16
Worker is modeled as a sampler
Result Worker
instance instruction"
0
1
行動1/行動2 =
Prob dist"
Face/non-face
Random var
Evaluation: Proposed Metrics
Nov. 4, 2014 HCOMP-2014 17
Task execution process Privacy invasion process
Model
Worker samples task result of instance In from p(R | In) R: Result or sensitive info" I: Instance" p(R | I): Instruction"
Measures
EI[ KL(p(R | I) || p’(R | I)) ]
■ Info loss caused by PP"■ Smaller, better
(Mutual info between R & I )" = EI[ KL(p’(R | I) || p’(R)) ]
■ Info leakage of R from I ■ Even wrong info should not leak from I ■ Smaller, better
w/o PP w/ PP
Design measures based on a worker model
0
1
face/non-face
Overview
■ Instance Privacy"□ Sensitive information in a task instance leaks"
■ Instance Clipping Protocol"□ Exploit the locality of (task, privacy)"
■ Proposed Evaluation Method"□ Define the task result quality & the amount of privacy invasion"
■ Experiment"□ Evaluate the performance on a task to detect faces in images"
Nov. 4, 2014 HCOMP-2014 18
A privacy issue in submitting tasks in crowdsourcing
Experiment: Setting
■ Stanford 40 Actions Dataset [Yao+, 11]"□ Task: Detect faces in images"□ Privacy: Infer the action s/he takes given a sub-instance"
• 10 choices:!cooking, fishing, running, playing Frisbee, watching TV,"feeding horse, playing guitar, texting, using computer,"writing on note"
□ Normalize images to fit in 500 × 500 pixels
Nov. 4, 2014 HCOMP-2014 19
Investigate properties of the IC protocol using the proposed metrics
Experiment: Result
■ Performance on Different Hyperparameters"□ Task quality: C = 100 is 1.1 times worse than C = 300 □ Privacy leakage: C = 300 is 1.9 times worse than C = 100 ⇒ Able to preserve privacy w/o degrading quality!
Nov. 4, 2014 HCOMP-2014 20
Proposed metrics can capture the trade-off
50 100 150 200 250 300C: clipping window size [pixels]
0.14
0.16
0.18
0.20
0.22
0.24
0.26
Task
info
rmat
ion
loss
[bit
s]
1.0
1.5
2.0
2.5
3.0
Priv
acy
info
rmat
ion
loss
[bit
s] Privacy"leakage
Task quality
Better
Size of sub-instance C [pixel]
Conclusion
■ Our Contributions"□ Design a PP Protocol: !
• Exploit different properties of (task, privacy)"
□ Evaluation Method:!1) Modeling task execution & privacy invasion processes"2) Propose metrics using the model"
□ Experiment:!• Evaluated the performance of the IC protocol"• Proposed evaluation method could capture the trade-off"
Nov. 4, 2014 HCOMP-2014 21
We propose a design principle of PP protocols as well as an evaluation method