Anupam DattaAnupam Datta

CMUCMU

Joint work with Adam Barth, John Mitchell Joint work with Adam Barth, John Mitchell (Stanford), Helen Nissenbaum (NYU) and Sharada (Stanford), Helen Nissenbaum (NYU) and Sharada Sundaram (TCS)Sundaram (TCS)

Privacy and Contextual Integrity:Framework and Applications

2

Problem Statement

Is an organization’s business process compliant with privacy regulations and internal policies?

Examples of organizations– Hospitals, financial institutions, other enterprises handling sensitive

information Examples of privacy regulations

– HIPAA, GLBA, COPPA, SB1386

Goal: Develop methods and tools to answer this question

3

Privacy Project Space

What is Privacy?[Philosophy, Law, Public Policy]

Formal Model, Policy Language,Compliance-check Algorithms

[Programming Languages, Logic]

Implementation-level Compliance[Software Engg, Formal Methods]

Data Privacy[Databases, Cryptography]

4

Project Overview

What is privacy?– Conceptual framework

Policy language – Privacy laws including HIPAA, COPPA, GLBA expressible

Compliance-check algorithms– Does system satisfy privacy and utility goals?

Case studies– Patient portal deployed at Vanderbilt Hospital– UPMC (ongoing discussions)– TCS

5

Contextual Integrity [N2004]

Philosophical framework for privacy Central concept: Context

– Examples: Healthcare, banking, education What is a context?

– Set of interacting agents in roles Roles in healthcare: doctor, patient, …

– Norms of transmission Doctors should share patient health information as per the HIPAA rules

– Purpose Improve health

6

Nurse

Secretary

MyHealth@Vanderbilt Workflow

Patient

Doctor

Health Answer

Health AnswerHealth Question

Appointment R

equest

Healt

h Q

uest

ion

Health Questio

n

Now that I have cancer,Should I eat more vegetables?

Yes! except broccoli

Privacy: HIPAA compliance+

Humans + Electronic system

Utility: Schedule appointments, obtain health answers

7

Nurse

Secretary

MyHealth@Vanderbilt Improved

Patient

Doctor

Health Answer

Health Answer

Health Question

Appointment R

equest

Health Question

Now that I have cancer,Should I eat more vegetables?

HealthQuestion

Yes! except broccoli

HealthAnswer

•Message tags used for policy enforcement

•Minimal disclosure

Responsibility: Doctor should answer health questions

8

Privacy vs. Utility

Privacy– Certain information should not

be communicated Utility

– Certain information should be communicated

Tension between privacy and utility

Perm

issiveness

Workflows

ViolatePrivacy

ViolateUtility

FeasibleWorkflows

“Minimum necessary”

9

Design-time Analysis: Big Picture

ContextualIntegrity

Business Objectives Privacy Policy

Business ProcessDesign

PrivacyChecker

(LTL)

UtilityChecker(ATL*)

UtilityEvaluation

PrivacyEvaluation

NormsPurpose

Assuming agents responsible

10

Auditing: Big Picture

Business ProcessExecution

AuditLogs

Run-time Monitor

Privacy PoliciesUtility Goals

AuditAlgos

Policy Violation+

Accountable Agent

Agents may not be responsible

11

In more detail…

Model and logicPrivacy policy examples

– GLBA – financial institutions– MyHealth portal

Compliance checking– Design time analysis (fully automated)– Auditing (using oracle)

Language can express HIPAA, GLBA, COPPA [BDMN2006]

12

Model

Alice

– Communication via send actions: Sender: Bob in role Patient Recipient: Alice in role Nurse Subject of message: Bob Tag: Health Question Message: Now that ….

– Data model & knowledge evolution: Agents acquire knowledge by:

– receiving messages – deriving additional attributes based on data model

Health Question Protected Health Information

BobNow that I have cancer,

Should I eat more vegetables?

HealthQuestion

contents(msg) vs. tags (msg)

Inspired by Contextual Integrity

13

Model

State determined by knowledge of each agent

Transitions change state– Set of concurrent send actions– Send(p,q,m) possible only if

agent p knows m

K0

K13

K11

......

K12

A11

A12

A13

Concurrent Game Structure

G = <k, Q, , , d, >

[BDMN06, BDMS07]

14

Logic of Privacy and Utility

Syntax ::= send(p1,p2,m) p1 sends p2 message m

| contains(m, q, t) m contains attrib t about q | tagged(m, q, t) m tagged attrib t about q | inrole(p, r) p is active in role r | t t’ Attrib t is part of attrib t’ | | | x. Classical operators | U | S | O Temporal operators

| <<p>> Strategy quantifier Semantics

Formulas interpreted over concurrent game structure

15

Specifying Privacy

MyHealth@Vanderbilt

In all states, only nurses and doctors receive health questions

G p1, p2, q, m

send(p1, p2, m) contains(m, q, health-question)

inrole(p2, nurse) inrole(p2, doctor)

16

Specifying Utility

MyHealth@Vanderbilt

Patients have a strategy to get their health questions answered

p inrole(p, patient)

<<p>> F q, m. send(q, p, m) contains(m, p, health-answer)

17

MyHealth Responsibilities

Tagging Nurses should tag health questions

G p, q, s, m. inrole(p, nurse) send(p, q, m) contains(m, s, health-question)

tagged(m, s, health-question) Progress

– Doctors should answer health questionsG p, q, s, m. inrole(p, doctor) send(q, p, m)

contains(m, s, health-question) F m’. send(p, s, m’) contains(m’, s, health-answer)

18

Sender role Subject roleAttribute

Temporal condition

Gramm-Leach-Bliley Example

Recipient role

Financial institutions must notify consumers if they share their non-public personal information with non-affiliated companies, but the notification may occur either before or after the information sharing occurs

19

Workflow Design Results

Theorems:Assuming all agents act responsibly, checking whether workflow

achieves – Privacy is in PSPACE (in the size of the formula describing the

workflow) Use LTL model-checking algorithm

– Utility is decidable for a restricted class of formulas ATL* model-checking is undecidable for concurrent game structures with

imperfect information, but decidable with perfect information

Idea: – Check that all executions satisfy privacy and utility properties

Definition and construction of minimal disclosure workflow

Algorithms implemented in model-checkers, e.g. SPIN, MOCHA

20

Auditing Results

Who to blame? Accountability– Irresponsibility + causality

Design of audit log– Use Lamport causality structure, standard concept from distributed

computing Algorithms

– Finding agents accountable for policy violation in graph-based workflows using audit log

– Finding agents who act irresponsibly using audit log Algorithms use oracle:

– O(msg) = contents(msg)– Minimize number of oracle calls

21

Conclusions

Framework inspired by contextual integrity Business Process as Workflow

Role-based responsibility for human and mechanical agents Compliance checking

Workflow design assuming agents responsible Privacy, utility decidable (model-checking) Minimal disclosure workflow constructible

Auditing logs when agents irresponsible From policy violation to accountable agents Finding irresponsible agents

Case studies– MyHealth patient portal deployed at Vanderbilt University hospital– Ongoing interactions with UPMC

Using oracle

Automated

22

Future Work

Framework– Do we have the right concepts? – Adding time , finer-grained data model– Priorities of rules, inconsistency, paraconsistency

Compliance vs. risk management Privacy principles

– Minimum necessary one example; what else? Improve algorithmic results

– Utility decidability; small model theorem– Auditing algorithms

Privacy analysis of code– Current results apply to system specification

More case studies– Focusing on healthcare

Detailed specification of privacy laws– Immediate focus on HIPAA, GLBA, COPPA

Legal, economic incentives for responsible behavior

23

Publications/Credits

– A. Barth, A. Datta, J. C. Mitchell, S. Sundaram

Privacy and Utility in Business Processes, to appear in Proceedings of 20th IEEE Computer Security Foundations Symposium, July 2007.

– A. Barth, A. Datta, J. C. Mitchell, H. Nissenbaum

Privacy and Contextual Integrity: Framework and Applications, in Proceedings of 27th IEEE Symposium on Security and Privacy , pp. 184-198, May 2006.

Work covered in The Economist, IEEE Security &

Privacy editorial

24

Thanks

Questions?

25

Additional Technical Slides

26

Related Languages

Model Sender Recipient Subject Attributes Past Future Combination

RBAC Role Identity

XACML Flexible Flexible Flexible o o

EPAL Fixed Role Fixed o

P3P Fixed Role Fixed o o

LPU Role Role Role

Legend: unsupportedo partially supported fully supported

LPU fully supports attributes, combination, temporal conditions

Utility not considered

27

Deciding Utility

ATL* model-checking of concurrent game structures is – Decidable with perfect information– Undecidable with imperfect information

Theorem:There is a sound decision procedure for deciding whether workflow

achieves utility Intuition:

– Translate imperfect information into perfect information by considering all possible actions from one player’s point of view

28

Local communication game

Quotient structure under invisible actions, Gp

– States:Smallest equivalence relation

K1 ~p K2 if K1 K2 and a is invisible to p – Actions:

[K] [K’] if there exists K1 in [K] and K2 in [K’] s.t. K1 K2

Lemma: For all LTL formulas visible to p, Gp |= <<p>> implies G |= <<p>>

29

Auditing Results

Definitions– Policy compliance, locally compliant– Causality, accountability

Design of audit log Algorithms

– Finding agents accountable for locally-compliant policy violation in graph-based workflows using audit log

– Finding agents who act irresponsibly using audit log Algorithms use oracle:

– O(msg) = contents(msg)– Minimize number of oracle calls

30

Policy compliance/violation

Strong compliance [BDMN2006]– Action does not violate current policy requirements– Future policy requirements after action can be met

Locally compliant policy– Agents can determine strong compliance based on their local view of history

Policy

History

Contemplated ActionJudgment

Future Reqs

31

Causality

Lamport Causality

[1978]“happened-before”

32

Accountability & Audit Log

Accountability– Causality + Irresponsibility

Audit log design– Records all Send(p,q,m) and Receive(p,q,m) events executed– Maintains causality structure

O(1) operation per event logged

33

Auditing Algorithm

GoalFind agents accountable for a policy violation

Algorithm(Audit log A, Violation v)1. Construct G, the causality graph for v in A2. Run BFS on G.

At each Send(p, q, m) node, check if tags(m) = O(m). If not, and p missed a tag, output p as accountable

Theorem: – The algorithm outputs at least one accountable agent for every

violation of a locally compliant policy in an audit log of a graph-based workflow that achieves the policy in the responsible

model

34

Proof Idea

Causality graph G includes all accountable agents– Accountability = Causality + Irresponsibility

There is at least one irresponsible agent in G– Policy is satisfied if all agents responsible – Policy is locally compliant

In graph-based workflows, safety responsibilities violated only by mistagging

– O(msg) = tags(msg) check identifies all irresponsible actions

35

MyHealth Example

1. Policy violation: Secretary Candy receives health-question mistagged as

appointment-request

2. Construct causality graph G and search backwards using BFS Candy received message m from Patient Jorge. O(m) = health-question, but tags(m) = appointment-request. Patient responsible for health-question tag. Jorge identified as accountable