Automated Software Engineering Research Group 1 Fix 12?: Title should be Limitations (?? Not...

Automated Software Engineering Research Group 1

Fix

• 12?: Title should be Limitations (?? Not Challenges)

• Slide 18: Verification -> counterexample collectoin

Computer Science

2

Mining Likely Properties of Access Control Policies

via Association Rule Mining

JeeHyun Hwang Advisor: Dr. Tao Xie

Preliminary Oral ExaminationDepartment of Computer Science

North Carolina State University, Raleigh


Access Control Mechanism

• Access control mechanisms control which subjects (such as users or processes) have access to which resources

• Policy defines rules according to which access control must be regulated

Request Response

(Permit, Deny)

Policy

Policy Evaluation

Engine


Access Control Mechanism

• Access control mechanisms control which subjects (such as users or processes) have access to which resources

• Policy defines rules according to which access control must be regulated

Policy

Request Response

(Permit, Deny)

Policy Evaluation

Engine

Faults


Research Accomplishments

• Quality of Access Control• Automated test generation [SRDS 08][SSIRI 09]

• Likely-property mining [DBSec 10]

• Property Quality Assessment [ACSAC 08]

• (Tool) Access Control Policy Tool (ACPT) [POLICY Demo 10]

• Debugging• Fault localization for

firewall policies [SRDS 09 SP]

• Automated fault correction for firewall policies [USENIX LISA 10]

• Performance• Efficient policy

evaluation engine [Sigmetrics 08]


Outline

•Motivation•Our approach•Future work


Outline



Motivation• Access control is used to control access to a large

number of resources [1,2]

• Specifying and maintaining correct access control policies is challenging [1,2]

– Authorized users should have access to the data– Unauthorized users should not have access to the data

• Faults in access control policies lead to security problems [1,3]

1. A. Wool. A quantitative study of firewall configuration errors. Computer, 37(6):62–67, 2004.2. Lujo Bauer, Lorrie Cranor, Robert W. Reeder, Michael K. Reiter, and Kami Vaniea . Real Life Challenges

in Access-control Management. CHI 20093. Sara Sinclair, Sean W. Smith. What's Wrong with Access Control in the Real World?. IEEE Security and

Privacy 2010


Motivation – cont.• Need to ensure the correct behaviours of policies

– Property verification [1,2]

• Model a policy and verify properties against the policies• Check whether properties are satisfied by a policy• Violations of a property expose policy faults

1. Kathi Fisler, Shriram Krishnamurthi, Leo A. Meyerovich, Michael Carl Tschantz. Verification and change-impact analysis of access-control policies. ICSE 2005

2. Vladimir Kolovski, James Hendler, Bijan Parsia, Analyzing Web Access Control Policies, WWW 20073. Michael Carl Tschantz, Shriram Krishnamurthi. Towards Reasonability Properties for Access-Control Policy Languages.

SACMAT 2006

PolicyPolicy

Property VerificationPropertyProperty Satisfy?

(True, False)1. Faculty member is permitted to assign grades [3]

2. Subject (who is not a faculty member) is permitted to enroll in courses [3]


Problem

• Quality of properties is assessed in terms of fault-detection capability [1]

• Properties help detect faults• Confidence on policy correctness is dependent on

the quality of specified properties• CONTINUE subject [2]: 25% fault-detection

capability with its seven properties• In practice, writing properties of high quality is not

trivial 1. Evan Martin, JeeHyun Hwang, Tao Xie, and Vincent C. Hu. Assessing quality of policy properties in verification of access

control policies. ACSAC 20082. Kathi Fisler, Shriram Krishnamurthi, Leo A. Meyerovich, Michael Carl Tschantz. Verification and change-impact analysis

of access-control policies. ICSE 2005


Proposed Solution

• Mine likely properties automatically based on correlations of attribute values (e.g., write and modify)• High quality of properties (with high fault-detection

capability)• Our assumption: • Policy may include faults • Mine likely properties,

which are true for all or most of the policy behaviors (>= threshold)

Policy

Faults

Likely Properties

Mine

Detect Faults


Limitations (?? Not Challenges)• Policy is domain-specific

• Mine likely properties within a given policy• Limited set of decisions

• Two decisions (Permit or Deny) for any request• Prioritization

• Which counterexamples should be inspected first?

• Expressiveness of likely properties• How to find counterexamples?


XACML Policy Example

RBAC_schoolpolicy

<Policy PolicySetId="n" PolicyCombiningAlgId=“First-Applicable"> <Target/> <Rule RuleId=“1" Effect=“Permit"><Target> <Subjects><Subject> Faculty </Subject></Subjects> <Resources><Resource> InternalGrade </Resource>

<Resource> ExternalGrade </Resource> </Resources> <Actions><Action> View </Action>

<Action> Assign </Action> </Actions></Target></Rule>

Rule 1

<Rule RuleId=“3" Effect=“Permit "><Target> <Subjects><Subject> FacultyFamily </Subject></Subjects> <Resources><Resource> ExternalGrade </Resource</Resources> <Actions><Action> Receive </Action> </Actions> </Target></Rule>

Rule 3

<Rule RuleId=“2" Effect=“Permit"><Target> <Subjects><Subject> Student</Subject></Subjects> <Resources><Resource> ExternalGrade </Resource</Resources> <Actions><Action> Receive </Action> </Actions> </Target></Rule>

Rule 2If role = Faculty

and resource = (ExternalGrade or InternalGrade)and action = (View or Assign) then Permit


XACML Policy Example – cont.

Rule 3: Jim can change grades orrecords.

RBAC_schoolpolicy

<Rule RuleId=“4" Effect=“Permit"><Target> <Subjects><Subject> Lecturer </Subject></Subjects> <Resources><Resource> InternalGrade </Resource> <Resource> ExternalGrade </Resource> </Resources> <Actions><Action> View </Action> <Action> Assign </Action> </Actions></Target></Rule>

Rule 4

<Rule RuleId=“5" Effect=“Permit"><Target> <Subjects><Subject> TA</Subject></Subjects> <Resources><Resource> InternalGrade</Resource> </Resources> <Actions><Action> View </Action>

<Action> Assign </Action> </Actions> </Target></Rule>

Rule 5

Rule 6<Rule RuleId=“6" Effect=“Deny"><Target/> </Rule></Policy>


XACML Policy Example – cont.

Rule 3: Jim can change grades orrecords.

RBAC_schoolpolicy

<Rule RuleId=“4" Effect=“Permit"><Target> <Subjects><Subject> Lecturer </Subject></Subjects> <Resources><Resource> InternalGrade </Resource> <Resource> ExternalGrade </Resource> </Resources> <Actions><Action> View </Action> <Action> Assign </Action> </Actions></Target></Rule>

Rule 4

<Rule RuleId=“5" Effect=“Permit"><Target> <Subjects><Subject> TA</Subject></Subjects> <Resources><Resource> InternalGrade</Resource> </Resources> <Actions><Action> View </Action>

<Action> Assign </Action> </Actions> </Target></Rule>

Rule 5

Rule 6<Rule RuleId=“6" Effect=“Deny"><Target/> </Rule></Policy>

View Receive

Inject a fault

(Receive instead of View)

Incorrect Policy Behaviors: 1. TA is Denied to View InternalGrade2. TA is Permitted to Receive InternalGrade

(View) Permit → (Assign) Permit : Frequency: 4 (100%)(Assign) Permit → (View) Permit : Frequency: 4 (80%)(Assign) Permit → (Receive) Deny : Frequency: 4 (80%)

(View) Permit → (Assign) Permit : Frequency: 5 (100%)(Assign) Permit → (View) Permit : Frequency: 5 (100%)(Assign) Permit → (Receive) Deny : Frequency: 5 (100%)


Policy Model• Role-Based Access Control Policy [1]

– Permissions are associated with roles– Subject (role) is allowed or denied access to

certain objects (i.e., resources) in a system• Subject: role of a person• Action: command that a subject executes on the

resource• Resource: object• Environment: any other related constraints (e.g.,

time, location, etc.)

1. XACML Profile for Role Based Access Control (RBAC), 2004


Likely-Property Model• Implication relation

• Correlate decision (dec1) for an attribute value (v1) with decision (dec2) for another attribute value (v2)

(v1) dec1 → (v2) dec2

• Types• Subject attribute

(TA) permit → (Faculty) permit

• Action attribute(Assign) permit → (View) permit

• Subject-action attribute:(TA & Assign) permit → (Faculty & View) permit


Framework

• Our assumption: • Policy may include faults • Mine likely properties, which are true for all

or most of the policy behaviors (>= threshold)

Counter example collection


Relation Table Generation

• Find all possible request-response pairs in a policy • Generate relation tables (including all request-

response pairs) of interest• Input for an association rule mining tool

1. Faculty is Permitted to Assign ExternalGrade

2. Faculty is Permitted to View ExternalGrade


Association Rule Mining

1. Agrawal, R., Srikant, R.: Fast algorithms for mining association rules in large databases. VLDB 19942. Borgelt, C.: Apriori - Association Rule Induction/Frequent Item Set Mining. 2009

• Given a relation table, find implication relations of attributes via association rule mining [1,2]

• Find three types of likely properties • Report likely properties with confidence values over

a given thresholdSupport: Supp (X) = D / T

% of the total number of records- T is #total rows- D is #rows that includes attribute-decision

X = (Assign) Permit , Y = Supp (View) Permit

Supp (X) = 5/10 = 0.5, Supp (Y) = 4/10 = 0.4Supp (X Y) = 4/10 = 0.4∪

• Confidence : Confidence (X → Y) =

supp(X Y)/supp(X)∪

* Likelihood of an likely property

Confidence (X → Y) = 4/5 = 0.8


Likely Property Verification• Verify a policy with given likely properties and find

counterexamples

Counterexample: (v1) dec1 → (v2) ¬ dec2

• Inspect to determine whether counterexamples expose a fault

Rationale: counterexamples (which do not satisfy the likely properties) deviate from the policy’s normal behaviors and are special cases for inspection


Basic and Prioritization Techniques

• Basic technique: inspect counterexamples in no particular order

Prioritization technique designed to reduce inspection effort

• Inspect counterexamples by the order of their fault-detection likelihood• Duplicate CE first• CE produced from

likely properties with fewer CE

Likely Properties

CE : Counterexamples

CE

Detect

CE Remove Duplication


Evaluation• RQ1: fault-Detection Capability

– How higher percentage of faults are detected by our approach compared to an existing related approach [1]?

• RQ2: cost– How lower percentage of distinct counterexamples are

generated by our approach compared to the existing approach [1]?

• RQ3: cost– For cases where a fault in a faulty policy is detected by our

approach, how high percentage of distinct counterexamples (for inspection) are reduced by our prioritization?

1. Evan Martin and Tao Xie. Inferring Access-Control Policy Properties via Machine Learning. POLICY 2006


Metrics• Fault-detection ratio (FR)• Counterexample count (CC)• Counterexample-reduction ratio (CRB) for our

approach over the existing approach• Counterexample-reduction ratio (CRP) for the

prioritization technique over the basic technique


Mutation Testing

• Fault-detection capability [1,2]

– Seed a fault into a firewall policy and generate a mutant (a faulty version)

– # of detected faults / Total # of faults

DecisionsCounterexample

Mutant(faulty version)

Expected Decisions

Policy(correct version)

1. Evan Martin, JeeHyun Hwang, Tao Xie, and Vincent C. Hu. Assessing quality of policy properties in verification of access control policies. ACSAC 2008

2. Evan Martin and Tao Xie. A Fault Model and Mutation Testing of Access Control Policies. WWW 2007


Evaluation Setup• Seed a policy with faults for synthesizing faulty policies

– One fault in each faulty policy for ease of evaluation– Four fault types [1]

• Change-Rule Effect (CRE)• Rule-Target True (RTT)• Rule-Target False (RTF)• Removal Rule (RMR)

• Compare results of our approach with those of the previous DT approach based on decision tree[2]

1. Evan Martin and Tao Xie. A Fault Model and Mutation Testing of Access Control Policies. WWW 20072. Evan Martin and Tao Xie. Inferring Access-Control Policy Properties via Machine Learning. POLICY 2006


4 XACML Policy Subjects

• Real-life access control policies– codeD2 : modified version of codeD [1]

– continue-a, continue-b [1] : policies for a conference review system

– Univ [2] : policies for a univ.

• The number of rules ranges 12-306 rules1. Kathi Fisler, Shriram Krishnamurthi, Leo A. Meyerovich, Michael Carl Tschantz. Verification and change-impact analysis

of access-control policies. ICSE 20052. Stoller, S.D., Yang, P., Ramakrishnan, C., Gofman, M.I.. Efficient policy analysis for administrative role based access

control. CCS 2007


Evaluation Results (1/2) – CRE Mutants

FR: Fault-detection ratio CC: Counterexample countCRB: Counterexample-reduction ratio for our approach over DT approachCRP: Counterexample-reduction ratio for the prioritization technique over the basic technique

• Fault detection ratios:• DT (25.9%), Basic (62.3%), Prioritization (62.3%)

• Our approach (including Basic and Prioritization techniques) outperform DT in terms of fault-detection capability


Evaluation Results (1/2) – CRE Mutants

FR: Fault-detection ratio CC: Counterexample countCRB: Counterexample-reduction ratio for our approach over DT approachCRP: Counterexample-reduction ratio for the prioritization technique over the basic technique

• Our approach reduced the number of counterexamples by 55.5% over DT

• Our approach reduced the number of counterexamples while our approach detected a higher percentage of faults (addressed in RQ1)

• Prioritization reduced averagely 38.5% of counterexamples (for inspection) (in Column “% CRP”) over Basic


Evaluation Results (2/2) – Other Mutants

• Prioritization and Basic achieve the highest fault-detection capability for policies with RTT, RTF, or RMR faults

Fault-detection ratios of faulty policies


Conclusion

• A new approach that mines likely properties characterizing correlations of policy behaviors w.r.t. attribute values

• An evaluation on 4 real-world XACML policies– Our approach achieved >30% higher fault-

detection capability than that of the previous related approach based on decision tree

– Our approach helped reduce >50% counterexamples for inspection compared to the previous approach


Outline



Future Work Dissertation Goal

Improving quality of access control: Automated Test Generation, likely properties mining

Debugging Fault Localization

•.Policy Combination• Access Control Policy Tool (ACPT)• Testing of policies in healthcare system• e.g., interoperability and regular compliance (e.g., HIPPA)


Questions?


Other Challenges

Generate properties of high qualityCover a large portion of policy behaviours

Obligation/Delegation/Environments


Related Work

• Assessing quality of policy properties in verification of access control policies [Martin et al. ACSAC 2008]

• Inferring access-control policy properties via machine learning [Martin&Xie Policy 2006]

• Detecting and resolving policy misconfigurations in access-control systems [Bauer et al. SACMAT 2008]


My Other Research Work


Systematic Structural Testing of Firewall Policies

JeeHyun Hwang1, Tao Xie1, Fei Chen2, and Alex Liu2

North Carolina State University1

Michigan State University2

(SRDS 2008)


Problem

• Factors for misconfiguration– Conflicts among rules– Rule-set complexity– Mistakes in handling corner cases

• Systematic testing of firewall policies– Exhaustive testing is impractical– Considering test effort and their

effectiveness together– Complementing firewall verification

How to test

Firewall?


Firewall Policy Structure• A Policy is expressed as a set of rules.

Rule Src SPort Dest DPort Protocol Decision

r1 * * 192.168.*.* * * accept

r2 1.2.*.* * * * TCP discard

r3 * * * * * discard

• A Rule is represented as <predicate> → <decision>• <predicate> is a set of <clauses>

predicate

decision

• <decision> is “accept” or “discard”• Given a packet (Src, Sport, Dest, Dport, Protocol)

– When <predicate> is evaluated “True”, <decision> is returned

• <clauses>– Src, Sport, Dest, Dport, Protocol– Representing Integer range

• Given a packet (Src, Sport, Dest, Dport, Protocol)– <clause> can be evaluated “True” or “False”.

Firewall Format: CISCO REFLEXIVE ACLS


Random Packet Generation

• Given domain range (e.g., IP addresses [0, 28-1]), random packets are generated within the domain.

Src SPort Dest DPort Protocol

Domain * * * * *

162.168.12.5 5 192.168.0.5 10 TCP

• Easy to generate packets• Due to its randomness, difficult to achieve high

structural coverage


Packet Generation based on Local Constraint Solving

• Considering an individual rule, generates packets to evaluate constraints of clauses in a specified way

– For example, every value is evaluated to true

162.168.12.5 5 168.1.0.5 10 TCP162.168.12.5 5 192.168.0.5 10 TCP

– For example, Dest field value is evaluated to false, and the remaining values are evaluated to true


r1 * * 192.168.*.* * * accept

T T T TTT T T TF

• Conflicts among rules


Packet Generation based on Global Constraint Solving

• Considering preceding rules are not applicable, generates packets to evaluate constraints of certain rule’s clauses in a specified way

– Packet is applicable to r3 (considering that r1and r2 are not applicable)

162.168.12.5 5 1.5.0.5 10 TCPT T T TT

• Resolving conflicts among rules and require analysis time to solving such conflicts


r1 * * 192.168.*.* * * accept

r2 1.2.*.* * * * TCP discard

r3 * * * * * discard

FF


Mutation Testing

• Why mutation testing?– Measure the quality of a test packet set (i.e., fault detection

capability)• Seed a fault into a firewall policy and generate a mutant

(a faulty version)

.

Decisions

Test Packets

Mutant(faulty version)

Expected Decision

s

Firewall(correct version)

• Compare their decisions– The fault is detected in a mutant (i.e., the mutant is “killed”).


11 Mutation Operators

Remove RuleRMR

Change Rule DecisionCRD

Change Rule OrderCRO

Change Range End point OperatorCREO

Change Range Start point OperatorCRSO

Change Range End point ValueCREV

Change Range Start point ValueCRSV

Rule Clause FalseRCF

Rule Clause TrueRCT

Rule Predicate False RPF

Rule Predicate True RPT

DescriptionOperator


Experiment

• Given a firewall policy (assuming correct!) – Mutants– Packet sets (for each technique)

• Investigating the following correlations– Packet sets and their achieved structural coverage– Structural coverage criteria and fault-detection

capability– Packet sets and their reduced packet sets in terms

of fault-detection capability• Characteristics of each mutation operator


Experiment (Contd...)

• Notations– Rand : packet set generated by the random packet

generation technique– Local : packet set generated by the packet

generation technique based on local constraint solving

– Global : packet set generated by the packet generation technique based on global constraint solving

– R-Rand, R-Local, and R-Global are their reduced packet sets


Subjects• We used 14 firewall policies• Number of test packets : approximately 2 packets per rule

# Rules :

Number of rules

# Mutants :

Number of mutants

Gen time (milliseconds) :

packet generation time

(particularly for Global)

Global :

global constraint solving


Measuring Rule Coverage

Rand < Local ≤ Global

• Rand achieves the lowest rule coverage

• In general, Global achieves slightly higher rule coverage than Local


Reducing the number of packet sets

• Reduced packet set (e.g., R-Rand)

– Maintain same level of structural coverage

– R-Rand (5% of Rand), R-Local (66% of Local), and R-Global (60% of Global)

– Compare their fault-detection capabilities


Fault detection capability by subject policies

R-Rand ≤ Rand < R-Local ≤ Local < R-Global≤ Global

• Packet set with higher structural coverage has higher fault-detection capability


Fault detection capability by mutation operators

• Mutant killing ratios vary by mutation operators– Above 85% : RPT

– 30% - 40% : RPF, RMR

– 10 – 20% : CRSV, CRSO

– 0% - 10% : RCT, RCF, CREV, CREO, CRO


Related Work

• Testing of XACML access control policies [Martin et al. ICICS 2006, WWW 2007]

• Specification-based testing of firewalls [J¨urjens et al. PSI 2001]– State transition model between firewall and its

surrounding network• Defining policy criteria identified by

interactions between rules [El-Atawy et al. Policy 2007]


Conclusion• Firewall policy testing helps improve our

confidence of firewall policy correctness• Systematic testing of firewall policies

– Structural coverage criteria– Three automated packet generation techniques

• Measured Coverage: Rand < Local ≤ Global

• Mutation testing to show the fault detection capability– Generally, a packet set with higher structural

coverage has higher fault-detection capability– Worthwhile to generate test packet set for

achieving high structural coverage


Fault Localization for Firewall Policies

55

JeeHyun Hwang1, Tao Xie1, Fei Chen2, and Alex Liu2

North Carolina State University1

Michigan State University2

Symposium on Reliable Distributed Systems (SRDS 2009)


Fault Model

• Faults in an attribute in a Rule– Rule Decision Change (RDC)

• Change Rule Decision– R1: F1∈[0,10]∧F2∈[3, 5] →accept– R1’: F1∈[0,10]∧F2∈[3, 5] →deny

– Rule Field Interval Change (RFC)• Change selected rule’s interval randomly

– R1: F1∈[0,10]∧F2∈[3, 5] →accept– R1’: F1∈[2,7]∧F2∈[3, 5] →accept


Overview of Approach

• Input– Faulty Firewall Policy– Failed and Passed Test Packets

• Techniques– Covered-Rule-Fault Localization– Rule Reduction Technique – Rule Ranking Technique

• Output– Set of likely faulty rules (with their

ranking)


Covered-Rule-Fault Localization

R1: F1 [0,10]∈ ∧F2 [3, 5]∈ ∧F3 [3, 5] → ∈ accept

R2: F1 [5, 7]∈ ∧F2 [0, 10]∈ ∧F3 [3, 5] → ∈ discard

R3: F1 [5, 7]∈ ∧F2 [0, 10]∈ ∧F3 [6, 7] → ∈ accept

R4: F1 [2,10]∈ ∧F2 [0, 10]∈ ∧F3 [5,10] → ∈ discard

R5: F1 [0,10]∈ ∧F2 [0, 10]∈ ∧F3 [0,10] → ∈ discard

• Inspect a rule covered by a failed test

• R4 is selected for inspection• RDC faulty rule is effectively filtered out

# Rule Cov Se-lected

#Failed

#Passed

00020

22202

●ac-cept

Inject a Rule Decision Change Fault in R4

• “accept” rather than “dis-card”


Rule Reduction Technique

• Reduce # of rules for inspection• The earliest-placed rule covered by failed

tests; R4• Other rules with following criterion

• Rules above r’; R1, R2, R3• Rules with different decision of r’; R1, R3

R1: F1 [0,10]∈ ∧F2 [3, 5]∈ ∧F3 [3, 5] ∈ → accept

R2: F1 [5, 7]∈ ∧F2 [0, 10]∈ ∧F3 [3, 5] → ∈ discard

R3: F1 [5, 7]∈ ∧F2 [0, 10]∈ ∧F3 [6, 7] → ∈ accept

R4: F1 [2,10]∈ ∧F2 [0, 10]∈ ∧F3 [5,10] → ∈ discard

R5: F1 [0,10]∈ ∧F2 [0, 10]∈ ∧F3 [0,10] → ∈ discard

# Rule Cov Se-lected#Fa

iled

#Passed

00021

22212

●

●●

Inject a Field Interval Change Fault in R1’s F3

• F3∈[3, 3] rather than F3∈[3, 5] F3 [3, 3]∈


Rule Ranking Technique

• Rank rules based on their likelihood of being faulty using clause coverage.– FC1 <= FC2

• FC1: # clauses that are evaluated to false in a faulty rule• FC2: # clauses that are evaluated to false in other rules

– Ranking is calculated based on the following formula

FF(r) : # clauses evaluated to falseFT(r) : # clauses evaluated to true


Experiments

• 14 firewall policies

# Rules: # of rules

# Tests: # generated test

packets

# RDC: # RDC faulty poli-

cies

#RFC # RFC faulty poli-

cies


Results: Covered-Rule-Fault Localization

• 100 % of RDC faulty rules are detected• 69 % of RFC faulty rules are detected• 31% of RFC faulty rules are not covered by

only failed test


Results: Rule Reduction for Inspection

• Rule reduction percentage: % Reduce (30.63 % of

Rules)• Ranking-based rule reduction

percentage: % R-Reduce (66% of

Rules)


Conclusion and Future Work

• Our techniques help policy authors locating faults effectively by reducing # of rules for inspection– 100% of RDC faulty rules and 69% of

RFC faulty rules can be detected by inspecting covered rules

– 30.63% of rules on average are reduced for inspection based on our rule reduction technique and 56.53% of rule ranking technique

• We plan to investigate fault localization for multiple faults in a firewall policy

Automated Software Engineering Research Group 1 Fix 12?: Title should be Limitations (?? Not...

Documents

Transcript of Automated Software Engineering Research Group 1 Fix 12?: Title should be Limitations (?? Not...