A Mission-Centric Framework for Cyber Situational Awareness Metrics, Lifecycle of Situational...

A Mission-Centric Framework for Cyber Situational Awareness

Metrics, Lifecycle of Situational Awareness, and Impact of Automated Tools on Analyst Performance

S. Jajodia, M. AlbaneseGeorge Mason University

ARO-MURI on Cyber-Situation Awareness Review MeetingSanta Barbara, CA , November 18-19, 2014

ARO-MURI on Cyber-Situation Awareness Review Meeting

2

Outline

November 18-19, 2014

Overview of Mason’s Role Year 5 Statistics Metrics

Measuring Security Risk Network Diversity

Lifecycle of Situational Awareness Impact of SA on Analyst Performance Conclusions

3


Overview of Mason’s Role



4

Where We Stand in the Project

System Analysts

Computer network

•Software•Sensors, probes•Hyper Sentry•Cruiser

Mu

lti-

Sen

sory

H

um

an

C

om

pu

ter

Inte

racti

on

• Enterprise Model• Activity Logs • IDS reports• Vulnerabilities

Cognitive Models & Decision Aids

• Instance Based Learning Models

• Simulation• Measures of SA & Shared SA

Data

C

on

dit

ion

ing

Associa

tion

&

Corr

ela

tion

• • •

Automated Reasoning Tools• R-CAST• Plan-based

narratives• Graphical

models• Uncertainty

analysis

Information Aggregation & Fusion•Transaction Graph methods•Damage assessment

Computer

network

Real World

Test-bed


• • •


5

Situation Knowledge Reference

Model[Attack Scenario

Graphs]

Index & Data Structures

Topological Vulnerability

Analysis

Our Vision

Monitored Network

Analyst

Alerts/Sensory Data

Cauldron

Switchwall

Vulnerability Databases

NVD OSVDCVE

Stochastic Attack Models

GeneralizedDependency

Graphs

Graph Processing

and Indexing

Dependency AnalysisNSDMin

er

Scenario Analysis & Visualization

Network Hardening

Unexplained Behavior Analysis

Zero-day Analysis

Cauldron



6

Overview of Contribution – Year 1 Technical accomplishments

A topological approach to Vulnerability Analysis that overcomes the drawbacks of traditional point-wise vulnerability analysis

Preliminary data structures and graph-based techniques and algorithms for processing alerts/sensory data

A novel security metric, k-zero day safety, to assess how many zero-day vulnerabilities are required for compromising a network asset

Major breakthroughs Capability of processing massive amounts of alerts in real-

time Capability of forecasting possible futures of the current

situation Capability of hardening a network against zero day

vulnerabilities



7


Generalized dependency graphs, which capture how network components depend on one other

Probabilistic temporal attack graphs, which encode probabilistic and temporal knowledge of the attacker’s behavior

Attack scenario graphs, which combine dependency and attack graphs Efficient algorithms for both detection and prediction

A preliminary model to identify “unexplained” cyber activities, i.e., activities incompatible with any given known activity model

Major breakthroughs Capability of generating and ranking future attack

scenarios in real timeNovember 18-19, 2014


8


An efficient and cost-effective algorithm to harden a network with respect to given security goals

A probabilistic framework for localizing attackers in mobile networks

A probabilistic framework for assessing the completeness and quality of available attack models (joint work with UMD and ARL)

A suite of novel techniques to automatically discover dependencies between network services from passively collected network traffic

Switchwall, an Ethernet-based network fingerprinting technique for detecting unauthorized changes to the L2/L3 network topology

Major breakthroughs Capability of automatically and efficiently executing

several important analysis tasks, namely hardening, dependency analysis, and attacker localization



9


Effective and efficient methods for generating partial attack graphs on demand in order to enable efficient analysis of zero-day vulnerabilities

A three-step process to assess the risk associated with zero-day vulnerabilities

A prototype of the probabilistic framework for unexplained activity analysis

Major breakthroughs Capability to reason about zero-day vulnerabilities

and efficiently assess the risk associated with such vulnerabilities without generating the entire attack graphNovember 18-19, 2014


10


A suite of metrics for measuring network-wide cyber security risk based on attack graphs

An approach to model network diversity as a security metric for evaluating the robustness of networks against zero-day attacks

An analysis of how situational awareness forms and evolves during the several stages of the cyber defense process

An analysis of how automated CSA tools can be used for improving analyst performance

Major breakthroughs Capability of quantifying risk and resiliency using

several metrics



11

Quad Chart - Year 5


Objectives: Improve Cyber Situation Awareness via• Metrics for measuring network-wide cyber security risk • An better understanding of the impact of network diversity on the

robustness of networks against zero-day attacks• A better understanding of how situational awareness forms and evolves• A better understanding of how automated CSA tools can improve analyst

performance

DoD Benefit: • Ability to quantitatively evaluate network-wide security risks• Ability to better design automated CSA tools that can effectively reduce

the workload for the analysts and improve their performance

Scientific/Technical Approach• Defining a hierarchy of attack graph based metrics, and developing

metrics • Studying diversity as a network-wide metrics to asses resilience against

zero-day attacks, and defining several diversity-based metrics:

biodiversity inspired, least attacking effort, and average attacking effort• Studying situational awareness capabilities from a functional point of

view, and identifying inputs, outputs, and lifecycle of the derived

awareness• Examining the impact of automated tools on analyst performance

Major Accomplishments• Defined a suite of metrics for measuring network-wide cyber

security risk based on a model of multi-step attack vulnerability (attack graph)• Modeled network diversity as a security metric for evaluating the

robustness of networks against zero-day attacks• Studied how situational awareness forms and evolves during the

several stages of the cyber defense process, and how automated CSA tools can be used for improving analyst performance

Challenges

• Defining solid metrics that accurately capture risk and resilience

12


Year 5 Statistics



13

Year 5 Statistics (1/2)

Publications & presentations 3 papers published in peer-reviewed conference

proceedings 1 paper published in a peer-reviewed journal 2 book chapters 1 book

L. Wang, M. Albanese, and S. Jajodia, “Network Hardening: An Automated Approach to Improving Network Security,” ISBN 978-3-319-04611-2, SpringerBriefs in Computer Science, 2014, 60 pages

Supported personnel 2 faculty 1 doctoral student 1 undergraduate student



14

Year 5 Statistics (2/2)


Patents Awarded during the reporting period Sushil Jajodia, Lingyu Wang, and Anoop Singhal, “Interactive

Analysis of Attack Graphs Using Relational Queries”, United States Patent No. 8,566,269 B2, October 22, 2013.

Steven Noel, Sushil Jajodia, and Eric Robertson, “Intrusion Event Correlation System”, United States Patent No. 8,719,943 B2, May 6, 2014.

Patents Disclosed during the reporting period Massimiliano Albanese, Sushil Jajodia, and Steven Noel,

“Methods and Systems for Determining Hardening Strategies”, United States Patent Application No. US 2014/0173740 A1, June 19, 2014.

Honors & Awards Max Albanese received the 2014 Mason Emerging

Researcher/Scholar/Creator Award

15


Steven Noel and Sushil Jajodia, “Metrics suite for network attack graph analytics,” Proceedings of the 9th Cyber and Information Security Research Conference (CISR 2014), Oak Ridge, TN, USA, April 8-10, 2014

Metrics: Measuring Security Risk



16

Overview


Attack (vulnerability dependency) graphs Combine information about topology, policy, and

vulnerabilities Identify network vulnerability paths Provide qualitative rather than quantitative

insights Attack graph metrics

Capture trends over time Enable comparisons across organizations Look at complementary dimensions of security


17

Cauldron Attack Graph



18

Attack Graph Metrics


Network Topology

Firewall Rules

Host Vulnerabilities

Attack GraphAnalysis

MetricsEngine

MetricsDashboard

NessusRetinanCirclenmap

…

Cisco ASACisco IOS

Juniper JUNOSJuniper ScreenOS

…

XMLCSV

Graphical


19

Attack Graph Metrics Families


Victimization: Individual vulnerabilities and exposed services each have elements of risk We score the entire network across individual vulnerability

victimization dimensions

Size: The size of attack graphs is a prime indication of risk The larger the graph, the more ways to be compromised

Containment: Networks are generally administered in pieces (subnets, domains, etc.) Risk mitigation should aim to reduce attacks across such

boundaries

Topology: The connectivity, cycles, and depth of the attack graph indicate how graph relationships enable network penetration


20

Metrics Hierarchy


Overall

Victimization

Existence

Exploitability

Impact

Size

Vectors

Machines

Containment

Vectors

Machines

Vuln Types

Topology

Connectivity

Cycles

Depth

Network Score

Metrics

Family

Individual

Metrics


21

Victimization Metrics


Existence – relative number of ports that are vulnerable (on a 0 to 10 scale)

Exploitability – average CVSS Exploitability

Impact – average CVSS Impact

UueU

i i

1lityExploitabi

,Impact1

UumU

i i

nv

v

ss

s

10Existence


22

Size Family: Vectors Metric


Within domain (implicit vectors)

Across domains:explicit vectors

jiv ,

im

j ji vm 1

d

ji ji

d

i

m

j jia vvmv i

, ,1 torsAttack vec

m

i ip smv 1 torsattack vec possible Total

p

a

v

v10Size Vectors


23

Size Family: Machines Metric


Vulnerable machines

d

i irr1

Non-vulnerable machines

d

j jmm1

mr

r

10Size Machines


24

Containment Family: Vectors Metric


Within domain (implicit vectors)

Across domains:explicit vectors

jiv ,

im

j ji vm 1

d

ji ji

d

i

m

j jia vvmv i

, ,1 torsAttack vec

d

ji jic vv, ,domains across torsAttack vec

a

c

v

v10tContainmen Vectors


25

Containment Family: Machines Metric


Victims across domains

Victimswithin domain only

d

i iiw Vmmmm , d

i iia Vmmmm ,

wa

a

mm

m

10tContainmen Machines


26

Containment Family: Vulnerability Types


Vulnerability typesacross

domains

Vulnerability typeswithin domain only

d

i iiiiw Vmtmmtt ,

d

i iiiia Vmtmmtt ,

wa

a

tt

t

10tContainmen Types Vuln


27

Attack Graph Connectivity


OneComponent

TwoComponents

ThreeComponents

Motivation: Better to have attack graph as disconnected parts versus connected whole

LessSecure

MoreSecure


28

Topology Family: Connectivity Metric


1 component 4 components 5 components

10111

11110Metric

7111

14110Metric

6111

15110Metric

1

1110Metricd

w


29

Attack Graph Cycles


Motivation: For a connected attack graph, better to avoid cycles among subgraphs

LessSecure

MoreSecure


30

Topology Family: Cycles Metric


4 components 5 components 10 components

7111

14110Metric

6111

15110Metric

1111

110110Metric

1

1110Metricd

s


31

Attack Graph Depth


One StepDeep

2 StepsDeep

3 StepsDeep

LessSecure

MoreSecure

Motivation: Better to have attack graph deeper versus

shallower


32

Topology Family: Depth Metric


Shortest path 3/8 Shortest path 4/8Shortest paths 2/3 and 1/57.5

18

3110Metric

3.4

18

4110Metric

3.2

15

115

13

213

82

10Metric

n

i i

ii c

sc

dn 1 11

10Metric


33

Metrics Dashboard



34

Trend Summary



35

Example Network Topology


PartnerDomains

InternalDomains

DMZ


36

Attack Graph – Before Hardening



37

Attack Graph – After Hardening


38


L. Wang, M. Zhang, S. Jajodia, A. Singhal, and M. Albanese, “Modeling Network Diversity for Evaluating the Robustness of Networks against Zero-Day Attacks,” Proceedings of the 19th European Symposium on Research in Computer Security (ESORICS 2012), Wroclaw, Poland, September 7-11, 2014

Metrics: Network Diversity



39

Overview


Zero-day attacks are a real threat to mission critical networks

Governments and cybercriminals are stockpiling zero-day vulnerabilities1

The NSA spent more than $25 million a year to acquire software vulnerabilities

Example. Stuxnet exploits 4 different/complementary zero day vulnerabilities to infiltrate a SCADA network

But what can we do about unknown attacks?1 http://krebsonsecurity.com/2013/12/how-many-zero-days-hit-you-

today/


40

How Could Diversity Help?


Stuxnet’s attack strategy 3rd party (e.g., contractor) organization’s

network machine with Siemens Step 7 PLC

The degree of software diversity along potential attack paths can be considered a good metric for the network’s capability of resisting Stuxnet


41

Existing Work on Diversity


Software diversity has long been regarded as a security mechanism for improving robustness The degree of diversity along potential attack

paths is an indicator of the network’s capability of resisting attacks

Tolerating attacks as Byzantine faults by comparing outputs or behaviors of diverse variants

Limitations: At a higher abstraction level, as a global property of an entire network, network diversity and its impact on security has not been formally modeled


42

Our Contribution

We take the first step towards formally modeling network diversity as a security metric We propose a network diversity function based on

well known mathematical models of biodiversity in ecology

We design a network diversity metric based on the least attacking effort

We design a probabilistic network diversity metric to reflect the average attacking effort

We evaluate the metrics and algorithms through simulation

The modeling effort helps understand diversity and enables quantitative hardening approaches



43

Bio-Diversity and Richness of Species

Literature on biodiversity confirms a positive relationship between biodiversity and the ecosystem’s resistance to invasion and diseases

Richness of species The number of different species in an ecosystem Limitation: ignores the relative abundance of each species

Effective number or resources Measures the equivalent number of equally-common species, even

if in reality all species are not equally common Limitation: assumes all resources are equally different

Similarity-Sensitive Effective Richness We can use a resource similarity function to account for differences

between resources November 18-19, 2014

44


Resource Graph

Syntactically equivalent to an attack graph Models causal relationships between

network resources (rather than vulnerabilities)

Vertices: zero-day exploits, their pre- and post-conditions

Edges: AND between pre-conditions, OR between exploits On which path should we compute the

diversity metrics?



45

Selecting the Least Diverse Path(s) Intuitively, it should be the “shortest” path

1 or 2 have the minimum number of steps, but 4 may take less effort than 1!

2 or 4 have the minimum number of resources? But they both have 2 resources, so which one is better?

4 minimizes #resources/#steps? But what if there is a path with 9 steps and 3 resources? 1/3<2/4, but it clearly does not represent the least attack effort!



46

Network Diversity in Least Attack Effort We define network diversity as:

Note: These may or may not be the same path! In this case: 2 (path 2, 4) / 3 (path 1, 2)

Determining the network diversity is NP-hard Our heuristic algorithm only keeps a limited number

of local optima at each step



47

Network Diversity in Average Effort The least attacking effort-based metric only

provides a partial picture of the threat We now define a probabilistic network

diversity metric based on the average attacking effort Defined as , where

is the probability an attacker can compromise a given asset now, and

is the probability he/she can still compromise it if all the resources were to be made different (i.e., every resource type would appear at most once)



Simulation ResultsAccuracy and Performance


48

49


M. Albanese and S. Jajodia, “Formation of Awareness,” to appear in Cyber Defense and Situational Awareness, A. Kott, R. Erbacher, C. Wang, eds., Springer Advances in Information Security, 2014.

Lifecycle of Situational Awareness



50

Cyber Defense Process at a Glance


The overall process of cyber defense relies on the combined knowledge of actual attacks and effective defenses It ideally involves every part of the ecosystem

The enterprise, its employees and customers, and other stakeholders It also entails the participation of individuals in every role within

the organization Threat responders, security analysts, technologists, tool developers, users,

policymakers, auditors, etc. Defensive actions are not limited to preventing the initial

compromise They also address detection of already-compromised machines

and prevention or disruption of attackers’ subsequent actions The defenses identified deal with reducing the initial attack

surface Hardening device configurations, addressing long-term threats (such as

APTs), disrupting attackers’ command-and-control of implanted malicious code, and establishing an adaptive defense and response capability


51

Cyber Defense Critical Functions


Learning from attacks Using knowledge of actual attacks that have compromised a system to

provide the foundation to learn from these events and build effective, practical defenses

Prioritization Prioritizing controls that will provide the greatest risk reduction and

protection against current and future threats Metrics

Establishing common metrics to provide a shared language for all parties involved to measure the effectiveness of security controls

Continuous diagnostics and mitigation Carrying out continuous measurement to test and validate the

effectiveness of current security controls, and to help drive the prioritization of the next steps

Automation Automating defenses so that organizations can achieve reliable, scalable,

and continuous monitoring of security relevant events and variables


52

Cyber Defense Roles


Security Analyst Responsible for analyzing and assessing existing vulnerabilities in the

IT infrastructure, and investigating available tools and countermeasures

Security Engineer Responsible for performing security monitoring, detecting security

incidents, and initiating incident response

Security Architect Responsible for designing a security system or its major components

Security Administrator Responsible for managing organization-wide security systems

Security consultant/specialist Responsible for different task related to protecting computers,

networks, software, data, and/or information systems against cyber threats


53

Questions

Internet

Web Server (A)

Mobile App Server (C)

Catalog Server (E)

Order Processing Server (F)

DB Server (G)

Local DB Server (D)

Local DB Server (B)

Current situation. Is there any ongoing attack? If yes, where is the attacker?

Impact. How is the attack impacting the enterprise or mission? Can we asses the damage?

Evolution. How is the situation evolving? Can we track all the steps of an attack?

Behavior. How are the attackers expected to behave? What are their strategies?

Forensics. How did the attacker create the current situation? What was he trying to achieve?

Information. What information sources can we rely upon? Can we assess their quality?

Prediction. Can we predict plausible futures of the current situation?

Scalability. How can we ensure that solutions scale well for large networks?



54

1 – Current Situation


Is there any ongoing attack? If yes, what is the stage of the intrusion and where is the attacker?

Capability Effectively detecting ongoing intrusions, and identifying the

assets that might have been compromised already Input

IDS logs, firewall logs, and data from other security monitoring tools

Output A detailed mapping of current intrusive activities

Lifecycle This type of SA may quickly become obsolete – if not updated

frequently – as the intruder progresses within the system


55

2 – Impact


How is the attack impacting the organization or mission? Can we assess the damage?

Capability Accurately assessing the impact (so far) of ongoing attacks

Input Knowledge of the organization’s assets along with some

measure of each asset’s value Output

An estimate of the damage caused so far by the intrusive activity

Lifecycle This type of SA must be frequently updated to remain

useful, as damage will increase as the attack progresses


56

3 – Evolution


How is the situation evolving? Can we track all the steps of an attack?

Capability Monitoring ongoing attacks, once such attacks have been

detected Input

Situational awareness generated in response to the questions 1 &2

Output A detailed understanding of how the attack is progressing

Lifecycle This capability can help address the limitations on the useful life

of the situational awareness generated in response to questions 1 & 2


57

4 – Behavior


How are the attackers expected to behave? What are their strategies?

Capability Modeling the attacker’s behavior in order to understand its

goals and strategies Input

Past observations and knowledge of organization’s assets Output

A set of formal models (e.g., game theoretic, stochastic) of the attacker’s behavior

Lifecycle The attacker’s behavior may change over time, therefore

models need to adapt to a changing adversarial landscape


58

5 – Forensics


How did the attacker create the current situation? What was he trying to achieve?

Capability Analyzing the logs after the fact and correlating observations

in order to understand how an attack originated and evolved Input

Situational awareness gained is response to question 4 Output

A detailed understanding of the weaknesses and vulnerabilities that made the attack possible

Lifecycle This information can help security engineers and

administrators harden system configurations to prevent similar incidents from happening again


59

6 – Prediction


Can we predict plausible futures of the current situation?

Capability Predicting possible moves an attacker may take in the

future Input

Situational awareness gained in response to questions 1, 3, and 4

Output A set of possible alternative scenarios that may realize

in the future Lifecycle

This type of SA may quickly become obsolete


60

7 – Quality of Information


What information sources can we rely upon? Can we assess their quality?

Capability Assessing the quality of the information sources all other

tasks depend upon Input

Information sources Output

A detailed understanding of how to weight different sources when processing information in response to other questions

Lifecycle Needs to be updated when the information sources

change

61


M. Albanese, H. Cam, and S. Jajodia, “Automated Cyber Situation Awareness Tools for Improving Analyst Performance,” Cybersecurity Systems for Human Cognition Augmentation, Springer 2014.

Impact of SA on Analyst Performance



62

Overview


Automated Cyber Situation Awareness tools and models can enhance performance, cognition and understanding for cyber professionals monitoring complex cyber systems

In most current solutions, human analysts are heavily involved in every phase of the monitoring and response process

Ideally, we should move from a human-in-the loop scenario to a human-on-the loop scenario Human analysts should have the responsibility to oversee the

automated processes and validate the results of automated analysis of monitoring data

To this aim, it is highly desirable to have temporal models such as Petri nets to model and integrate the concurrent operations of cyber-physical systems with the cognitive processing of analyst


63

Petri Net Models for SA


P3P2

P5

P4

T1

P6

P9

P1

P7P8

P10 P11 P12

P13

T2 T3

T4

T5T6 T7

T8 T9

passdeny

T1: Apply firewall ruleset against packetsT2: Alarm probability exceeds thresholdT3: Find new vulnerabilitiesT4: Activated malicious packetsT5: Intrusion attemptsT6: Propagate impact of damagesT7: Patch vulnerabilities, and recover damagesT8: Evict compromised non-recoverable assetsT9: Recover assets fullyT10: Analyst creates a hypothesisT11: Analyst takes an action to verify his/her hypothesisT12: Analyst determines the difference (error) between actual impact

and his/her intended impact of action

P1: Firewall receives packetsP2: Sensor’s measurements are collectedP3: Vulnerability scanner scansP4: Recovery tools runP5: Reject firewall rule-matched packetsP6: Pass rule-nonmatched packetsP7: Attackability conditions of systemP8: Vulnerabilities existP9: Active malicious codesP10: Assets compromisedP11: Impact of assets damages P12: Assets recovered partiallyP13: Available assetsP14: Analyst observes eventsP15: Analyst considers potential actionsP16: Analyst determines impact of actions

P14

P15

P16

T10

T11

T12

Integrating Cybersecurity Operations with Cognitive Analytical Reasoning of Analysts

64


Conclusions



65

Conclusions

The focus in Year 5 was on integration of previous contributions refinement of the CSA framework

definition of metrics attack graph based diversity based

better understanding the overall process lifecycle of CSA role of the analyst

Some of these capabilities will be further refined in a side project


66


Questions?


A Mission-Centric Framework for Cyber Situational Awareness Metrics, Lifecycle of Situational...

Documents

Transcript of A Mission-Centric Framework for Cyber Situational Awareness Metrics, Lifecycle of Situational...