Survivability Metrics- A View From the Trenches Partha Pal Rick Schantz, Franklin Webber, Michael...

Survivability Metrics- A View Survivability Metrics- A View From the TrenchesFrom the Trenches

Partha PalRick Schantz, Franklin Webber, Michael Atighetchi

DSN METRICS CHALLENGE WORKSHOP 2007 June 27, 2007

2

Metrics– What is the Point?Metrics– What is the Point?(From a system designer’s POV)(From a system designer’s POV)

• Evaluating or calibrating emerging technologies or approaches– Is the idea worthwhile?– How good is the realization of the idea?– Does it make any difference?

• Yardsticks differ from a point solution (e.g., a new IDS or a firewall) to overarching techniques/combination of technologies (e.g., survivability architecture)

• Evaluating or calibrating a system– Does the system meet the (specified) requirements?– How does the system compare

• With the undefended version of the same system? • With another defended version of the same system?

– E.g., two competitive designs• With a similar defended system?

– E.g., how does a system compare with a known one (say current high- water mark) in terms of defense and survivability characteristics?

3

ScoringScoring

• What to measure so that conclusion about the “point” can be drawn– Binary properties

• Mission completed?• Data revealed?

– Time intervals• Time taken to achieve some

goal• Effect of attack on “round

trip”– Time based

• Effect of attack on “frequency”

– Count• How many attacks/ attack

steps?• How many failures tolerated?

(prevented? Succeeded?)

• How to measure them

– Experiments• Red Team • Cooperative Red Team• Injection

– Analytic• White boarding• Model based• Logical arguments

– Field Observation• Realistic environment• Ground truth?

Can provide quantitative result

Qualitative (subjective) conclusion can be drawn in any of these methods

Disproving a Hypothesis

4

The DPASA Red Team Exercise ContextThe DPASA Red Team Exercise ContextConfiguration of the defense-enabled system: Nov 2005

Exp3

Exp2

X = 192.168.3.0/24Y = 192.168.4.0/24Z = 192.168.5.0/24

X.101

X.102dpasagw

X.99

SeLinux

WinXP Pro

Solaris 8

RedHat

ADF NIC

Experiment Control Network

Win2000

Bump In Wire w/ADF

Y.126

Y.125

Y.124

Y.123

Y.122

Y.121

Y.114

SPANSPAN

X.126 Q1SM

Q1PS

Q1COR

Q1PSQ

Q1DC

Q1NIDS

Q1AP

X.125

X.124

X.123

X.122

X.121

X.114

X.174 Y.174

Y.173

Y.172

Y.171

Y.170

Y.169

Y.162

SPANSPAN

Q4SM

Q4PS

Q4COR

Q4PSQ

Q4DC

Q4NIDS

Q4AP

X.173

X.172

X.171

X.170

X.169

X.162

X.158 Y.158

Y.157

Y.156

Y.155

Y.154

Y.153

Y.146

SPANSPAN

Q3SM

Q3PS

Q3COR

Q3PSQ

Q3DC

Q3NIDS

Q3AP

X.157

X.156

X.155

X.154

X.153

X.146

X.142 Y.142

Y.141

Y.140

Y.139

Y.138

Y.137

Y.130

SPANSPAN

Q2SM

Q2PS

Q2COR

Q2PSQ

Q2DC

Q2NIDS

Q2AP

X.141

X.140

X.139

X.138

X.137

X.130

SPANSPAN

SPANSPAN

Z.146 Z.162Z.130

SPANSPAN

Z.114

SPANSPAN

AMC CONUS LAN

Wing Ops LAN

PLANNING LAN

ENVIRONMENTAL LAN

MAFX.202

Z.202

CombOpsX.198

Z.198

AODBX.210

Z.210

TARGETX.211

Z.211

CAFX.212

Z.212

TAPX.213

Z.213

swdist aodbsvr tapdb

Hub

ChemHazX.179

Z.179

EDCX.180

Z.180

JEESX.181

Z.181

WxHazX.178

Z.178

Z.200/29

Z.192/29Z.208/28

Z.176/28

QUAD 1 QUAD 2 QUAD 3 QUAD 4Z.128/30 Z.144/30Z.112/30 Z.160/30

HP 2626 Core Switch

VLAN

ANIDSX.203

Z.203

ENIDSX.184

Z.184

WNIDSX.196

Z.196

PNIDSX.215

Z.215

ScorebotX.214

Z.214

VLAN 2Z.113

VLAN 3Z.129

VLAN 4Z.145

VLAN 5Z.161

VLAN 6Z.177 VLAN 7

Z.209 VLAN 9Z.193

VLAN 10Z.201

Cisco 3750 Layer 3 Switch

HP 2626Switch

Exp1X.100

#Bits Mask #Hosts/24 255.255.255.0 254/25 255.255.255.128 126/26 255.255.255.192 62/27 255.255.255.224 30/28 255.255.255.240 14/29 255.255.255.248 6/30 255.255.255.252 2

SPANSPAN

SPANSPAN

SPANSPAN

SPANSPAN

Configuration of the defense-enabled system:

Exp3Exp3

Exp2Exp2

X = 192.168.3.0/24Y = 192.168.4.0/24Z = 192.168.5.0/24

X.101

X.102dpasagwdpasagw

X.99

SeLinuxSeLinux

WinXP ProWinXP Pro

Solaris 8Solaris 8

RedHatRedHat

ADF NICADF NIC

Experiment Control Network

Win2000Win2000

Bump In Wire w/ADFBump In Wire w/ADF

Y.126

Y.125

Y.124

Y.123

Y.122

Y.121

Y.114

SPANSPAN

X.126X.126 Q1SM

Q1PS

Q1COR

Q1PSQ

Q1DC

Q1NIDS

Q1AP

X.125X.125

X.124X.124

X.123X.123

X.122X.122

X.121X.121

X.114X.114

X.174X.174 Y.174

Y.173

Y.172

Y.171

Y.170

Y.169

Y.162

SPANSPAN

Q4SM

Q4PS

Q4COR

Q4PSQ

Q4DC

Q4NIDS

Q4AP

X.173X.173

X.172X.172

X.171X.171

X.170X.170

X.169X.169

X.162X.162

X.158X.158 Y.158

Y.157

Y.156

Y.155

Y.154

Y.153

Y.146

SPANSPAN

Q3SM

Q3PS

Q3COR

Q3PSQ

Q3DC

Q3NIDS

Q3AP

X.157X.157

X.156X.156

X.155X.155

X.154X.154

X.153X.153

X.146X.146

X.142X.142 Y.142

Y.141

Y.140

Y.139

Y.138

Y.137

Y.130

SPANSPAN

Q2SM

Q2PS

Q2COR

Q2PSQ

Q2DC

Q2NIDS

Q2AP

X.141X.141

X.140X.140

X.139X.139

X.138X.138

X.137X.137

X.130X.130

SPANSPAN

SPANSPAN

Z.146 Z.162Z.130

SPANSPAN

Z.114

SPANSPAN

AMC CONUS LAN

Wing Ops LAN

PLANNING LAN

ENVIRONMENTAL LAN

MAFX.202

Z.202

MAFX.202 MAFX.202X.202

Z.202

CombOpsX.198 CombOpsX.198X.198

Z.198

AODBX.210

Z.210

AODBX.210 AODBX.210X.210

Z.210

TARGETX.211

Z.211

TARGETX.211 TARGETX.211X.211

Z.211

CAFX.212

Z.212

CAFX.212 CAFX.212X.212

Z.212

TAPX.213X.213

Z.213

swdist aodbsvr tapdb

Hub

ChemHazX.179

Z.179

ChemHazX.179 ChemHazX.179X.179

Z.179

EDCX.180X.180

Z.180

JEESX.181X.181

Z.181

WxHazX.178

Z.178

WxHazX.178 WxHazX.178X.178

Z.178

Z.200/29

Z.192/29Z.208/28

Z.176/28

QUAD 1 QUAD 2 QUAD 3 QUAD 4Z.128/30 Z.144/30Z.112/30 Z.160/30

HP 2626 Core Switch

VLANVLAN

ANIDSX.203X.203

Z.203

ENIDSX.184X.184

Z.184

WNIDSX.196X.196

Z.196

PNIDSX.215X.215

Z.215

ScorebotX.214

Z.214

ScorebotX.214 ScorebotX.214X.214

Z.214

VLAN 2Z.113

VLAN 3Z.129

VLAN 4Z.145

VLAN 5Z.161

VLAN 6Z.177 VLAN 7

Z.209 VLAN 9Z.193

VLAN 10Z.201

Cisco 3750 Layer 3 Switch

HP 2626Switch

Exp1Exp1X.100

#Bits Mask #Hosts/24 255.255.255.0 254/25 255.255.255.128 126/26 255.255.255.192 62/27 255.255.255.224 30/28 255.255.255.240 14/29 255.255.255.248 6/30 255.255.255.252 2

#Bits Mask #Hosts/24 255.255.255.0 254/25 255.255.255.128 126/26 255.255.255.192 62/27 255.255.255.224 30/28 255.255.255.240 14/29 255.255.255.248 6/30 255.255.255.252 2

SPANSPANSPANSPAN

SPANSPANSPANSPAN

SPANSPANSPANSPAN

SPANSPANSPANSPAN

PIX VPN Router

5

The DPASA ExerciseThe DPASA Exercise

• Had both “technology/approach evaluation” and “survivable system evaluation” flavors– Demonstrate that survivability architecture can be realized– Achieve specific requirements

• Results must not be tainted by assumptions– Red team was nearly unrestricted

• Blue needed to focus on many technical details that typically get ignored in a research project– up to date patches, removing unused services, tight policy definitions– Blue team’s task became more complex

• White team was heavily concerned with measurability– Interpretation of requirements

• One of the highest priorities for the Red teams was to jeopardize the mission within 12 hrs

6

ConsequencesConsequences

• As a result– The internals of the defended system were not

tested– Red team attack focus remained within the

external periphery of the system

• Brute force slow down• Zero day exploit to partition COTS VPN

• Scores like “number of successful server operations” do not reflect the reality

Both targeted the COTS routers that connect a local enclave to the IP Network that connects the enclaves

7

Some resultsSome results

• Sample Requirement– 95% prevention of attacker objectives for 12 hours

• Interpretation– defended system’s ability to prevent the attacker objectives being met on a per-attack basis. If an

attack achieved its stated objective, it was considered successful irrespective of the JBI mission outcome

• Scoring rules– (p/T*100), where p is the number of attacks that failed to produce the a-priori defined success

indicators, and T is total number of attacks • Scores

– Undefended: 0 out of 17– Defended Run 1: 6 out of 9– Defended Run 2: 0 out of 1

• Overall impression of the runs– Undefended Run 1: Undefended system was easily taken over– Defended Run 1: Despite delays mission was completed correctly– Defended Run 2: Mission stalled within 1 hr of start

• Discussion:– Consistent with overall impression (for this requirement)– but what does it say about the quality of

defense of the system?– For another requirement the score was not so consistent (undefended system scored higher)– Despite these difficulties, this was a significant attempt in quantifying survivability

8

RemedyRemedy

• Cooperative Red team– Red starts the system with pre-positioned attack code

that they can control• Developed in cooperation with a “traitor” blue team member

– Blue cannot act unless they face a symptom• They do not know where the attacker is, how many machines

are they on or what /how many attacks were going to run

– End result of a run:• Compressed scenario• Defender’s sole objective is to succeed the mission using the

defenses integrated in the survivability architecture

9

Some resultsSome results

75% of such runs resulted in successful mission completion. Required experts to operate the gauges and levers of the defense mechanisms integrated in the architecture

Scenario completed after 3 hrs227m11/1820

Scenario is not completed165m11/1719

Scenario complete55m11/1718



Aborted11/1515


Scenario completed60m11/1413













OutcomeRun LengthDateRun

Scenario completed after 3 hrs227m11/1820





Aborted11/1515















OutcomeRun LengthDateRun

10

ConclusionConclusion

• Cooperative Red team exercises stressed the architecture more than the formal red team exercises– Expert participation– next issue for survivability research needs to deal

with • In line with the notion of self-aware/self-regenerative systems

• Examples of confidence boosting conclusions (and related metrics) from the system developer/owner’s POV– There is only a small # of attack paths to achieve the high valued targets – The attacker faces multiple # of defenses to achieve any of his

objectives• More diverse defenses are better

– Attacker work factor increased (or decreased) by x %• We note that measuring the AWF has typically been a problem

– The amount of access and privilege needed by the attacker to succeed has increased (or decreased)

• Along with an assessment of whether that is reasonable or practical

Survivability Metrics- A View From the Trenches Partha Pal Rick Schantz, Franklin Webber, Michael...

Documents

Transcript of Survivability Metrics- A View From the Trenches Partha Pal Rick Schantz, Franklin Webber, Michael...