Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio...

IRRIIS- FP6-2005–IST-4

EC - LOGO

Introduction to IRRIIS testing platform

IRRIIS MIT Conference ROME 8 February 2007

Claudio Balducelli

IRRIIS

Summary

Design a testing environment for MIT Modelling and running attack and fault

behaviours Testing strategies for MIT components Proposed test-bed configuration Conclusions

IRRIIS

Target Infrastructures

Models

Vulnerabilities of the Target

Infrastructures

Fault/attack Scenarios Generation

Models of faults & attacksUse domain

knowledge

Considervulnerabilities

Design a testing environment for MIT

IRRIIS

Meaning of attacks and faults

Attacks: A disturbance of the LCCI generated by eventscoming from outside the LCCI

Faults: A disturbance of the LCCI generated by eventscoming from the components that are part of theLCCI

Definition of the meaning of attacks and faults

IRRIIS

Meaning of attacks and faults

Attacks:

Natural disaster (earthquake, flood, etc)

Premeditated terrorist attack

Cyber attacks (cyber-intrusion)

Operator errors

………….….

Faults:

Physical component failure (aging, stress, etc.)

Software component failure (bug, wrong istal. etc)

Wrong component activation

………….….

IRRIIS

Normal behavior & fault behavior in SimCIP

Activationevent

t1

Start Comp. 1

Comp. 1

Start Comp. 2

Comp. 2End

Start Comp. 3

t2 Comp. 3End

Activationevent

Normal behavior consists in an initial state and a sequence of events represented in form of a petri net oriented graph

IRRIIS

Initiatingevent

t1

Failure of Comp. 1

t2

Failure ofComp. 2

t3

RestartComp. 1

t4 t5

Loss ofservice 2


Loss ofService 1

Fault behavior may be represented in a similar way

Fault eventsIn LCCI-1

Failure ofComp. 2

t6

Failure of Comp. 1

t7

Fault eventsIn LCCI-2

IRRIIS

For a certain LCCI normal behaviors are well known and their number is

limited the number and the combinations of fault behaviors are

very high and not always known in advance how to design fault behaviors? how to select fault behaviors? utilisation of a model based on attack/fault trees seem

useful to formalise and manage the knowledge needed to generate attack/fault behaviour


IRRIIS

G0

A1 A2 A2

The root of the tree (G) represents an event that could significantly harm the infrastructure’s mission.

The terminal leafs (A) of the tree represent the actions to execute for reaching the high level goals

Every path in the attack tree represents a unique type of attack

Goal G0AND A1 A2 A3

Goal G0OR A1 A2 A3

The attack trees could be visualized also in textual form

G0

A1 A2 A3

Every node could be decomposed inside lower level nodes using <AND>, <XOR> and <OR> decomposition types

AND

OR

Modelling attack knowledge attack/fault trees

IRRIIS

G0

S1 A2 S2

A3 A4 A5 A6

The tree generate the following two attack patterns

<A3, A2, A5, A6>

<A4, A2, A5, A6>

The “terminal leafs” of the tree (A1..An) represent the actions steps needed to execute the attack

The “intermediate nodes” (S1..Sn) represent the steps in which a decision has to be taken

The attack tree generates attack patterns (attack behaviors), composed by sequences of actions.

Attack goal


IRRIIS

TE

S1 C2 S3

C11 C12 C31 C32

The tree generate the following two

fault patterns

<C11, C2, C31, C32>

<C12, C1, C31, C32>

The “terminal leafs” of the tree (C..) represent the elementary failures of the single components of LCCI.

The “intermediate nodes” (S…) represent failures of subsystems or services for which the components contribute

The fault tree generates fault patterns (fault behaviors), composed by sequences of elementary failures.

Top event Fault trees


IRRIIS

And gate

Or gateOR gate

AND gate

Example of attack tree to model an attack in a local area network (tree structure)

The reference model take in account the:Fault Tree Handbook ofUS Nuclear Regulatory Commission

IRRIIS

And gate

Or gateOR gate

AND gate


Verify theaccessibility to a subnet

IRRIIS

And gate

Or gateOR gate

AND gate


Discover the target locations & addresses

IRRIIS

And gate

Or gateOR gate

AND gate


Make sniffing activity or damages

IRRIIS

And gate

Or gateOR gate

AND gate


Generated behaviours table------------------------------------------------------------------------------------------------Attack behaviour 0 <A1, A2, A4, A5, A6, A7, A8 >

Attack behaviour 1 <A1, A2, A4, A5, A6, A7, A9>



Attack behaviour 4 <A1, A3, A4, A5, A6, A7, A8 >



Attack behaviour 7 <A1, A3, A4, A5, A6, A7, A11> ------------------------------------------------------------------------------------------------

IRRIIS

Example of attack tree to model an attack:associating difficulties to the actions

OR gate

AND gate

0.8

0.9 0.2

0.950.95

0.95

0.30.6

0.2

0.80.8

0.0 = maximum difficulty1.0 = minimum difficulty

Generated behaviours table ordered by action difficulties------------------------------------------------------------------------------------------------Attack behaviour 0 <A1, A2, A4, A5, A6, A7, A8 > with 0,39 of difficulty

Attack behaviour 2 <A1, A2, A4, A5, A6, A7, A10> with 0,24 of difficulty

Attack behaviour 1 <A1, A2, A4, A5, A6, A7, A9> with 0.12 of difficulty


Attack behaviour 4 <A1, A3, A4, A5, A6, A7, A8 > with 0.08 of difficulty



Attack behaviour 7 <A1, A3, A4, A5, A6, A7, A11> with 0.02 of difficulty------------------------------------------------------------------------------------------------

IRRIIS

Macro scenarios:

how to compose attack and fault trees

Attack tree

Fault tree

Attack tree

Wait for malfunction

IRRIIS

Composite attack and fault behavior

t1

BasicAction 0

t2

BasicAction 2

Final Action 0

t3 t4

Final Action 1

Networkmalfunction

BasicEvent 0

Attackbehavior

Attackbehavior

Faultbehavior

Attack escalation

IRRIIS

Testing MIT components (meaning)

REQUIREMENTS:

Risk Ass. (1) - The Risk estimator assessment of cascading and escalating effects shall be performed in near real-time.Risk Ass. (2) - The Risk estimator assessment of cascading and escalating effects shall be performed in a predictive way.Risk Ass. (3) - The Risk estimator shall estimate immediate risk to the LCCI.Risk Ass. (4) - The Risk estimator may estimate expected risk to the LCCI.Risk Ass. (5) - The Risk estimator shall estimate potential cascading effects.

Objective of the TEST: validate the requirements

Risk Ass. (1) - OK

Risk Ass. (2) - OK

Risk Ass. (3) - OK

Risk Ass. (4) - NOT OK

Risk Ass. (5) - NOT OK

IRRIIS

Testing MIT components (meaning)

One of the main objective of the MIT components test inside SimCIP simulated environment is the evaluation of the rate of false/true alarms.

The second is to evaluate how much the rate of false alarms may be acceptable for the LCCIs operators

IRRIIS

Detecting interdependency alarms

Real statesPredicted states

Alarm No Alarm

P(Alarm) A B P(No Alarm) C D

A = Number of alarm states correctly predicted

D = Number of no alarm states correctly predicted

B = Number of no alarm states predicted as true (FALSE POSITIVE)

C = Number of alarm states not predicted (FALSE NEGATIVE)

The goal is: max(A + D), min(B + C)

Evaluation Table

IRRIIS


Real statesPredicted states

Alarm No Alarm

P(Alarm) A B P(No Alarm) C D

Fn = C / ( C + D )Observed False Negative Ratio (FNR)

Fp = B / ( A + B )Observed False Positive Ratio (FPR)

IRRIIS

Be not afraid to discover false alarms during the tests. This is the tests objective!!

In many cases false alarms could be simply reduced tuning the “sensitivity” level of a MIT component.

To evaluate true/false alarms ratio is not sufficient a single attack/fault behavior. Many alternative behaviors are needed!!

Logging facilities are very important during experimentations, are the tests results must be archived and documented


IRRIIS

Proposed testing strategy

IRRIIS testing operator

Attack/Fault tree editor

Design or modify a scenario

tree

GA

S1 A2 S2

A3 A4 A5 A6

<A3, A2, A5, A6>

<A4, A2, A5, A6>

<A3, A2, A5, A6>

<A4, A2, A5, A6>

Fault behaviors

editor

Generate & modify fault behaviors,

insert timing information etc

Documentation console

View logsEdit test

documents

Logs

Test documen

ts

Fault behavior execution

Execute behavious,

sets monitors

Attacks/faultsexecution in

SimCIP

Test designentry point

Test designexit point

Testdesign

IRRIIS





tree

GA

S1 A2 S2

A3 A4 A5 A6

<A3, A2, A5, A6>

<A4, A2, A5, A6>

<A3, A2, A5, A6>

<A4, A2, A5, A6>

Fault behaviors

editor




View logsEdit test

documents

Logs

Test documen

ts


Execute behavious,

sets monitors


SimCIP

Test execution entry point

Test execution exit point

Fasttesting

IRRIIS





tree

GA

S1 A2 S2

A3 A4 A5 A6

<A3, A2, A5, A6>

<A4, A2, A5, A6>

<A3, A2, A5, A6>

<A4, A2, A5, A6>

Fault behaviors

editor




View logsEdit test

documents

Logs

Test documen

ts


Execute behavious,

sets monitors


SimCIP

Test entry point

Test exit point

Exhaustivetesting

IRRIIS

Physical TESTBED Configurations

LAMPSSys RTI

GUI Logger

To

ol 1

Electricity

SimulatorLCCI

Data

Com

Simulator

To

ol 2

Agent / Scenario

Behaviours

An

alysis 1

An

alysis 2Fault /

Attack

Tool

MITA

nalysis 3

SimCIP Architecture

IRRIIS

Physical TESTBED Configurations

GUILogger

LAMPSSys RTI

Agent / Scenario

Behaviours

Electricity

Simulator

Com

Simulator

LCCI Electricity

Data Base

Tool 1

Tool 2

Analysis 1, 2, 3 ..

LCCI Telecom

Data Base

Simple SimCIPconfiguration

IRRIIS

Physical TESTBED Configuration

LAMPSSys RTI

Agent / Scenario

Behaviours

Electricity

Simulator

Com

Simulator

LCCI Electricity

Data Base

Fault /Attack

Tool

Tool 1

Tool 2

Analysis 1, 2, 3 ..

LCCI Telecom

Data Base

SimCIPfor testing

attacks and faults without MIT

GUILogger

IRRIIS


GUILogger

LAMPSSys RTI

Agent / Scenario

Behaviours

Electricity

Simulator

Com

Simulator

LCCI Electricity

Data Base

LCCI Telecom

Data Base

MT communicationElectricity Add-on Telecom Add-on

SimCIPfor testing MIT with normal

behaviors(detect false positive alarms)

IRRIIS


GUILogger

LAMPSSys RTI

Agent / Scenario

Behaviours

Electricity

Simulator

Com

Simulator

LCCI Electricity

Data Base

LCCI Telecom

Data Base

MT communicationElectricity Add-on Telecom Add-on

SimCIP for testing MIT in presence of

attacks/faults (detect false negative alarms)

Fault /Attack

Tool

Tool 1

Tool 2

Analysis 1, 2, 3 ..

IRRIIS

Conclusions

Testing of MIT components will be a continuous and iterative process

It is necessary to distinguish between the fast tests of the more simple requirements and the exhaustive test process aimed to evaluate the MIT efficiency in detecting interdependency alarms

Test designing, reports logging/archiving in a standard way and with the support of a common tool, will help to have sets of comparable tests also if produced in different SimCIP installations.

The testing environment will be one of the major a research product of the project, where experimentation may continue also after the end of the project.

QUESTIONS?

Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio...

Documents

Transcript of Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio...