In: Colbert, E. and Kott, A. (eds.),  "Cyber Security of Industrial Control Systems, Including SCADA Systems,"  

Springer, NY, 2016 

Security Metrics in Industrial Control Systems 

ZacharyA.Collier1,MaheshPanwar2,AlexanderA.Ganin3,AlexKott4,IgorLinkov1*

1USArmyEngineerResearch&DevelopmentCenter,Concord,MA,USA

2ContractortoUSArmyEngineerResearch&DevelopmentCenter,Concord,MA,USA

3UniversityofVirginia,Charlottesville,VA,USA

4USArmyResearchLaboratory,Adelphi,MD,USA

*CorrespondingAuthor,Igor.Linkov@usace.army.mil

1.1 Introduction 

Risk–thetopicofthepreviouschapter–isthebestknownandperhapsthebeststudiedexamplewithinamuchbroaderclassofcybersecuritymetrics.However,riskisnottheonlypossiblecybersecuritymetric.OthermetricssuchasresiliencecanexistandcouldbepotentiallyveryvaluabletodefendersofICSsystems.

Often,metricsaredefinedasmeasurablepropertiesofasystemthatquantifythedegreetowhichobjectivesofthesystemareachieved.MetricscanprovidecyberdefendersofanICSwithcriticalinsightsregardingthesystem.Metricsaregenerallyacquiredbyanalyzingrelevantattributesofthatsystem.

Intermsofcybersecuritymetrics,ICSstendtohaveuniquefeatures:inmanycases,thesesystemsareoldertechnologiesthatweredesignedforfunctionalityratherthansecurity.Theyarealsoextremelydiversesystemsthathavedifferentrequirementsandobjectives.Therefore,metricsforICSsmustbetailoredtoadiversegroupofsystemswithmanyfeaturesandperformmanydifferentfunctions.

Inthischapter,wefirstoutlinethegeneraltheoryofperformancemetrics,andhighlightexamplesfromthecybersecuritydomainandICSinparticular.Wethenfocusonaparticularexampleofaclassofmetricsthatisdifferentfromtheonewehaveconsideredinearlierchapters.Insteadofrisk,hereweconsidermetricsofresilience.ResilienceisdefinedbytheNationalAcademyofSciences

2

(2012)as“Theabilitytoprepareandplanfor,absorb,recoverfrom,ormoresuccessfullyadapttoactualorpotentialadverseevents”.

Thischapterpresentstwoapproachesforthegenerationofmetricsbasedontheconceptofresilienceusingamatrix‐basedapproachandanetwork‐basedapproach.Finally,adiscussionofthebenefitsanddrawbacksofdifferentmethodsispresentedalongwithaprocessandtipsintendedtoaidindevisingeffectivemetrics.

1.2 Motivation 

UnderPresidentGeorgeW.Bush,theDepartmentofEnergyissuedbestpracticesforimprovedindustrialcontrolsystem(ICS)security(USDepartmentofEnergy,2002).Someoftheseincludetakingstepssuchas"disconnectunnecessaryconnectionstotheSCADAnetwork","establisharigorous,ongoingriskmanagementprocess"and"clearlyidentifycybersecurityrequirements."Additionally,ExecutiveOrder13636,signedbyPresidentBarackObamain2013,broughtforththeissueofcybersecurityandresilience,and

proposedthedevelopmentofarisk‐based“CybersecurityFramework”(EO13636,2013).TheframeworkwaspresentedbytheNationalInstituteofStandardsandTechnology(NIST)andoffersorganizationsguidanceonimplementingcybersecuritymeasures.

Despiteexistingguidelinesandframeworks,designingandmanagingforsecurityincyber‐enabledsystemsremainsdifficult.Thisisinlargepartduetothechallengesassociatedwiththemeasurementofsecurity.PfleegerandCunningham(2010)outlineninereasonswhymeasuringsecurityisadifficulttaskasitrelatestocybersecurityingeneral,butallofwhichalsoapplytothesecurityofICSdomain(Table1).

PfleegerandCunningham(2010)notethatonewaytoovercomethesechallengesistothoughtfullydevelopaclearsetofsecuritymetrics.Unfortunately,thislackofmetricshappenstobeoneofthegreatestbarrierstosuccessinimplementingICSsecurity.WhenICSswerefirstimplemented,"networksecuritywashardlyevenaconcern"(Igureetal,2006).Althougheffortsarebeingmadetodraftandenactcybersecuritymeasures,thatgaphasyettobeclosed,evenatatimeofgreaterrisk.

3

Table1:ChallengeswithCybersecurityMeasurement(adaptedfromPfleeger&Cunningham,2010)

Challenge DescriptionWecan’ttestallsecurityrequirements

Itisnotpossibletoknowallpossibleconfigurationsandstatesofthesystem,intendedusesandunintendedmisusesfromusers,etc.

Environment,abstraction,andcontextaffectsecurity

Systemsarebuilttoevolveastheyprocessnewinformation,andnotallsystemchangesarederivedfrommalicioussources

Measurementandsecurityinteract

Knowledgeaboutasystem’svulnerabilitiesandsafeguardscanaffectthetypesoffurthersecuritymeasuresimplemented,aswellasmodifytherisksthatusersarewillingtotake

Nosystemstandsalone Systemsarenetworkedtointeractwithothercybersystemsandassets

Securityismultidimensional,emergent,andirreducible

Securityexistsatmultiplelevelsofsystemabstraction,andthesecurityofthewholesystemcannotbedeterminedfromthesecurityofthesumofitsparts

Theadversarychangestheenvironment

Developinganaccuratethreatlandscapeisdifficultduetoadaptiveadversarieswhocontinuallydevelopnovelattacks

Measurementisbothanexpectationandanorganizationalobjective

Differentorganizationswithdifferentmissionsandpreferencesplacedifferingvaluesonthebenefitsofsecurity

We’reoveroptimistic Userstendtounderestimatethelikelihoodthattheirsystemcouldbethetargetofattack

Weperceivegainsdifferentlythanlosses

Biasesininterpretingexpectedgainsandlossesbasedonproblemframingtendtoaffectrisktoleranceanddecisionmakingunderuncertaintyinpredictablebutirrationalways

1.3 Background on Resilience Metrics 

1.3.1 What Makes a Good Metric? 

Accordingtothemanagementadage,“whatgetsmeasuredgetsdone”.Assuch,well‐developedmetricscanassistanorganizationinreachingitsstrategicgoals(Marr,2010).

Reichertetal.(2007)definemetricsas“measurablepropertiesthatquantifythedegreeto

whichobjectiveshavebeenachieved”.Metricsprovidevitalinformationpertainingtoagivensystem,andaregenerallyacquiredbywayofanalyzingrelevantattributesofthatsystem.Someresearchersandpractitionersmakeadistinctionbetweenameasureanda

4

metric(Blacketal.,2008,Linkovetal.,2013a),whereasothersmayrefertothemasperformancemeasures(Neelyetal.,1997),keyperformanceindicators(Marr,2010)orstrategicmeasures(Allen,2011).Forthepurposesofthischapter,thesearereferredtogenerallyasmetrics.

Whenusedefficiently,metricscanhelptoclarifyone’sunderstandingoftheprocessesofaparticularareaofasystem,andfromthere,provideinformationforexternalreviewandassisttowardsfurtherimprovement,amongotheroutputs(Marr,2010).Thiscanbedonebyestablishingbenchmarksforagivenmetric,wherethresholdsorrangescanbeestablished(Blacketal.,2008).Benchmarks,orstandards,helpformthebasisfordecisionmakingandtakingcorrectiveaction(Williamson,2006).

Acriticalelementinelicitingameaningfulmetricistogathertherelevantinformation

aboutone’ssystemandtoalignthatmetricwithmeasurablegoalsandstrategicobjectiveswhichliewithinthescopeofagivenprojectorthedomainofaparticularorganizationalstructure(Beasleyetal.2010,Neelyetal.1997).Thereisalsotheissueofscaleandadaptability.Smallerorganizationmayhavemetricsdealingwithrudimentarysecuritymeasures,butastheygrowlarger,thesemeasuresmayneedtobescaledappropriatelytodealwiththesecurityneededforalargerorganization(Blacketal.,2008).

Therearekeyelementsthatcontributetoproducingasuccessfulmetric.Metricsshouldbeactionable:theyarenotsimplyaboutmeasuringnumerousattributesofaproject;merelygatheringinformationwithoutagoalinmindwillnotprovideadiscerniblesolution(Marr,2010).Suchinformationinandofitselfwouldnotbesubstantialenoughtobeconsideredametric.Gatheringrelevantmetricsrequiresdelvingdeeperintotheissuesfacedbyagivensystemandaskingpertinentquestionswhichcanleadtoactionableimprovement.These

includequestionssuchas“Doesitlinktostrategy?Canitbequantified?Doesitdrivethe

rightbehavior?”(Eckerson,2009).Fromthese,onecanobtainmetricswhichcaninturninformactionableresults.Table2summarizesthedesirablecharacteristicsofmetricsingeneralterms,andapplytoalltypesofsystemsincludingICSs.

5

Table2:CharacteristicsofGoodMetrics(adaptedfromMcKayetal.2012,KeeneyandGregory2005)Characteristic DescriptionRelevant MetricsaredirectlylinkedtodecisionmakinggoalsandobjectivesUnambiguous ConsequencesofalternativescanbeclearlymeasuredbymetricsDirect MetricsclearlyaddressanddescribeconsequencesofinterestOperational DataexistandareavailableforthemetricofinterestUnderstandable MetricscanbeunderstoodandcommunicatedeasilyComprehensive Thesetofmetricsaddressacompletesuiteofgoalsandconsequences

Metricsmaybedescribedasnatural,constructed,orproxy.Naturalmetricsdirectly

describeanobjectiveinunitsthatarestraightforward(e.g.,dollarsasametricfor“costs

associatedwithICSdowntime”).Constructedmetricsmaybeusedwhennaturalmetricsdonotexist(e.g.,scalesfrom1to10whereeachnumbercorrespondstoadefinedlevelofICSperformance),andusuallyincorporateexpertjudgment.Proxymetricscanbeusedtoindirectlymeasureanobjective(e.g.,thenumberofuserswithcertainadministrativeprivilegesasaproxyforaccess)(McKayetal.2012,KeeneyandGregory2005).

Therearedifferenttypesofinformationthatmetricsgaugeandtheprojectteamhastheresponsibilityofappropriatelyselectingandevaluatingthem.Thesecanbeseparatedintoquantitative,semi‐quantitativeandqualitativeapproaches.Quantitativemetricshavemeasurable,numericalvaluesattachedtothem.Semi‐quantitativemetricsarenotstrictlyquantifiablebutcanbecategorized.Qualitativemetricsprovidenon‐numericinformation,forexampleintheformofaesthetics.

1.3.2 Metrics for IT Systems  

AsdescribedaboveinTable1,cybersystemsprovideuniquechallenges.Inparticular,thecyberdomainextendsbeyondjusttheimmediatesystemandrequiresaholisticviewpoint,withmanydifferenttechnicalandhumanfactorstobeaccountedfor(Collieretal.,2014).Threatstothesystemarealsoconstantlyevolvingandgrowinginsophistication,andasaresult,thereisahighdegreeofadaptabilityrequiredinordertoremaincurrent.Duetotheconstantlyevolvingthreatspace,thereisoftenlittlehistoricaldataforpotentialthreats(Collieretal.,2014).

6

Withcybermetrics,asignificantnumberofthemainissuesaretailoredtowardssecurityandresilience.TheDefenseScienceBoard(2013)arguesthateffectivecybermetricsshouldbebroadenoughtofitdifferenttypesofsystems,yetalsobepreciseenoughtodialdownintothespecificsofagivensystem.Thefollowingaresomeexamplesofcybersecuritymetricscurrentlyinuse.

TheCommonVulnerabilityScoringSystem(CVSS)wasintroducedtoprovidevariousorganizationswithactionableinformationinregardstoassessingITvulnerabilities(Melletal.,2007).CVSSgroupstheirmetricsintothreecategories,namelyBase,Temporal,andEnvironmentalmetrics.AfewofthesesecuritymetricsincludeCollateralDamagePotential,TargetDistribution,ReportConfidence,Exploitability,AccessComplexity,AccessVector,Authentication,IntegrityImpact,AvailabilityImpact,andConfidentialityImpact(Melletal.,2007).Therearegeneralscoringtipsforthewaythatvulnerabilitiesareassessed;vulnerabilitiesarenotscoredbasedoninteractionswithothervulnerabilities,rather,theyarescoredindependently.Themainmeasureofvulnerabilityisitsimpactonthekeyservice.Vulnerabilitiesarescoredaccordingtocommonlyusedprivileges,whichmightbeadefaultsettingincertainsituations.Ifavulnerabilitycanbeexploitedbymultipleexploits,itisscoredwiththeexploitthatwillpresentthemaximumimpact(Mell,etal.,2007).CVSSallowsvulnerabilityscorestobestandardized,andBasemetricsare

normalizedonascaleof0–10.TheycanbeoptionallyrefinedbyincludingvaluesfromTemporalandEnvironmentalmetrics.

TheCenterforInternetSecurity(CIS)hasalsoestablishedmetricsfororganizationstouse(CIS,2010).CIShasdividedtheirmetricsintosixcriticalbusinessfunctions.TheseareIncidentManagement,VulnerabilityManagement,PatchManagement,ConfigurationManagement,ChangeManagementandApplicationSecurity.Italsorecognizeshierarchiesandinterdependenciesofmetrics,forinstancecitingmanagementmetricsasbeingofprimaryimportancetoanorganization,whilenotingthatsomeofthosemetricsmaydependonthepriorimplementationoftechnicalmetrics(CIS,2010).SomeofthemetricsincludeCostofIncidentsandPatchPolicyCompliance.CostofIncidentsreferstoanumber

ofpotentiallosses,suchascustomerlistsortradesecretsundera“directloss”anda“cost

ofrestitution”,forexampleintheeventthatfinesareleviedduetoanincident.Thisismeasuredbythesummationofthenumericalvaluesofallthecostsassociatedwiththemetric.ExamplesrelatingtosecurityincludeMeanTimetoIncidentDiscovery,MeanTimeBetweenSecurityIncidentsandMeanTimetoIncidentRecovery(CIS,2010).Foranexampleofmeasurement,MeanTimetoIncidentDiscoverymeasuresthesummationofthe

7

timebetweenincidentsanddiscoveriesofincidents,dividedbytotalnumberofincidentsrecoveredduringthosetimeframes(CIS,2010).

TheCybersecurityFrameworkdevelopedbyNISTstemmingfromEO13636wasreleasedinFebruary2014(NIST2014a).ThefinalCybersecurityFrameworkconsistsofa

FrameworkCore,whichpresentsasetoffive“concurrentandcontinuousFunctions–

Identify,Protect,Detect,Respond,Recover”(NIST2014).Thesefunctionsarethe“high‐

level,strategicviewofthelifecycleofanorganization’smanagementofcybersecurityrisk,”whichfeaturesubsequentcategoriesandsubcategoriesforthefunctions,relatingtooutcomesandactivities(NIST2014).Forexample,theRespondfunctionconsistsoffivecategories,amongwhichincludesMitigation.Mitigationisthenfurthersubdividedintometricsrelatedtocontaininganderadicatingincidents.TheFrameworkCoreisusedasa

scorecardofprogress–thecurrentguidancecallsforfirstdevelopinganorganization’s

CurrentProfile,whichconsistsofassignedscoresbasedontheorganization’sperformanceineachofthecategoriesandsubcategories.ThisCurrentProfileisthencomparedtoaTargetProfile,representingthedesiredstateoftheorganizationineachofthesamecategoriesandsubcategories.Theshortfallsbetweentheseprofilescanbeviewedasgaps

inanorganization’scyber‐riskmanagementcapabilitieswhichcaninformprioritizationofcorrectivemeasures(NIST2014;Collieretal.2014).

TheSoftwareEngineeringInstitute(SEI)atCarnegieMellonUniversitydevelopedaframeworkforassessingoperationalresiliencewhichfeaturesasetofTopTenStrategicMeasures,whichaimtobemappeddowntothelevelofspecificProcessAreameasures(AllenandCurtis,2011).UndertheheadingofHigh‐ValueServicesandAssets,oneofthemeasuresisrelatedtothepercentageofhigh‐valueservicesthatdonotsatisfytheirassignedresiliencerequirements(AllenandCurtis,2011).TheSEIframeworkalsocontainsalargeamountofresiliencemeasures,spanning26differentProcessAreas.Forexample,undertheProcessAreaofEnvironmentalControl,therearemeasuressuchasPercentageofFacilityAssetsthathavebeenInventoried,ElapsedTimeSincetheFacilityAssetInventorywasReviewed,andElapsedTimeSinceRiskAssessmentofFacilityAssets

Performed(AllenandCurtis,2011),wheretheterm“assets”appliestohigh‐valueservices.Thesearepresentedinatablewithtraceability,assigninganidentificationnumbertoeachmetricalongwiththeirapplicabilitytogoalswithintheProcessAreas.

8

MITREproposedaframeworkentitledCyberResiliencyEngineeringFramework,which,

amongitsgoalsaimsto“motivateandcharacterizecyberresiliencymetrics”(Bodeau2011).TheframeworkcontainsfourCyberResiliencyGoals:Anticipate,Withstand,Recover,andEvolve.Thereareatotalofeightobjectiveswhichareasubsetofthegoals.ForexampleAnticipatehasthreeobjectives:Predict,Prevent,andPrepare(Bodeau,2011).Thishierarchycanbeusedtoinformandcategorizetheappropriateresiliencemetrics.Thesearemeanttobeperformedsimultaneously,andbeararesemblancetotheNISTframeworkmentionedearlier.

1.3.3 Metrics for ICS Networks 

Theabovemetricsweredevelopedfor“cyber”systemsgenerallyspeaking,notspecificallyforICSs,althoughtheycanbetailoredwithICSsinmind.ICSsinparticularareauniquecase;inmanysituations,thesesystemshaveoldermodels,andweredesignedforfunctionalityratherthansecurity(USDepartmentofEnergy,2002).Theyconstituteadiversegroupofsystemsthathavedifferentrequirementsfortheirvariousoperations(Pollet,2002).

SpecificallyasitrelatestoICSs,time,safetyandcontinuationofservicesareofgreatimportance,sincemanysystemsareinapositionwhereafailurecanresultinathreattohumanlives,environmentalsafety,orproductionoutput(Stouffer,2011).Sincetheserisksaredifferentthanthosefacedbyinformationtechnology(IT)systems,differentprioritiesarealsonecessary.Examplesofsomeuniqueconsiderationsincomparisontocybersecurityincludethelongerlifespanofsystemcomponents,physicallydifficulttoreachcomponents,andcontinuousavailabilityrequirements(Stouffer,2011).Additionally,thesesystemstypicallyoperateinseparatefieldsthancybersecurity,suchasinthegasandelectricindustries,andsometricsmustbeadaptedtofitthesedifferentorganizationalstructures(McIntyreetal.,2007).CriticalinfrastructuresarecommonforICSs,andasa

result“downtimeandhaltingofproductionareconsideredunacceptable”(McIntyreetal.,2007).

9

Stoufferetal.(2011)comparethedifferencesbetweeninformationtechnology(IT)system

andICSs,focusingonthesafety‐criticalnatureofmanyICSnetworks.Forexample,“high

delayandjittermaybeacceptable”asaperformancerequirementforITsystems,whereasforICSs,itmaynotbeacceptable(Stouffer,2011).Thisisduetothefactthatthereisatime‐criticalnaturetoICSs,whereasforITsystemsthereishighthroughput,allowingfor

somejitter(Stouffer,2011).Similarly,forIT,“systemsaredesignedforusewithtypical

operatingsystems”andforICSs,thereare“differingandpossiblyproprietaryoperating

systems,oftenwithoutsecuritycapabilitiesbuiltin”.Therearealsoavailabilityrequirements,inthatsometimesanITstrategymayrequirerestartingorrebootingaprocess,somethingwhich,forICSprocesses,requiresmorecarefulplanningasunexpectedoutagesandquicklystoppingandstartingasystemarenotacceptablesolutions(Stouffer,2011).Withthesekeydifferencesbetweenthetwodomains,therearevaryinglevelsofadaptationneededinordertobegintheprocessofsecuringICSnetworks.

TheUSNationalSecurityAgency(NSA)draftedaframeworkforICSnetworks,focusingonpotentialimpactandlossrelatingtoanetworkcompromise(NSA,2010).Theysuggested

assigninglossmetricsincorporatingNIST’sframework:compromisespertainingtoConfidentiality,IntegrityandAvailabilityforeachnetworkasset(NSA,2010).A

Confidentialitycompromiseisdefinedasan“unauthorizedreleaseortheftofsensitive

information”e.g.theftofpasswords(NSA,2010).AnIntegritycompromiseisdefinedasan

“unauthorizedalterationormanipulationofdata”,e.g.manipulationofbillingdata(NSA,

2010).AnAvailabilitycompromiseisdefinedasa“lossofaccesstotheprimarymissionofa

networkedasset”e.g.deletionofimportantdatafromadatabase(NSA,2010).Thesemayalsobestreamlinedintoonemetric,usingthehighestvalue(e.g.ofLow,ModerateorHigh)amongthethreeareas.

Theassignmentofathreatmetricsateachpotentialattackvectorwassuggested,butspecificexampleswerenotprovided.Fivethreatsourceswereidentified:Insiders,TerroristsorActivists,HackersorCyber‐Criminals,Nation/StateSponsoredCyber‐WarfareandCompetitors(NSA,2010).Bothlossandthreatmetricscanberatedonaconstructedscale(Low,ModerateorHigh)andgivenanumericratingonasetscale.Itwasmentionedthattheimportantconsiderationistohaveascale,andthatthenumberofgraduationsin

10

thescaleisnotimportant,solongastheconstructedscaleremainconsistent(e.g.apotentialforlossoflifewillrankasHigh)(NSA,2010).Combiningresultsofmetricswasalsodiscussedasapossibility.Asanexample,foragivenpointinthenetwork,aLossMetricisassignedascoreofHighontheconstructedscale(3)andaThreatmetricatthatsamenetworkpointisratedatModerate(2).Fromthis,onecanarriveatacompositepriorityvalue,whichissimplythesumofthosetwoscores.Othersuchpointscanbeevaluatedandthenprioritizedandranked(NSA,2010).Thescoringmethodologyisabasicexample,(andnottheonlymethod‐weighingmetricswaslistedasapossibility(NSA,2010))andmorerobustmethodscanbedevised.

BoyerandMcQueen(2008)devisedasetofideal‐basedtechnicalmetricsforcontrolsystems.Theyexaminedsevensecuritydimensionsandpresentanideal,orbestcasescenario,foreachofthem.TheidealsareSecurityGroupKnowledge,AttackGroupKnowledge,Access,Vulnerabilities,DamagePotential,Detection,andRecovery.FortheAccessdimension,theidealstatesthatthesystemisinaccessibletoattackgroups.ThesecuritydimensionofVulnerabilitieshasanidealstatingthatthesystemhasnovulnerabilities(BoyerandMcQueen,2008).Bytheverynatureofanideal,thesemaybeimpossibletoachieveandmaintainintherealworld.Butfromthem,metricsweredevisedthatcouldbestrepresenttherealizationoftheseideals.Underthevulnerabilitydimension,

themetricVulnerabilityExposureisdefinedas“thesumofknownandunpatched

vulnerabilities,eachmultipliedbytheirexposuretimeinterval.”Itwassuggestedthatthismetriccouldbebrokendownintoseparatemetricsfordifferentvulnerabilitycategories,aswellasincludingaprioritizationofvulnerabilities,citingCVSS.UndertheAccessdimension,thereisthemetricRootPrivilegeCount,whichisthecountofallpersonnelwith

keyprivileges,arguinginfavoroftheprincipleofleastprivilege,whichstatesthat“everyprogramandeveryprivilegeduserofthesystemshouldoperateusingtheleastamountof

privilegenecessarytocompletethejob”(Saltzer,1974).Thislogicalorderingofmetricswithinthescopeofidealscanbeofvaluetothosewishingtodevisetheirownsetofmetrics.

Theideal‐basedmetrics(BoyerandMcQueen,2008)alsoacknowledgethephysicalspaceofICSnetworks.ThemetricRogueChangeDays,whichisthenumberofchangestothesystemmultipliedbythenumberofdaysundetected,includesProgrammableLogicControllersandHuman‐MachineInterfacesandotherICSrelatedsystems.ComponentTestCount,ametricmeasuringthenumberofcontrolsystemcomponentswhichhavenotbeentestedisasimplemeasure,butofsignificanceduetonumerouscomponentsinuseinanICSsystem.

11

Withintheideals,themetricofAttackSurface(definedbyManadhataandWing(2011)as

“thesubsetofthesystem’sresources(methods,channels,anddata)potentiallyusedin

attacksonthesystem”)wasdeterminedtonotbedevelopedenoughforrealworlduse.

BoyerandMcQueenfurtherarguethat“acrediblequantitativemeasureofsecurityriskis

notcurrentlyfeasible”(BoyerandMcQueen2008).Butwiththeinclusionofatheoreticalmetric,andaframeworkforsecurity,thisdemonstratesaforwardthinkingattitudethatcanbebuiltuponbythoseaimingtoestablishtheirownsecurityprotocols.ThisrepresentsimportantfutureworkfortheICSandsecuritycommunities.ComparisonsbetweentheNSAapproachandtheapproachoutlinedbyBoyerandMcQueenarepresentedinTable3.

Table3:ComparisonbetweenICSMetrics

NationalSecurityAgency(2010) BoyerandMcQueen(2008)

Focus LossandThreatfocusedMetrics(p.10,15)

Quantitativetechnicalmetrics(p.1),idealbased:attemptedtohavemetricsthatcouldstrivetowardidealscenarioswithinsevensecurityareas

Amount Threelossmetrics(pernetworkedasset),oneThreatmetric(perpotentialattackvector)

13totalmetrics(suggestedtotal:lessthan20)

AppliedorTheoretical

Suggestsdeployablemetrics Discussesbothdeployableandtheoreticalmetrics(p.10,11)

QuantitativeorQualitative

Semi‐qualitative(suggestsHigh,Medium,Low,withallowancefornumericattachmenttothesevalues)

Doesnotfocusonqualitativemetrics(p.1),butonquantitativemetrics

CombinationofMetrics

Presentsmethodtocombineresultsofmetricscoresforranking

Nocombinationofmetrics

ConsequenceConsiderations

LossMetricsarerelatedtoConfidentiality,Integrity,Availability

AcknowledgesthepurposeofsecurityisprotectionofConfidentiality,IntegrityandAvailability(p.4)

12

ComplementaryresearchtometricsdevelopmentintheICSrealmiscurrentlybeingconducted.OnesucheffortistodevelopastandardizedtaxonomyofcyberattacksonSCADAsystems(Zhuetal.,2011).AcommonlanguagefordescribingattacksacrosssystemscanfacilitatethedevelopmentoffurtherthreatandvulnerabilitymetricsforICSs.Inaddition,thedevelopmentofanationaltestbedforSCADAsystemsisbeingdevelopedbytheDepartmentofEnergywhichwillenablethemodelingandsimulationofvariousthreatandvulnerabilityscenarios,whichwillallowresearcherstodevelopabetterunderstandingofwhatmetricsmayormaynotbeusefulinmonitoringandmanagementofthesesystems(USDepartmentofEnergy,2009).Anotherdevelopmentrelatedtometricsresearchistheinvestigationoftradeoffsbetweencertaincriticalmetrics.Oneexampleisbetweenoptimizingsystemperformancewithsystemsecurity,whereadditionalsecuritymeasuresmayresultinreducedperformance.Zeng&Chow(2012),developedanalgorithmictechniquetodeterminetheoptimaltradeoffbetweenthesetwometrics,andthemethodcanbeextendedtotradeoffsbetweenothermetricsaswell.

1.4 Approaches for ICS Metrics 

Whilevariousframeworksandsetsofmetricsexist,suchastheonesmentionedintheprevioussection,itcanbedifficultformanagersandsystemoperatorstodecidewhethertoadoptormodifyanexistingset,ortocreateanentirelynewsetofmetrics.Balancingthetradeoffsbetweengeneralizablemetricsandspecificsystem‐levelandcomponent‐levelmetricscanbechallenging(DefenseScienceBoard,2013).Thefollowingapproachesprovideastructuredwaytothinkaboutdevelopingmetrics,allowinguserstoleverageexistingmetricsbutalsoidentifygapswherenewmetricsmayneedtobecreated.Theuseofsuchstructuredandformalizedprocessesrequiresthethoughtfulanalysisofthesystemsbeingmeasured,butalsohowtheyrelatetothebroaderorganizationalcontext,suchasgoals,constraints,anddecisions(Marr,2010).Moreover,thedevelopmentofastandardizedlistofquestionsortopicshelpstosimplifytheprocessofdesigningametric.Thedevelopmentofmetricsshouldbeasmoothprocess,andsuchalistcanprovideinsight

intothe“behavioralimplications”ofthegivenmetrics(Neelyetal.1997).

1.4.1 Cyber Resilience Matrix Example 

ThefirstmethodisbasedontheworkofLinkovetal.(2013a).Unliketraditionalrisk‐basedapproaches,thisapproachtakesaresilience‐centrictheme.Muchhasbeenwrittenelsewhereontherelativemeritsofaresilience‐focusedapproach(seeLinkovetal.,2013b,2014;Collieretal.2014;Roegeetal.2014;DiMaseetal.2015),butweshallbriefly

13

summarizetheargumenthere.TraditionalriskassessmentbasedonthetripletformulationproposedbyKaplanandGarrick(1981)becomesdifficulttoimplementinthecybersecuritycontextduetotheinabilitytoframeandevaluatemultipledynamicthreatscenarios,quantifyvulnerabilityagainstadaptiveadversaries,andestimatethelong‐termandwidelydistributedconsequencesofasuccessfulattack.Insteadofmerelyhardeningthesystemagainstpotentialknownthreatsinarisk‐basedapproach,thesystemcanbemanagedfromtheperspectiveofresilience,whichincludestheabilityofoneormore

criticalsystemfunctionalitiestoquickly“bounceback”toacceptablelevelsofperformance.Asaresult,aresilientsystemcanwithstandandrecoverfromawidearrayofknownandunknownthreatsthroughprocessesoffeedback,adaptation,andlearning.

Followingthisthoughtprocess,Linkovetal.(2013a)establishedamatrix‐basedmethod.Ononeaxis,thestepsoftheeventmanagementcycleidentifiedasnecessaryforresiliencebytheNationalAcademyofSciences(2012)arelisted,andincludePlan/Prepare,Absorb,Recover,andAdapt.Notethattheabilitytoplan/prepareisrelevantbeforeanadverseevent,andtheothercapabilitiesarerelevantafterdisruption.OntheotheraxisarelistedthefourdomainsinwhichcomplexsystemsexistasidentifiedbyAlberts(2002),andincludePhysical,Information,Cognitive,andSocialdomains.ThePhysicaldomainreferstothephysicalresourcesandcapabilitiesofthesystem.TheInformationdomainreferstotheinformationanddatathatcharacterizethePhysicaldomain.TheCognitivedomaindescribestheuseoftheotherdomainsfordecisionmaking.Finally,theSocialdomainreferstotheorganizationalstructureandcommunicationsystemsfortransmittinginformationandmakingdecisions(Alberts2002).

Together,theseaxesformasetofcellsthatidentifyareaswhereactionscanbetakenin

specificdomainstoenhancethesystem’soverallabilitytoplanfor,andabsorb,recover,andadaptto,variousthreatsordisruptions(Figure1).Eachcellisdesignedtoanswerthe

question:“Howisthesystem’sabilityto[plan/preparefor,absorb,recoverfrom,adaptto]

acyberdisruptionimplementedinthe[physical,information,cognitive,social]domain?”(Linkovetal.2013a).

14

Figure1:GenericResilienceMatrix

Aresultingsetof49metricsareproducedthatspanthevariouscellsofthematrix,andselectedmetricsareshowninTable4(seeLinkovetal.2013aforthecompletelist).Metricsaredrawnfromseveralsourcesandaremeanttobegeneralandnotnecessarily

comprehensive.Forexample,underAdaptandInformation,ametricstates“document

timebetweenproblemanddiscovery,discoveryandrecovery,”whichhasaparalleltothe

MeanTimetoIncidentDiscoverywithinSEI’sguidance.ThemetricsunderPlanandInformation,relatedtoidentifyinginternalandexternalsystemdependenciescanbecomparedtotheTemporalMetricofAccessComplexityfromCVSS,whichrelatestohoweasilyavulnerabilitycanbeexploited.ThemetricunderPrepareandSocialpresentsa

simpleyetimportantmessagethatholdstrueinalloftheframeworks:“establishacyber‐

awareculture.”

TheresiliencematrixapproachdescribedinLinkovetal.(2013a)hasseveralstrengthsinthatthemethodisrelativelysimpletouseandoncemetricshavebeengenerated,itcanserveasaplatformforamulti‐criteriadecisionaid(Collier&Linkov,2014).Ithasthepotentialtoserveasascorecardinordertocapturequalitativeinformationabouta

system’sresilience,andaidmanagersandtechnicalexpertsinidentifyinggapsinthe

system’ssecurity.However,theresiliencematrixdoesnotcapturetheexplicittemporalnatureofresilience(i.e.,mappingthecriticalfunctionalityovertime)orexplicitlymodelthesystemitself.Inthisregard,itcanbeviewedasahighlevelmanagementtoolthatcanbeusedtoidentifyasnapshotwheremoredetailedanalysesandmodelingcouldpotentiallybecarriedout.

Plan & 

Prepare

Absorb Recover Adapt

Physical

Information

Cognitive

Social

15

Table4:SelectedCybersecurityMetricsDerivedfromtheResilienceMatrix(adaptedfromLinkovetal.,2013a).

Plan/Prepare Absorb Recover AdaptPhysical Implement

controls/sensorsforcriticalassetsandservices

Useredundantassetstocontinueservice

Investigateandrepairmalfunctioningcontrolsorsensors

Reviewassetandserviceconfigurationinresponsetorecentevent

Information Prepareplansforstorageandcontainmentofclassifiedorsensitiveinformation

Effectivelyandefficientlytransmitrelevantdatatoresponsiblestakeholders/decisionmakers

Reviewandcomparesystemsbeforeandaftertheevent

Documenttimebetweenproblemanddiscovery,discoveryandrecovery

Cognitive Understandperformancetrade‐offsoforganizationalgoals

Focuseffortonidentifiedcriticalassetsandservices

Establishdecisionmakingprotocolsoraidstoselectrecoveryoptions

Reviewmanagementresponseanddecisionmakingprocesses

Social Establishacyber‐awareculture

Locateandcontactidentifiedexpertsandresponsiblepersonnel

Determineliabilityfortheorganization

Evaluateemployeesresponsetoeventinordertodeterminepreparednessandcommunicationseffectiveness

1.4.2 Network Simulation Example 

Thesecondmethodisbasedonmodelingofcomplexcyberandothersystemsasinterconnectednetworks,whereafailureinonesectorcancascadetootherdependentnetworksandassets(Vespignani,2010).ThisisareasonableassumptionforICSnetworks;forexample,adisruptionoftheelectricalgridcandirectlyimpactdependentsectorssuchasthenetworkcontrollingICSdevicesleadingtoacascadeoffailuresasitisbelievedtohavehappenedduringtheItalianblackoutin2003(Buldyrevetal.,2010).Thusthe

16

assessmentofthesecurityofasingleICSnetworkshouldbeviewedinthecontextofalargernetworkofinterdependentsystems.

Ganinetal.(2015)tookthisnetwork‐orientedviewindevelopingamethodologytoquantitativelyassesstheresilience(andthussecurity)ofnetworkedcybersystems.TheybuiltupontheNationalAcademyofSciences(2012)definitionofresilienceasasystempropertythatisinherentlytiedtoitsabilitytoplanfor,absorb,recoverfrom,andadapttoadverseevents.Inordertocapturethestateofthesystemtheauthorsproposetousetheconceptofcriticalfunctionalitydefinedasatime‐specificperformancefunctionofthe

systemconsideredandderivedbasedonthestakeholder’sinput.Forinstanceinthenetworkofpowerplants,thecriticalfunctionalitymightrepresentthetotaloperationalcapacity.Inthenetworkofcomputersitmightrepresentthefractionofserversandservicesavailable.Valuesofcriticalfunctionalityarerealnumbersfrom0to1.Otherkey

elementstoquantifyresiliencearethenetworkedsystem’stopologyanddynamics;therangeofpossibleadverseevents(forexample,acertaindamagetonodesofthenetwork);andthecontroltimeTC(thatisthetimerangeoverwhichtheperformanceofthesystemisevaluated).Thenthedependencyofthecriticalfunctionality(averagedoveralladverseevents)overtimeisbuilt.Ganinetal.(2015)refertothisdependencyastheresilienceprofile.Asitistypicallycomputationallyprohibitiveornotpossibleatall(incaseof

continuousvariablesdefiningnodes’states)toconsiderallthewaysanadverseeventcanhappen,itissuggestedtoutilizeasimulationbasedapproachwithMonte‐Carlosampling.

Givenitsprofileinnormalizedtime(wheretimeTCistakentobe1),theresilienceofthenetworkcanbemeasuredastheareaunderthecurve(yellowregioninFigure2).Thisallowsmappingoftheresiliencetorealvaluesrangingbetween0and1.

Anotherimportantpropertyofthesystemisobtainedbyfindingtheminimumoftheaveragecriticalfunctionality.SomeresearchersrefertothisvalueasrobustnessM

(Cimellaroetal.,2010),whileLinkovetal.(2014)notethat1–Mcorrespondstothemeasureofrisk.

17

Figure2:Ageneralizedresilienceprofile,whereasystem’sresilienceisequaltotheareabelowthe

criticalfunctionalitycurve(adaptedfromGaninetal.,2015).

IntheirpaperGaninetal.(2015)illustratedtheapproachonadirectedacyclicgraph.Eachlevelinthisgraphrepresentsasetofnodesfromcertaininfrastructuresystem(e.g.electricalgrid,computersetc.).Nodesofdifferentlevelsareconnectedbydirectedlinksrepresentingadependencyofthedestinationnodeonthesourcenode.Inthesimplestcaseanodeinacertainlevelrequiressupply(oradependencylink)fromanodeineachoftheupperlevelsanddoesnotdependonanynodesinthelowerlevels.Otherparametersofthe

modelincludenoderecoverytime(TR)–ameasureofhowquicklyanodecanreturntoan

activestateafterit’sbeeninactivatedasaresultofanadverseevent;redundancy(pm)–theprobabilitycontrollingthenumberofadditionalpotentialsupplylinksfromupperlevelstolowerlevels;andswitchingprobability(ps),controllingeaseofreplacementofadisruptedsupplylinkwithapotentialsupplylink.Theseparameterscouldbeextendedtoothersituationstoinformhowasystemmaydisplayresilientbehavior,andthusincreasingthesecurityofthesystemasawhole.

Theauthorsfoundthatthereisstrongsynergybetweenpmandps;increasingbothfactorstogetherproducesarapidincreaseinresilience,butincreasingonlyoneortheothervariablewillcausetheresiliencemetrictoplateau.Resilienceisstronglyaffectedbythetemporalswitchingtimefactor,TR.Thistemporalfactordeterminesthecharacteristicsoftherecoveryphaseandhasagreaterimpactonthecalculatedresiliencethandoesthepotentialincreaseinredundancy.Thisisparticularlytruewhentheswitchingprobabilitypsislow.Animportantlongtermchallengeistomodeladaptation,which,accordingtotheNationalAcademyofSciences,ispartoftheresponsecyclethatfollowsrestorationandincludesallactivitiesthatenablethesystemtobetterresistsimilaradverseeventsinthefuture.

18

Ganinetal.(2015)notethatthemainadvantagesoftheapproachincludeitsapplicabilitytoanysystemthatcanberepresentedasasetofnetworks.Alsoboththeresilienceandtherobustnessofasystemaremetricizedusingarealvalueinrangebetween0and1(where1correspondstotheperfectresilienceorrobustness)makingcomparisonofresilienceofdifferentsystemseasy.Ontheotherhandmappingtheresiliencepropertyofasystemtoa

singlevaluenecessarilyshadowssomesystem’simportantcharacteristics(forinstance,therateofrecovery).Theresilienceprofilecouldbeusedasamoreholisticrepresentationof

thesystem’sresiliencenotingthateveninthatcaseonlytheaveragevalueofcriticalfunctionality(ateachtimestep)istakenintoaccount.Tofullydescribeasystemoneshouldconsiderthedistributionofthevalueofcriticalfunctionality(ateachtimestep)fordifferentinitialadverseevents.Finally,itisnotpossibletosimulatealladverseeventsfromtherangeusedtoestimateresilienceandtheapproachisMonte‐Carlobased.Itmeansthatinorderfortheresultstobereliablethenumberofsimulationsistypicallyrequiredtobeveryhigh.

1.5 Tips for Generating Metrics  

1.5.1 Generalized Metric Development Process 

ThefollowingprocesstowardsthedevelopmentofmetricsisadaptedbyMcKayetal.2012.

1. ObjectiveSetting:Articulateclear,specificgoals.Thisshouldbedoneinastructuredmanner.GregoryandKeeney(2002)outlineastructuredapproachtodothis.

a. Writedownalloftheconcernsthattheprojectteamfeelsisrelevant.b. Convertthoseconcernsintosuccinctverb‐objectgoals(e.g.,minimizedowntime).c. Next,theseshouldbeorganized,oftenhierarchically,separatinggoalswhich

representmeansfromthosewhichrepresentends.d. Finally,reviewandclarificationshouldbeconductedwiththeprojectteam.This

maybeaniterativeprocess.2. DevelopMetrics:Oncetheobjectivesareclearlyarticulatedandorganized,metricscanbe

formallydeveloped.a. Thefirststepistoselectabroadsetofmetrics,whichmaybeselectedfromexisting

listsorguidelines,orcreatedbyaprojectteamorsubjectmatterexpertsfortheparticularpurposeathand.ThisstepiswheretheResilienceMatrixcouldfacilitatemetricdevelopment.

b. Next,thissetofmetricsshouldbeevaluatedandscreenedtodeterminewhetheritmeetstheprojectobjectivesandthedegreetowhichthemetricsmeetthedesirablequalitiesofmetrics,explainedearlierinthischapter.Atthisstage,remainingmetricscanbeprioritized.

19

c. Finallytheremainingmetricsshouldbedocumented,includingassumptionsandlimitations,andothersupportinginformation.

3. CombinationandComparison:Amethodshouldbedevelopedforhowthemetricswillultimatelybeusedtosupportdecisionmakinganddriveaction.Somemethodsinclude:

a. NarrativeDescription:Simpletechniqueswheretrade‐offsmaybesimplesuchaslistingevidenceorbestprofessionaljudgement.

b. ArithmeticCombination:Simplemathematicaltechniquesforcombiningdissimilarmetricssuchassimpleaggregationofmetricswithsimilarunits(e.g.,cost),convertingtosimilarunits(e.g.,monetization),ornormalizingtoasimilarscale(e.g.,0to1).

c. Multi‐CriteriaDecisionAnalysis:Amethodforweightingandscoringdissimilardecisioncriteriabasedontheirrelativeimportanceandperformancewithrespecttoanobjective.

d. InterdependentCombination:Forsystemsthatarecomplex,usuallyinvolvingintricateinternalrelationships,moreintensivemodelingeffortsmaybenecessary,suchasBayesiannetworksorothercomplexsystemsmodelingtechniques.

Theabove‐mentionedprocess,alongwithasolidmetricdevelopmentprocess,cangreatlyaidindevisingeffectivemetrics.Oftenitisnecessarytodevelopaconceptualmodelofthesysteminordertoidentifythefunctionalrelationshipsandcriticalelementsandprocesseswithinasystem.ThiscanbedoneusingaNetworkScienceapproachdescribedabove.

1.5.2 Best Practices in Metric Development and Validation 

Validationofmetricsisanoftenoverlookedaspectofthemetricdevelopmentprocess.Neelyetal.(1997)providesomequestionstoaskregardingwhethertheoutputfromthemetricsisappropriate,specificallywhetherthemetricshaveaspecificpurpose,arebasedonanexplicitformulaand/ordatasource,andareobjectiveandnotbasedsolelyonopinion(Neelyetal.,1997).Similarly,Eckerson(2009)laysoutaseriesofquestionsthatcanserveasaqualitycheckondevelopedmetrics,toensurethattheyareofhighquality:

•Doesitlinktostrategy?•Canitbequantified?•Doesitdrivetherightbehavior?•Isitunderstandable?•Isitactionable?•Doesthedataexist?

Regardingthenumberofmetricsnecessary,itisn’tnecessarilythequantityofmetricsthatconstituteasuccessfulimplementation,butwhetherthesemetricsarecollectivelycomprehensiveenoughtoaddresseverythingdeemedimportant(McKayetal.2012).Eckerson(2009)recommendsthatasetofmetricsbesparse,sincewithalimitednumber

20

ofmetricsitiseasiertoanalyzehowmetric‐levelchangesdrivetheperformanceinthesystem,aswellasthepracticalfactthatgathering,synthesizing,andpresentingmultipledatastreamsoftentakesquitesometime.Moregranular,process‐levelmetricsmaystillberequiredhowever,andEckerson(2009)proposesaMAD(monitor,analyze,drill)frameworkforpresentingdifferentlevelsofresolutiontodifferentusersofthatinformation.

Anotherongoingelementofvalidationistraceability,asevidencedintheframeworkpresentedbyNeelyetal.(1997),whichincludesalistofinformation(knownastheperformancemeasurerecordsheet)suchashowoftendataistobecollected,andby

whom,aswellasimportantquestionssuchas“whoactsonthedata?”and“whatdothey

do?”.Ifthesequestionsareconsideredandansweredastheneedarises,itisknownwhoisresponsibleformakingthemeasurementandwhatactionsaretobetakenasaresult.Thiscanrevealinsightintothemetricandhowtheyaremeasuredandbeingutilized,notjustforthecurrentprojectbutforfuturereference.Anitemonthelistaskswhatthemetric

“relatesto.”Thiscanassistinenteringthemindsetofapproachingmetricswithaninterconnectedandgoal‐orientedviewpoint.

Othervalidation‐relatedeffortsincludestandardizingmethodsforICSmetricdevelopmentandimplementation,aswellasinstitutionalizingaclearmeanstointegratemetricswithdecisionanalytictoolstosupporttheriskmanagementprocess.Finally,giventhedynamicnatureofcyberthreats,periodicreviewandupdatingofICSmetricsshouldbeconductedtokeepabreastofthelatestdevelopmentsinthefield.

1.6 Conclusions 

Despiteexistingguidelinesandframeworks,designingandmanagingforsecurityincyber‐enabledsystemsremainsdifficult.Thisisinlargepartduetothechallengesassociatedwiththemeasurementofsecurity.Acriticalelementinelicitingameaningfulmetricisingatheringtherelevantinformationaboutone’ssystemandaligningthatmetricwithmeasurablegoalsandstrategicobjectives.ForICSs,time,safetyandcontinuationofservicesfactorconsiderablyintooverallgoals,sincemanysystemsareinapositionwhereafailurecanresultinathreattohumanlives,environmentalsafety,orproductionoutput.Oftenitisnecessarytodevelopaconceptualmodelofthesystemordevelopastandardizedlistofquestionsortopicshelpstoidentifycriticalprocesselements,thefunctionalrelationshipsandcriticalelementsandprocesseswithinasystem.Inthischapter,wediscussindetailtwoapproachesforthegenerationofbroadlyapplicablesecurityandresiliencemetricsandtheirintegrationtoquantifysystemresilience.Thefirstmethodisasemi‐quantitativeapproachinwhichthestagesoftheeventmanagementcycle(plan/prepare,

21

absorb,recover,andadapt)areappliedacrossfourrelevantdomains(physical,information,cognitive,social),formingamatrixofpotentialsecuritymetrics.SecondisaquantitativeapproachbasedonNetworkScience,inwhichfeaturessuchasnetworktopologiescanbemodeledtoassessthemagnitudeandresponsivenessofthecriticalfunctionalitiesofnetworkedsystems.Validationofmetricsisanoftenoverlookedaspectofthemetricdevelopmentprocess;howeveraseriesofquestionscanserveasaqualitycheckondevelopedmetrics,toensurethattheyareofhighquality.

PermissionwasgrantedbytheUSACEChiefofEngineerstopublishthismaterial.TheviewsandopinionsexpressedinthispaperarethoseoftheindividualauthorsandnotthoseoftheUSArmy,orothersponsororganizations.

1.7 References 

Alberts,D.S.(2002)Informationagetransformation,gettingtoa21stcenturymilitary.Washington,

DC:DODCommandandControlResearchProgram,Retrievedfromhttp://www.dtic.mil/get‐tr‐doc/pdf?AD=ADA457904.

Allen,J.,&Curtis,P.(2011)MeasuresforManagingOperationalResilience.Pittsburgh,PA:Software

EngineeringInstitute,CarnegieMellonUniversity,Retrievedfromhttp://www.sei.cmu.edu/reports/11tr019.pdf

Beasley,M.S.,Branson,B.C.,&Hancock,B.V.(2010)BuildingKeyRiskIndicatorstoStrengthen

EnterpriseRiskManagement.Durham,NC:TheCommitteeofSponsoringOrganizationsoftheTreadwayCommission(COSO).

Bodeau,D.,&Graubart,R.(2011)MITRECyberResiliencyEngineeringFramework,MTR110237.

Bedford,MA:MITRECorporation,Retrievedfromhttp://www.mitre.org/sites/default/files/pdf/11_4436.pdf

Boyer,W.,&McQueen,M.(2008)IdealBasedCyberSecurityTechnicalMetricsforControlSystems.

Retrievedfromhttp://www.if.uidaho.edu/~amm/faculty/Ideal%20Based%20Cyber%20Security%20Technical%20Metrics%20for%20Control%20Systems.pdf

Black,P.,Scarfone,K.,&Souppaya,M.(2008)Cybersecuritymetricsandmeasures.In:Voeller,J.G.

(Ed.),HandbookofScienceandTechnologyforHomelandSecurity,Vol5,Hoboken,NJ:JohnWileyandSons,Inc.

Buldyrev,S.V.,Parshani,R.,Paul,G.,Stanley,H.E.,&Havlin,S.(2010)Catastrophiccascadeof

failuresininterdependentnetworks.Nature464,1025‐1028CIS(TheCenterforInternetSecurity)(2010)TheCISSecurityMetricsv1.1.0.EastGreenbush,NY:

TheCenterforInternetSecurity,Retrievedfromhttps://benchmarks.cisecurity.org/tools2/metrics/CIS_Security_Metrics_v1.1.0.pdf

Cimellaro,G.P.,Reinhorn,A.M.,&Bruneau,M.(2010)Frameworkforanalyticalquantificationof

disasterresilience.EngineeringStructures,32,3639–3649

22

Collier,Z.A.,Linkov,I.,DiMase,D.,Walters,S.,Tehranipoor,M.,&Lambert,J.H.(2014)Cybersecurity

standards:managingriskandcreatingresilience.Computer,47(9),70‐76Collier,Z.A.,&Linkov,I.(2014)Decisionmakingforresiliencewithinthecontextofnetworkcentric

operations.19thInternationalCommandandControlResearchandTechnologySymposium(ICCRTS),16‐19June,Alexandria,VA,USA.

DefenseScienceBoard(2013)Taskforcereport:resilientmilitarysystemsandtheadvancedcyber

threat.Washington,DC:OfficeoftheUnderSecretaryofDefenseforAcquisition,Technology,andLogistics,Retrievedfromhttp://www.acq.osd.mil/dsb/reports/ResilientMilitarySystems.CyberThreat.pdf

DiMase,D.,Collier,Z.A.,Heffner,K.,&Linkov,I.(2015)Systemsengineeringframeworkforcyber

physicalsecurityandresilience.EnvironmentSystems&Decisions,35(2),291‐300.Eckerson,W.W.(2009)PerformanceManagementStrategies:HowtoCreateandDeployEffective

Metrics.TDWIBestPracticesReport.Renton,WA:TheDataWarehousingInstitute.Retrievedfromhttps://tdwi.org/research/2009/01/bpr‐1q‐performance‐management‐strategies.aspx

ExecutiveOrderNo.13636,ImprovingCriticalInfrastructureCybersecurity,Retrievedfrom

http://www.gpo.gov/fdsys/pkg/FR‐2013‐02‐19/pdf/2013‐03915.pdfGanin,A.A.,Massaro,E.,Gutfraind,A.,Steen,N.,Keisler,J.M.,Kott,A.,Mangoubi,R.,&Linkov,I.

(2015)Resilientcomplexsystemsandnetworks:concepts,design,andanalysis.NatureScientificReports,submitted

Gregory,R.S.,&Keeney,R.L.(2002)Makingsmarterenvironmentalmanagementdecisions.Journal

oftheAmericanWaterResourcesAssociation.38(6):1601‐1612.Kaplan,S.,&Garrick,B.J.(1981)OntheQuantitativeDefinitionofRisk.RiskAnalysis,1(1),11–27Keeney,R.L.,&Gregory,R.S.(2005)Selectingattributestomeasuretheachievementofobjectives.

OperationsResearch53(1),1‐11Igure,V.,Laughter,S.,&Williams,R.(2006)SecurityIssuesinSCADANetworks.Computersand

Society,25(7),498‐506Linkov,I.,Eisenberg,D.A.,Plourde,K.,Seager,T.P.,Allen,J.,&Kott,A.(2013a)Resiliencemetricsfor

cybersystems.EnvironmentSystems&Decisions,33(4),471‐476Linkov,I.,Eisenberg,D.A.,Bates,M.E.,Chang,D.,Convertino,M.,Allen,J.H.,Flynn,S.E.,&Seager,T.P.

(2013b)Measurableresilienceforactionablepolicy.EnvironmentalScience&Technology,47(18),10108–10110

Linkov,I.,Bridges,T.,Creutzig,F.,Decker,J.,Fox‐Lent,C.,Kröger,W.,Lambert,J.H.,Levermann,A.,

Montreuil,B.,Nathwani,J.,Nyer,R.,Renn,O.,Scharte,B.,Scheffler,A.,Schreurs,M.,&Thiel‐Clemen,T.(2014)ChangingtheResilienceParadigm.NatureClimateChange,4,407–409

23

Manadhata,P.K.,&Wing,J.M.(2011)AnAttackSurfaceMetric.IEEETransactionsOnSoftwareEngineering,37(3),371–386.

Marr,B.(2010)HowtodesignKeyPerformanceIndicators.MiltonKeynes,UnitedKingdom:The

AdvancedPerformanceInstitute.Retrievedfromwww.ap‐institute.comMcIntyre,A.,Becker,B.,&Halbgewachs,R.(2007).SecurityMetricsforProcessControlSystems.

SAND2007‐2070P.Albuquerque,NM:SandiaNationalLaboratories,U.S.DepartmentofEnergy.

McKay,S.K.,Linkov,I.,Fischenich,J.C.,Miller,S.J.,&ValverdeJr,L.J.(2012)EcosystemRestoration

ObjectivesandMetrics,ERDCTN‐EMRRP‐EBA‐12‐16.Vicksburg,MS:U.S.ArmyEngineerResearchandDevelopmentCenter

Mell,P.,Scarfone,K.,&Romanosky,S.(2007)ACompleteGuidetotheCommonVulnerabilityScoring

SystemVersion2.0.Morrisville,NC:ForumforIncidentResponseandSecurityTeams.Retrievedfromhttps://www.first.org/cvss/cvss‐guide.pdf

NationalSecurityAgency(NSA).(2010).AFrameworkforAssessingandImprovingtheSecurity

PostureofIndustrialControlSystems(ICS).Retrievedfromhttps://www.nsa.gov/ia/_files/ics/ics_fact_sheet.pdf

Neely,A.,Richards,H.,Mills,J.,Platts,K.,&Bourne,M.(1997)Designingperformancemeasures:a

structuredapproach.InternationalJournalofOperations&ProductionManagement,17(11),1131‐1152

NIST(2014)FrameworkforImprovingCriticalInfrastructureCyberSecurity.Version1.0.

Gaithersburg,MD:NationalInstituteofStandardsandTechnology.Retrievedfromhttp://www.nist.gov/cyberframework/upload/cybersecurity‐framework‐021214‐final.pdfPfleeger,S.L.,&Cunningham,R.K.(2010)Whymeasuringsecurityishard.IEEESecurity&Privacy,

8(4),46‐54Pollet,J.(2002)DevelopingaSolidSCADAStrategy.Sicon/02–SensorsforIndustryConference.

Houston,Texas,USA.19‐21,November2002.Reichert,P.,Borsuk,M.,Hostmann,M.,Schweizer,S.,Sporri,C.,Tockner,K.,&Truffer,B.(2007)

Conceptsofdecisionsupportforriverrehabilitation.EnvironmentalModelingandSoftware22:188‐201.

Roege,P.E.,Collier,Z.A.,Mancillas,J.,McDonagh,J.A.,&Linkov,I.(2014)Metricsforenergy

resilience.EnergyPolicy,72(1),249–256Saltzer,J.H.(1974)ProtectionandthecontrolofinformationsharinginMultics.Communicationsof

theACM,17(7):388‐402.Stouffer,K.,Falco,J.,&Scarfone,K.(2011)GuidetoIndustrialControlSystems(ICS)Security.Special

Publication800‐82.Gaithersburg,MD:NationalInstituteofStandards.Retrievedfromhttp://csrc.nist.gov/publications/nistpubs/800‐82/SP800‐82‐final.pdf

24

USDepartmentofEnergy(2002)21StepstoImproveCyberSecurityofSCADANetworks.Washington,DC:USDepartmentofEnergy.Retrievedfromhttp://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/21_Steps_‐_SCADA.pdf

USDepartmentofEnergy(2009)NationalSCADATestBed:Enhancingcontrolsystemssecurityinthe

energysector.Washington,DC:USDepartmentofEnergy.Retrievedfromhttp://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/NSTB_Fact_Sheet_FINAL_09‐16‐09.pdf

Vespignani,A.(2010)Complexnetworks:thefragilityofinterdependency.Nature,464(7291),984–

985Williamson,R.M.(2006)Whatgetsmeasuredgetsdone:areyoumeasuringwhatreallymatters?

Columbus,NC:StrategicWorkSystems,Inc.Retrievedfromwww.swspitcrew.comZeng,W.,&Chow,M.Y.(2012)OptimalTradeoffBetweenPerformanceandSecurityinNetworked

ControlSystemsBasedonCoevolutionaryAlgorithms.IEEETransactionsonIndustrialElectronics,59(7):3016‐3025.

Zhu,B.,Joseph,A.,&Sastry,S.(2011).AtaxonomyofcyberattacksonSCADAsystems.InInternetof

things(iThings/CPSCom),2011internationalconferenceoncyber,physicalandsocialcomputing(pp.380‐388).