In: Colbert, E. and Kott, A. (eds.), "Cyber Security of Industrial Control Systems, Including SCADA Systems,"
Springer, NY, 2016
Security Metrics in Industrial Control Systems
ZacharyA.Collier1,MaheshPanwar2,AlexanderA.Ganin3,AlexKott4,IgorLinkov1*
1USArmyEngineerResearch&DevelopmentCenter,Concord,MA,USA
2ContractortoUSArmyEngineerResearch&DevelopmentCenter,Concord,MA,USA
3UniversityofVirginia,Charlottesville,VA,USA
4USArmyResearchLaboratory,Adelphi,MD,USA
*CorrespondingAuthor,[email protected]
1.1 Introduction
Risk–thetopicofthepreviouschapter–isthebestknownandperhapsthebeststudiedexamplewithinamuchbroaderclassofcybersecuritymetrics.However,riskisnottheonlypossiblecybersecuritymetric.OthermetricssuchasresiliencecanexistandcouldbepotentiallyveryvaluabletodefendersofICSsystems.
Often,metricsaredefinedasmeasurablepropertiesofasystemthatquantifythedegreetowhichobjectivesofthesystemareachieved.MetricscanprovidecyberdefendersofanICSwithcriticalinsightsregardingthesystem.Metricsaregenerallyacquiredbyanalyzingrelevantattributesofthatsystem.
Intermsofcybersecuritymetrics,ICSstendtohaveuniquefeatures:inmanycases,thesesystemsareoldertechnologiesthatweredesignedforfunctionalityratherthansecurity.Theyarealsoextremelydiversesystemsthathavedifferentrequirementsandobjectives.Therefore,metricsforICSsmustbetailoredtoadiversegroupofsystemswithmanyfeaturesandperformmanydifferentfunctions.
Inthischapter,wefirstoutlinethegeneraltheoryofperformancemetrics,andhighlightexamplesfromthecybersecuritydomainandICSinparticular.Wethenfocusonaparticularexampleofaclassofmetricsthatisdifferentfromtheonewehaveconsideredinearlierchapters.Insteadofrisk,hereweconsidermetricsofresilience.ResilienceisdefinedbytheNationalAcademyofSciences
2
(2012)as“Theabilitytoprepareandplanfor,absorb,recoverfrom,ormoresuccessfullyadapttoactualorpotentialadverseevents”.
Thischapterpresentstwoapproachesforthegenerationofmetricsbasedontheconceptofresilienceusingamatrix‐basedapproachandanetwork‐basedapproach.Finally,adiscussionofthebenefitsanddrawbacksofdifferentmethodsispresentedalongwithaprocessandtipsintendedtoaidindevisingeffectivemetrics.
1.2 Motivation
UnderPresidentGeorgeW.Bush,theDepartmentofEnergyissuedbestpracticesforimprovedindustrialcontrolsystem(ICS)security(USDepartmentofEnergy,2002).Someoftheseincludetakingstepssuchas"disconnectunnecessaryconnectionstotheSCADAnetwork","establisharigorous,ongoingriskmanagementprocess"and"clearlyidentifycybersecurityrequirements."Additionally,ExecutiveOrder13636,signedbyPresidentBarackObamain2013,broughtforththeissueofcybersecurityandresilience,and
proposedthedevelopmentofarisk‐based“CybersecurityFramework”(EO13636,2013).TheframeworkwaspresentedbytheNationalInstituteofStandardsandTechnology(NIST)andoffersorganizationsguidanceonimplementingcybersecuritymeasures.
Despiteexistingguidelinesandframeworks,designingandmanagingforsecurityincyber‐enabledsystemsremainsdifficult.Thisisinlargepartduetothechallengesassociatedwiththemeasurementofsecurity.PfleegerandCunningham(2010)outlineninereasonswhymeasuringsecurityisadifficulttaskasitrelatestocybersecurityingeneral,butallofwhichalsoapplytothesecurityofICSdomain(Table1).
PfleegerandCunningham(2010)notethatonewaytoovercomethesechallengesistothoughtfullydevelopaclearsetofsecuritymetrics.Unfortunately,thislackofmetricshappenstobeoneofthegreatestbarrierstosuccessinimplementingICSsecurity.WhenICSswerefirstimplemented,"networksecuritywashardlyevenaconcern"(Igureetal,2006).Althougheffortsarebeingmadetodraftandenactcybersecuritymeasures,thatgaphasyettobeclosed,evenatatimeofgreaterrisk.
3
Table1:ChallengeswithCybersecurityMeasurement(adaptedfromPfleeger&Cunningham,2010)
Challenge DescriptionWecan’ttestallsecurityrequirements
Itisnotpossibletoknowallpossibleconfigurationsandstatesofthesystem,intendedusesandunintendedmisusesfromusers,etc.
Environment,abstraction,andcontextaffectsecurity
Systemsarebuilttoevolveastheyprocessnewinformation,andnotallsystemchangesarederivedfrommalicioussources
Measurementandsecurityinteract
Knowledgeaboutasystem’svulnerabilitiesandsafeguardscanaffectthetypesoffurthersecuritymeasuresimplemented,aswellasmodifytherisksthatusersarewillingtotake
Nosystemstandsalone Systemsarenetworkedtointeractwithothercybersystemsandassets
Securityismultidimensional,emergent,andirreducible
Securityexistsatmultiplelevelsofsystemabstraction,andthesecurityofthewholesystemcannotbedeterminedfromthesecurityofthesumofitsparts
Theadversarychangestheenvironment
Developinganaccuratethreatlandscapeisdifficultduetoadaptiveadversarieswhocontinuallydevelopnovelattacks
Measurementisbothanexpectationandanorganizationalobjective
Differentorganizationswithdifferentmissionsandpreferencesplacedifferingvaluesonthebenefitsofsecurity
We’reoveroptimistic Userstendtounderestimatethelikelihoodthattheirsystemcouldbethetargetofattack
Weperceivegainsdifferentlythanlosses
Biasesininterpretingexpectedgainsandlossesbasedonproblemframingtendtoaffectrisktoleranceanddecisionmakingunderuncertaintyinpredictablebutirrationalways
1.3 Background on Resilience Metrics
1.3.1 What Makes a Good Metric?
Accordingtothemanagementadage,“whatgetsmeasuredgetsdone”.Assuch,well‐developedmetricscanassistanorganizationinreachingitsstrategicgoals(Marr,2010).
Reichertetal.(2007)definemetricsas“measurablepropertiesthatquantifythedegreeto
whichobjectiveshavebeenachieved”.Metricsprovidevitalinformationpertainingtoagivensystem,andaregenerallyacquiredbywayofanalyzingrelevantattributesofthatsystem.Someresearchersandpractitionersmakeadistinctionbetweenameasureanda
4
metric(Blacketal.,2008,Linkovetal.,2013a),whereasothersmayrefertothemasperformancemeasures(Neelyetal.,1997),keyperformanceindicators(Marr,2010)orstrategicmeasures(Allen,2011).Forthepurposesofthischapter,thesearereferredtogenerallyasmetrics.
Whenusedefficiently,metricscanhelptoclarifyone’sunderstandingoftheprocessesofaparticularareaofasystem,andfromthere,provideinformationforexternalreviewandassisttowardsfurtherimprovement,amongotheroutputs(Marr,2010).Thiscanbedonebyestablishingbenchmarksforagivenmetric,wherethresholdsorrangescanbeestablished(Blacketal.,2008).Benchmarks,orstandards,helpformthebasisfordecisionmakingandtakingcorrectiveaction(Williamson,2006).
Acriticalelementinelicitingameaningfulmetricistogathertherelevantinformation
aboutone’ssystemandtoalignthatmetricwithmeasurablegoalsandstrategicobjectiveswhichliewithinthescopeofagivenprojectorthedomainofaparticularorganizationalstructure(Beasleyetal.2010,Neelyetal.1997).Thereisalsotheissueofscaleandadaptability.Smallerorganizationmayhavemetricsdealingwithrudimentarysecuritymeasures,butastheygrowlarger,thesemeasuresmayneedtobescaledappropriatelytodealwiththesecurityneededforalargerorganization(Blacketal.,2008).
Therearekeyelementsthatcontributetoproducingasuccessfulmetric.Metricsshouldbeactionable:theyarenotsimplyaboutmeasuringnumerousattributesofaproject;merelygatheringinformationwithoutagoalinmindwillnotprovideadiscerniblesolution(Marr,2010).Suchinformationinandofitselfwouldnotbesubstantialenoughtobeconsideredametric.Gatheringrelevantmetricsrequiresdelvingdeeperintotheissuesfacedbyagivensystemandaskingpertinentquestionswhichcanleadtoactionableimprovement.These
includequestionssuchas“Doesitlinktostrategy?Canitbequantified?Doesitdrivethe
rightbehavior?”(Eckerson,2009).Fromthese,onecanobtainmetricswhichcaninturninformactionableresults.Table2summarizesthedesirablecharacteristicsofmetricsingeneralterms,andapplytoalltypesofsystemsincludingICSs.
5
Table2:CharacteristicsofGoodMetrics(adaptedfromMcKayetal.2012,KeeneyandGregory2005)Characteristic DescriptionRelevant MetricsaredirectlylinkedtodecisionmakinggoalsandobjectivesUnambiguous ConsequencesofalternativescanbeclearlymeasuredbymetricsDirect MetricsclearlyaddressanddescribeconsequencesofinterestOperational DataexistandareavailableforthemetricofinterestUnderstandable MetricscanbeunderstoodandcommunicatedeasilyComprehensive Thesetofmetricsaddressacompletesuiteofgoalsandconsequences
Metricsmaybedescribedasnatural,constructed,orproxy.Naturalmetricsdirectly
describeanobjectiveinunitsthatarestraightforward(e.g.,dollarsasametricfor“costs
associatedwithICSdowntime”).Constructedmetricsmaybeusedwhennaturalmetricsdonotexist(e.g.,scalesfrom1to10whereeachnumbercorrespondstoadefinedlevelofICSperformance),andusuallyincorporateexpertjudgment.Proxymetricscanbeusedtoindirectlymeasureanobjective(e.g.,thenumberofuserswithcertainadministrativeprivilegesasaproxyforaccess)(McKayetal.2012,KeeneyandGregory2005).
Therearedifferenttypesofinformationthatmetricsgaugeandtheprojectteamhastheresponsibilityofappropriatelyselectingandevaluatingthem.Thesecanbeseparatedintoquantitative,semi‐quantitativeandqualitativeapproaches.Quantitativemetricshavemeasurable,numericalvaluesattachedtothem.Semi‐quantitativemetricsarenotstrictlyquantifiablebutcanbecategorized.Qualitativemetricsprovidenon‐numericinformation,forexampleintheformofaesthetics.
1.3.2 Metrics for IT Systems
AsdescribedaboveinTable1,cybersystemsprovideuniquechallenges.Inparticular,thecyberdomainextendsbeyondjusttheimmediatesystemandrequiresaholisticviewpoint,withmanydifferenttechnicalandhumanfactorstobeaccountedfor(Collieretal.,2014).Threatstothesystemarealsoconstantlyevolvingandgrowinginsophistication,andasaresult,thereisahighdegreeofadaptabilityrequiredinordertoremaincurrent.Duetotheconstantlyevolvingthreatspace,thereisoftenlittlehistoricaldataforpotentialthreats(Collieretal.,2014).
6
Withcybermetrics,asignificantnumberofthemainissuesaretailoredtowardssecurityandresilience.TheDefenseScienceBoard(2013)arguesthateffectivecybermetricsshouldbebroadenoughtofitdifferenttypesofsystems,yetalsobepreciseenoughtodialdownintothespecificsofagivensystem.Thefollowingaresomeexamplesofcybersecuritymetricscurrentlyinuse.
TheCommonVulnerabilityScoringSystem(CVSS)wasintroducedtoprovidevariousorganizationswithactionableinformationinregardstoassessingITvulnerabilities(Melletal.,2007).CVSSgroupstheirmetricsintothreecategories,namelyBase,Temporal,andEnvironmentalmetrics.AfewofthesesecuritymetricsincludeCollateralDamagePotential,TargetDistribution,ReportConfidence,Exploitability,AccessComplexity,AccessVector,Authentication,IntegrityImpact,AvailabilityImpact,andConfidentialityImpact(Melletal.,2007).Therearegeneralscoringtipsforthewaythatvulnerabilitiesareassessed;vulnerabilitiesarenotscoredbasedoninteractionswithothervulnerabilities,rather,theyarescoredindependently.Themainmeasureofvulnerabilityisitsimpactonthekeyservice.Vulnerabilitiesarescoredaccordingtocommonlyusedprivileges,whichmightbeadefaultsettingincertainsituations.Ifavulnerabilitycanbeexploitedbymultipleexploits,itisscoredwiththeexploitthatwillpresentthemaximumimpact(Mell,etal.,2007).CVSSallowsvulnerabilityscorestobestandardized,andBasemetricsare
normalizedonascaleof0–10.TheycanbeoptionallyrefinedbyincludingvaluesfromTemporalandEnvironmentalmetrics.
TheCenterforInternetSecurity(CIS)hasalsoestablishedmetricsfororganizationstouse(CIS,2010).CIShasdividedtheirmetricsintosixcriticalbusinessfunctions.TheseareIncidentManagement,VulnerabilityManagement,PatchManagement,ConfigurationManagement,ChangeManagementandApplicationSecurity.Italsorecognizeshierarchiesandinterdependenciesofmetrics,forinstancecitingmanagementmetricsasbeingofprimaryimportancetoanorganization,whilenotingthatsomeofthosemetricsmaydependonthepriorimplementationoftechnicalmetrics(CIS,2010).SomeofthemetricsincludeCostofIncidentsandPatchPolicyCompliance.CostofIncidentsreferstoanumber
ofpotentiallosses,suchascustomerlistsortradesecretsundera“directloss”anda“cost
ofrestitution”,forexampleintheeventthatfinesareleviedduetoanincident.Thisismeasuredbythesummationofthenumericalvaluesofallthecostsassociatedwiththemetric.ExamplesrelatingtosecurityincludeMeanTimetoIncidentDiscovery,MeanTimeBetweenSecurityIncidentsandMeanTimetoIncidentRecovery(CIS,2010).Foranexampleofmeasurement,MeanTimetoIncidentDiscoverymeasuresthesummationofthe
7
timebetweenincidentsanddiscoveriesofincidents,dividedbytotalnumberofincidentsrecoveredduringthosetimeframes(CIS,2010).
TheCybersecurityFrameworkdevelopedbyNISTstemmingfromEO13636wasreleasedinFebruary2014(NIST2014a).ThefinalCybersecurityFrameworkconsistsofa
FrameworkCore,whichpresentsasetoffive“concurrentandcontinuousFunctions–
Identify,Protect,Detect,Respond,Recover”(NIST2014).Thesefunctionsarethe“high‐
level,strategicviewofthelifecycleofanorganization’smanagementofcybersecurityrisk,”whichfeaturesubsequentcategoriesandsubcategoriesforthefunctions,relatingtooutcomesandactivities(NIST2014).Forexample,theRespondfunctionconsistsoffivecategories,amongwhichincludesMitigation.Mitigationisthenfurthersubdividedintometricsrelatedtocontaininganderadicatingincidents.TheFrameworkCoreisusedasa
scorecardofprogress–thecurrentguidancecallsforfirstdevelopinganorganization’s
CurrentProfile,whichconsistsofassignedscoresbasedontheorganization’sperformanceineachofthecategoriesandsubcategories.ThisCurrentProfileisthencomparedtoaTargetProfile,representingthedesiredstateoftheorganizationineachofthesamecategoriesandsubcategories.Theshortfallsbetweentheseprofilescanbeviewedasgaps
inanorganization’scyber‐riskmanagementcapabilitieswhichcaninformprioritizationofcorrectivemeasures(NIST2014;Collieretal.2014).
TheSoftwareEngineeringInstitute(SEI)atCarnegieMellonUniversitydevelopedaframeworkforassessingoperationalresiliencewhichfeaturesasetofTopTenStrategicMeasures,whichaimtobemappeddowntothelevelofspecificProcessAreameasures(AllenandCurtis,2011).UndertheheadingofHigh‐ValueServicesandAssets,oneofthemeasuresisrelatedtothepercentageofhigh‐valueservicesthatdonotsatisfytheirassignedresiliencerequirements(AllenandCurtis,2011).TheSEIframeworkalsocontainsalargeamountofresiliencemeasures,spanning26differentProcessAreas.Forexample,undertheProcessAreaofEnvironmentalControl,therearemeasuressuchasPercentageofFacilityAssetsthathavebeenInventoried,ElapsedTimeSincetheFacilityAssetInventorywasReviewed,andElapsedTimeSinceRiskAssessmentofFacilityAssets
Performed(AllenandCurtis,2011),wheretheterm“assets”appliestohigh‐valueservices.Thesearepresentedinatablewithtraceability,assigninganidentificationnumbertoeachmetricalongwiththeirapplicabilitytogoalswithintheProcessAreas.
8
MITREproposedaframeworkentitledCyberResiliencyEngineeringFramework,which,
amongitsgoalsaimsto“motivateandcharacterizecyberresiliencymetrics”(Bodeau2011).TheframeworkcontainsfourCyberResiliencyGoals:Anticipate,Withstand,Recover,andEvolve.Thereareatotalofeightobjectiveswhichareasubsetofthegoals.ForexampleAnticipatehasthreeobjectives:Predict,Prevent,andPrepare(Bodeau,2011).Thishierarchycanbeusedtoinformandcategorizetheappropriateresiliencemetrics.Thesearemeanttobeperformedsimultaneously,andbeararesemblancetotheNISTframeworkmentionedearlier.
1.3.3 Metrics for ICS Networks
Theabovemetricsweredevelopedfor“cyber”systemsgenerallyspeaking,notspecificallyforICSs,althoughtheycanbetailoredwithICSsinmind.ICSsinparticularareauniquecase;inmanysituations,thesesystemshaveoldermodels,andweredesignedforfunctionalityratherthansecurity(USDepartmentofEnergy,2002).Theyconstituteadiversegroupofsystemsthathavedifferentrequirementsfortheirvariousoperations(Pollet,2002).
SpecificallyasitrelatestoICSs,time,safetyandcontinuationofservicesareofgreatimportance,sincemanysystemsareinapositionwhereafailurecanresultinathreattohumanlives,environmentalsafety,orproductionoutput(Stouffer,2011).Sincetheserisksaredifferentthanthosefacedbyinformationtechnology(IT)systems,differentprioritiesarealsonecessary.Examplesofsomeuniqueconsiderationsincomparisontocybersecurityincludethelongerlifespanofsystemcomponents,physicallydifficulttoreachcomponents,andcontinuousavailabilityrequirements(Stouffer,2011).Additionally,thesesystemstypicallyoperateinseparatefieldsthancybersecurity,suchasinthegasandelectricindustries,andsometricsmustbeadaptedtofitthesedifferentorganizationalstructures(McIntyreetal.,2007).CriticalinfrastructuresarecommonforICSs,andasa
result“downtimeandhaltingofproductionareconsideredunacceptable”(McIntyreetal.,2007).
9
Stoufferetal.(2011)comparethedifferencesbetweeninformationtechnology(IT)system
andICSs,focusingonthesafety‐criticalnatureofmanyICSnetworks.Forexample,“high
delayandjittermaybeacceptable”asaperformancerequirementforITsystems,whereasforICSs,itmaynotbeacceptable(Stouffer,2011).Thisisduetothefactthatthereisatime‐criticalnaturetoICSs,whereasforITsystemsthereishighthroughput,allowingfor
somejitter(Stouffer,2011).Similarly,forIT,“systemsaredesignedforusewithtypical
operatingsystems”andforICSs,thereare“differingandpossiblyproprietaryoperating
systems,oftenwithoutsecuritycapabilitiesbuiltin”.Therearealsoavailabilityrequirements,inthatsometimesanITstrategymayrequirerestartingorrebootingaprocess,somethingwhich,forICSprocesses,requiresmorecarefulplanningasunexpectedoutagesandquicklystoppingandstartingasystemarenotacceptablesolutions(Stouffer,2011).Withthesekeydifferencesbetweenthetwodomains,therearevaryinglevelsofadaptationneededinordertobegintheprocessofsecuringICSnetworks.
TheUSNationalSecurityAgency(NSA)draftedaframeworkforICSnetworks,focusingonpotentialimpactandlossrelatingtoanetworkcompromise(NSA,2010).Theysuggested
assigninglossmetricsincorporatingNIST’sframework:compromisespertainingtoConfidentiality,IntegrityandAvailabilityforeachnetworkasset(NSA,2010).A
Confidentialitycompromiseisdefinedasan“unauthorizedreleaseortheftofsensitive
information”e.g.theftofpasswords(NSA,2010).AnIntegritycompromiseisdefinedasan
“unauthorizedalterationormanipulationofdata”,e.g.manipulationofbillingdata(NSA,
2010).AnAvailabilitycompromiseisdefinedasa“lossofaccesstotheprimarymissionofa
networkedasset”e.g.deletionofimportantdatafromadatabase(NSA,2010).Thesemayalsobestreamlinedintoonemetric,usingthehighestvalue(e.g.ofLow,ModerateorHigh)amongthethreeareas.
Theassignmentofathreatmetricsateachpotentialattackvectorwassuggested,butspecificexampleswerenotprovided.Fivethreatsourceswereidentified:Insiders,TerroristsorActivists,HackersorCyber‐Criminals,Nation/StateSponsoredCyber‐WarfareandCompetitors(NSA,2010).Bothlossandthreatmetricscanberatedonaconstructedscale(Low,ModerateorHigh)andgivenanumericratingonasetscale.Itwasmentionedthattheimportantconsiderationistohaveascale,andthatthenumberofgraduationsin
10
thescaleisnotimportant,solongastheconstructedscaleremainconsistent(e.g.apotentialforlossoflifewillrankasHigh)(NSA,2010).Combiningresultsofmetricswasalsodiscussedasapossibility.Asanexample,foragivenpointinthenetwork,aLossMetricisassignedascoreofHighontheconstructedscale(3)andaThreatmetricatthatsamenetworkpointisratedatModerate(2).Fromthis,onecanarriveatacompositepriorityvalue,whichissimplythesumofthosetwoscores.Othersuchpointscanbeevaluatedandthenprioritizedandranked(NSA,2010).Thescoringmethodologyisabasicexample,(andnottheonlymethod‐weighingmetricswaslistedasapossibility(NSA,2010))andmorerobustmethodscanbedevised.
BoyerandMcQueen(2008)devisedasetofideal‐basedtechnicalmetricsforcontrolsystems.Theyexaminedsevensecuritydimensionsandpresentanideal,orbestcasescenario,foreachofthem.TheidealsareSecurityGroupKnowledge,AttackGroupKnowledge,Access,Vulnerabilities,DamagePotential,Detection,andRecovery.FortheAccessdimension,theidealstatesthatthesystemisinaccessibletoattackgroups.ThesecuritydimensionofVulnerabilitieshasanidealstatingthatthesystemhasnovulnerabilities(BoyerandMcQueen,2008).Bytheverynatureofanideal,thesemaybeimpossibletoachieveandmaintainintherealworld.Butfromthem,metricsweredevisedthatcouldbestrepresenttherealizationoftheseideals.Underthevulnerabilitydimension,
themetricVulnerabilityExposureisdefinedas“thesumofknownandunpatched
vulnerabilities,eachmultipliedbytheirexposuretimeinterval.”Itwassuggestedthatthismetriccouldbebrokendownintoseparatemetricsfordifferentvulnerabilitycategories,aswellasincludingaprioritizationofvulnerabilities,citingCVSS.UndertheAccessdimension,thereisthemetricRootPrivilegeCount,whichisthecountofallpersonnelwith
keyprivileges,arguinginfavoroftheprincipleofleastprivilege,whichstatesthat“everyprogramandeveryprivilegeduserofthesystemshouldoperateusingtheleastamountof
privilegenecessarytocompletethejob”(Saltzer,1974).Thislogicalorderingofmetricswithinthescopeofidealscanbeofvaluetothosewishingtodevisetheirownsetofmetrics.
Theideal‐basedmetrics(BoyerandMcQueen,2008)alsoacknowledgethephysicalspaceofICSnetworks.ThemetricRogueChangeDays,whichisthenumberofchangestothesystemmultipliedbythenumberofdaysundetected,includesProgrammableLogicControllersandHuman‐MachineInterfacesandotherICSrelatedsystems.ComponentTestCount,ametricmeasuringthenumberofcontrolsystemcomponentswhichhavenotbeentestedisasimplemeasure,butofsignificanceduetonumerouscomponentsinuseinanICSsystem.
11
Withintheideals,themetricofAttackSurface(definedbyManadhataandWing(2011)as
“thesubsetofthesystem’sresources(methods,channels,anddata)potentiallyusedin
attacksonthesystem”)wasdeterminedtonotbedevelopedenoughforrealworlduse.
BoyerandMcQueenfurtherarguethat“acrediblequantitativemeasureofsecurityriskis
notcurrentlyfeasible”(BoyerandMcQueen2008).Butwiththeinclusionofatheoreticalmetric,andaframeworkforsecurity,thisdemonstratesaforwardthinkingattitudethatcanbebuiltuponbythoseaimingtoestablishtheirownsecurityprotocols.ThisrepresentsimportantfutureworkfortheICSandsecuritycommunities.ComparisonsbetweentheNSAapproachandtheapproachoutlinedbyBoyerandMcQueenarepresentedinTable3.
Table3:ComparisonbetweenICSMetrics
NationalSecurityAgency(2010) BoyerandMcQueen(2008)
Focus LossandThreatfocusedMetrics(p.10,15)
Quantitativetechnicalmetrics(p.1),idealbased:attemptedtohavemetricsthatcouldstrivetowardidealscenarioswithinsevensecurityareas
Amount Threelossmetrics(pernetworkedasset),oneThreatmetric(perpotentialattackvector)
13totalmetrics(suggestedtotal:lessthan20)
AppliedorTheoretical
Suggestsdeployablemetrics Discussesbothdeployableandtheoreticalmetrics(p.10,11)
QuantitativeorQualitative
Semi‐qualitative(suggestsHigh,Medium,Low,withallowancefornumericattachmenttothesevalues)
Doesnotfocusonqualitativemetrics(p.1),butonquantitativemetrics
CombinationofMetrics
Presentsmethodtocombineresultsofmetricscoresforranking
Nocombinationofmetrics
ConsequenceConsiderations
LossMetricsarerelatedtoConfidentiality,Integrity,Availability
AcknowledgesthepurposeofsecurityisprotectionofConfidentiality,IntegrityandAvailability(p.4)
12
ComplementaryresearchtometricsdevelopmentintheICSrealmiscurrentlybeingconducted.OnesucheffortistodevelopastandardizedtaxonomyofcyberattacksonSCADAsystems(Zhuetal.,2011).AcommonlanguagefordescribingattacksacrosssystemscanfacilitatethedevelopmentoffurtherthreatandvulnerabilitymetricsforICSs.Inaddition,thedevelopmentofanationaltestbedforSCADAsystemsisbeingdevelopedbytheDepartmentofEnergywhichwillenablethemodelingandsimulationofvariousthreatandvulnerabilityscenarios,whichwillallowresearcherstodevelopabetterunderstandingofwhatmetricsmayormaynotbeusefulinmonitoringandmanagementofthesesystems(USDepartmentofEnergy,2009).Anotherdevelopmentrelatedtometricsresearchistheinvestigationoftradeoffsbetweencertaincriticalmetrics.Oneexampleisbetweenoptimizingsystemperformancewithsystemsecurity,whereadditionalsecuritymeasuresmayresultinreducedperformance.Zeng&Chow(2012),developedanalgorithmictechniquetodeterminetheoptimaltradeoffbetweenthesetwometrics,andthemethodcanbeextendedtotradeoffsbetweenothermetricsaswell.
1.4 Approaches for ICS Metrics
Whilevariousframeworksandsetsofmetricsexist,suchastheonesmentionedintheprevioussection,itcanbedifficultformanagersandsystemoperatorstodecidewhethertoadoptormodifyanexistingset,ortocreateanentirelynewsetofmetrics.Balancingthetradeoffsbetweengeneralizablemetricsandspecificsystem‐levelandcomponent‐levelmetricscanbechallenging(DefenseScienceBoard,2013).Thefollowingapproachesprovideastructuredwaytothinkaboutdevelopingmetrics,allowinguserstoleverageexistingmetricsbutalsoidentifygapswherenewmetricsmayneedtobecreated.Theuseofsuchstructuredandformalizedprocessesrequiresthethoughtfulanalysisofthesystemsbeingmeasured,butalsohowtheyrelatetothebroaderorganizationalcontext,suchasgoals,constraints,anddecisions(Marr,2010).Moreover,thedevelopmentofastandardizedlistofquestionsortopicshelpstosimplifytheprocessofdesigningametric.Thedevelopmentofmetricsshouldbeasmoothprocess,andsuchalistcanprovideinsight
intothe“behavioralimplications”ofthegivenmetrics(Neelyetal.1997).
1.4.1 Cyber Resilience Matrix Example
ThefirstmethodisbasedontheworkofLinkovetal.(2013a).Unliketraditionalrisk‐basedapproaches,thisapproachtakesaresilience‐centrictheme.Muchhasbeenwrittenelsewhereontherelativemeritsofaresilience‐focusedapproach(seeLinkovetal.,2013b,2014;Collieretal.2014;Roegeetal.2014;DiMaseetal.2015),butweshallbriefly
13
summarizetheargumenthere.TraditionalriskassessmentbasedonthetripletformulationproposedbyKaplanandGarrick(1981)becomesdifficulttoimplementinthecybersecuritycontextduetotheinabilitytoframeandevaluatemultipledynamicthreatscenarios,quantifyvulnerabilityagainstadaptiveadversaries,andestimatethelong‐termandwidelydistributedconsequencesofasuccessfulattack.Insteadofmerelyhardeningthesystemagainstpotentialknownthreatsinarisk‐basedapproach,thesystemcanbemanagedfromtheperspectiveofresilience,whichincludestheabilityofoneormore
criticalsystemfunctionalitiestoquickly“bounceback”toacceptablelevelsofperformance.Asaresult,aresilientsystemcanwithstandandrecoverfromawidearrayofknownandunknownthreatsthroughprocessesoffeedback,adaptation,andlearning.
Followingthisthoughtprocess,Linkovetal.(2013a)establishedamatrix‐basedmethod.Ononeaxis,thestepsoftheeventmanagementcycleidentifiedasnecessaryforresiliencebytheNationalAcademyofSciences(2012)arelisted,andincludePlan/Prepare,Absorb,Recover,andAdapt.Notethattheabilitytoplan/prepareisrelevantbeforeanadverseevent,andtheothercapabilitiesarerelevantafterdisruption.OntheotheraxisarelistedthefourdomainsinwhichcomplexsystemsexistasidentifiedbyAlberts(2002),andincludePhysical,Information,Cognitive,andSocialdomains.ThePhysicaldomainreferstothephysicalresourcesandcapabilitiesofthesystem.TheInformationdomainreferstotheinformationanddatathatcharacterizethePhysicaldomain.TheCognitivedomaindescribestheuseoftheotherdomainsfordecisionmaking.Finally,theSocialdomainreferstotheorganizationalstructureandcommunicationsystemsfortransmittinginformationandmakingdecisions(Alberts2002).
Together,theseaxesformasetofcellsthatidentifyareaswhereactionscanbetakenin
specificdomainstoenhancethesystem’soverallabilitytoplanfor,andabsorb,recover,andadaptto,variousthreatsordisruptions(Figure1).Eachcellisdesignedtoanswerthe
question:“Howisthesystem’sabilityto[plan/preparefor,absorb,recoverfrom,adaptto]
acyberdisruptionimplementedinthe[physical,information,cognitive,social]domain?”(Linkovetal.2013a).
14
Figure1:GenericResilienceMatrix
Aresultingsetof49metricsareproducedthatspanthevariouscellsofthematrix,andselectedmetricsareshowninTable4(seeLinkovetal.2013aforthecompletelist).Metricsaredrawnfromseveralsourcesandaremeanttobegeneralandnotnecessarily
comprehensive.Forexample,underAdaptandInformation,ametricstates“document
timebetweenproblemanddiscovery,discoveryandrecovery,”whichhasaparalleltothe
MeanTimetoIncidentDiscoverywithinSEI’sguidance.ThemetricsunderPlanandInformation,relatedtoidentifyinginternalandexternalsystemdependenciescanbecomparedtotheTemporalMetricofAccessComplexityfromCVSS,whichrelatestohoweasilyavulnerabilitycanbeexploited.ThemetricunderPrepareandSocialpresentsa
simpleyetimportantmessagethatholdstrueinalloftheframeworks:“establishacyber‐
awareculture.”
TheresiliencematrixapproachdescribedinLinkovetal.(2013a)hasseveralstrengthsinthatthemethodisrelativelysimpletouseandoncemetricshavebeengenerated,itcanserveasaplatformforamulti‐criteriadecisionaid(Collier&Linkov,2014).Ithasthepotentialtoserveasascorecardinordertocapturequalitativeinformationabouta
system’sresilience,andaidmanagersandtechnicalexpertsinidentifyinggapsinthe
system’ssecurity.However,theresiliencematrixdoesnotcapturetheexplicittemporalnatureofresilience(i.e.,mappingthecriticalfunctionalityovertime)orexplicitlymodelthesystemitself.Inthisregard,itcanbeviewedasahighlevelmanagementtoolthatcanbeusedtoidentifyasnapshotwheremoredetailedanalysesandmodelingcouldpotentiallybecarriedout.
Plan &
Prepare
Absorb Recover Adapt
Physical
Information
Cognitive
Social
15
Table4:SelectedCybersecurityMetricsDerivedfromtheResilienceMatrix(adaptedfromLinkovetal.,2013a).
Plan/Prepare Absorb Recover AdaptPhysical Implement
controls/sensorsforcriticalassetsandservices
Useredundantassetstocontinueservice
Investigateandrepairmalfunctioningcontrolsorsensors
Reviewassetandserviceconfigurationinresponsetorecentevent
Information Prepareplansforstorageandcontainmentofclassifiedorsensitiveinformation
Effectivelyandefficientlytransmitrelevantdatatoresponsiblestakeholders/decisionmakers
Reviewandcomparesystemsbeforeandaftertheevent
Documenttimebetweenproblemanddiscovery,discoveryandrecovery
Cognitive Understandperformancetrade‐offsoforganizationalgoals
Focuseffortonidentifiedcriticalassetsandservices
Establishdecisionmakingprotocolsoraidstoselectrecoveryoptions
Reviewmanagementresponseanddecisionmakingprocesses
Social Establishacyber‐awareculture
Locateandcontactidentifiedexpertsandresponsiblepersonnel
Determineliabilityfortheorganization
Evaluateemployeesresponsetoeventinordertodeterminepreparednessandcommunicationseffectiveness
1.4.2 Network Simulation Example
Thesecondmethodisbasedonmodelingofcomplexcyberandothersystemsasinterconnectednetworks,whereafailureinonesectorcancascadetootherdependentnetworksandassets(Vespignani,2010).ThisisareasonableassumptionforICSnetworks;forexample,adisruptionoftheelectricalgridcandirectlyimpactdependentsectorssuchasthenetworkcontrollingICSdevicesleadingtoacascadeoffailuresasitisbelievedtohavehappenedduringtheItalianblackoutin2003(Buldyrevetal.,2010).Thusthe
16
assessmentofthesecurityofasingleICSnetworkshouldbeviewedinthecontextofalargernetworkofinterdependentsystems.
Ganinetal.(2015)tookthisnetwork‐orientedviewindevelopingamethodologytoquantitativelyassesstheresilience(andthussecurity)ofnetworkedcybersystems.TheybuiltupontheNationalAcademyofSciences(2012)definitionofresilienceasasystempropertythatisinherentlytiedtoitsabilitytoplanfor,absorb,recoverfrom,andadapttoadverseevents.Inordertocapturethestateofthesystemtheauthorsproposetousetheconceptofcriticalfunctionalitydefinedasatime‐specificperformancefunctionofthe
systemconsideredandderivedbasedonthestakeholder’sinput.Forinstanceinthenetworkofpowerplants,thecriticalfunctionalitymightrepresentthetotaloperationalcapacity.Inthenetworkofcomputersitmightrepresentthefractionofserversandservicesavailable.Valuesofcriticalfunctionalityarerealnumbersfrom0to1.Otherkey
elementstoquantifyresiliencearethenetworkedsystem’stopologyanddynamics;therangeofpossibleadverseevents(forexample,acertaindamagetonodesofthenetwork);andthecontroltimeTC(thatisthetimerangeoverwhichtheperformanceofthesystemisevaluated).Thenthedependencyofthecriticalfunctionality(averagedoveralladverseevents)overtimeisbuilt.Ganinetal.(2015)refertothisdependencyastheresilienceprofile.Asitistypicallycomputationallyprohibitiveornotpossibleatall(incaseof
continuousvariablesdefiningnodes’states)toconsiderallthewaysanadverseeventcanhappen,itissuggestedtoutilizeasimulationbasedapproachwithMonte‐Carlosampling.
Givenitsprofileinnormalizedtime(wheretimeTCistakentobe1),theresilienceofthenetworkcanbemeasuredastheareaunderthecurve(yellowregioninFigure2).Thisallowsmappingoftheresiliencetorealvaluesrangingbetween0and1.
Anotherimportantpropertyofthesystemisobtainedbyfindingtheminimumoftheaveragecriticalfunctionality.SomeresearchersrefertothisvalueasrobustnessM
(Cimellaroetal.,2010),whileLinkovetal.(2014)notethat1–Mcorrespondstothemeasureofrisk.
17
Figure2:Ageneralizedresilienceprofile,whereasystem’sresilienceisequaltotheareabelowthe
criticalfunctionalitycurve(adaptedfromGaninetal.,2015).
IntheirpaperGaninetal.(2015)illustratedtheapproachonadirectedacyclicgraph.Eachlevelinthisgraphrepresentsasetofnodesfromcertaininfrastructuresystem(e.g.electricalgrid,computersetc.).Nodesofdifferentlevelsareconnectedbydirectedlinksrepresentingadependencyofthedestinationnodeonthesourcenode.Inthesimplestcaseanodeinacertainlevelrequiressupply(oradependencylink)fromanodeineachoftheupperlevelsanddoesnotdependonanynodesinthelowerlevels.Otherparametersofthe
modelincludenoderecoverytime(TR)–ameasureofhowquicklyanodecanreturntoan
activestateafterit’sbeeninactivatedasaresultofanadverseevent;redundancy(pm)–theprobabilitycontrollingthenumberofadditionalpotentialsupplylinksfromupperlevelstolowerlevels;andswitchingprobability(ps),controllingeaseofreplacementofadisruptedsupplylinkwithapotentialsupplylink.Theseparameterscouldbeextendedtoothersituationstoinformhowasystemmaydisplayresilientbehavior,andthusincreasingthesecurityofthesystemasawhole.
Theauthorsfoundthatthereisstrongsynergybetweenpmandps;increasingbothfactorstogetherproducesarapidincreaseinresilience,butincreasingonlyoneortheothervariablewillcausetheresiliencemetrictoplateau.Resilienceisstronglyaffectedbythetemporalswitchingtimefactor,TR.Thistemporalfactordeterminesthecharacteristicsoftherecoveryphaseandhasagreaterimpactonthecalculatedresiliencethandoesthepotentialincreaseinredundancy.Thisisparticularlytruewhentheswitchingprobabilitypsislow.Animportantlongtermchallengeistomodeladaptation,which,accordingtotheNationalAcademyofSciences,ispartoftheresponsecyclethatfollowsrestorationandincludesallactivitiesthatenablethesystemtobetterresistsimilaradverseeventsinthefuture.
18
Ganinetal.(2015)notethatthemainadvantagesoftheapproachincludeitsapplicabilitytoanysystemthatcanberepresentedasasetofnetworks.Alsoboththeresilienceandtherobustnessofasystemaremetricizedusingarealvalueinrangebetween0and1(where1correspondstotheperfectresilienceorrobustness)makingcomparisonofresilienceofdifferentsystemseasy.Ontheotherhandmappingtheresiliencepropertyofasystemtoa
singlevaluenecessarilyshadowssomesystem’simportantcharacteristics(forinstance,therateofrecovery).Theresilienceprofilecouldbeusedasamoreholisticrepresentationof
thesystem’sresiliencenotingthateveninthatcaseonlytheaveragevalueofcriticalfunctionality(ateachtimestep)istakenintoaccount.Tofullydescribeasystemoneshouldconsiderthedistributionofthevalueofcriticalfunctionality(ateachtimestep)fordifferentinitialadverseevents.Finally,itisnotpossibletosimulatealladverseeventsfromtherangeusedtoestimateresilienceandtheapproachisMonte‐Carlobased.Itmeansthatinorderfortheresultstobereliablethenumberofsimulationsistypicallyrequiredtobeveryhigh.
1.5 Tips for Generating Metrics
1.5.1 Generalized Metric Development Process
ThefollowingprocesstowardsthedevelopmentofmetricsisadaptedbyMcKayetal.2012.
1. ObjectiveSetting:Articulateclear,specificgoals.Thisshouldbedoneinastructuredmanner.GregoryandKeeney(2002)outlineastructuredapproachtodothis.
a. Writedownalloftheconcernsthattheprojectteamfeelsisrelevant.b. Convertthoseconcernsintosuccinctverb‐objectgoals(e.g.,minimizedowntime).c. Next,theseshouldbeorganized,oftenhierarchically,separatinggoalswhich
representmeansfromthosewhichrepresentends.d. Finally,reviewandclarificationshouldbeconductedwiththeprojectteam.This
maybeaniterativeprocess.2. DevelopMetrics:Oncetheobjectivesareclearlyarticulatedandorganized,metricscanbe
formallydeveloped.a. Thefirststepistoselectabroadsetofmetrics,whichmaybeselectedfromexisting
listsorguidelines,orcreatedbyaprojectteamorsubjectmatterexpertsfortheparticularpurposeathand.ThisstepiswheretheResilienceMatrixcouldfacilitatemetricdevelopment.
b. Next,thissetofmetricsshouldbeevaluatedandscreenedtodeterminewhetheritmeetstheprojectobjectivesandthedegreetowhichthemetricsmeetthedesirablequalitiesofmetrics,explainedearlierinthischapter.Atthisstage,remainingmetricscanbeprioritized.
19
c. Finallytheremainingmetricsshouldbedocumented,includingassumptionsandlimitations,andothersupportinginformation.
3. CombinationandComparison:Amethodshouldbedevelopedforhowthemetricswillultimatelybeusedtosupportdecisionmakinganddriveaction.Somemethodsinclude:
a. NarrativeDescription:Simpletechniqueswheretrade‐offsmaybesimplesuchaslistingevidenceorbestprofessionaljudgement.
b. ArithmeticCombination:Simplemathematicaltechniquesforcombiningdissimilarmetricssuchassimpleaggregationofmetricswithsimilarunits(e.g.,cost),convertingtosimilarunits(e.g.,monetization),ornormalizingtoasimilarscale(e.g.,0to1).
c. Multi‐CriteriaDecisionAnalysis:Amethodforweightingandscoringdissimilardecisioncriteriabasedontheirrelativeimportanceandperformancewithrespecttoanobjective.
d. InterdependentCombination:Forsystemsthatarecomplex,usuallyinvolvingintricateinternalrelationships,moreintensivemodelingeffortsmaybenecessary,suchasBayesiannetworksorothercomplexsystemsmodelingtechniques.
Theabove‐mentionedprocess,alongwithasolidmetricdevelopmentprocess,cangreatlyaidindevisingeffectivemetrics.Oftenitisnecessarytodevelopaconceptualmodelofthesysteminordertoidentifythefunctionalrelationshipsandcriticalelementsandprocesseswithinasystem.ThiscanbedoneusingaNetworkScienceapproachdescribedabove.
1.5.2 Best Practices in Metric Development and Validation
Validationofmetricsisanoftenoverlookedaspectofthemetricdevelopmentprocess.Neelyetal.(1997)providesomequestionstoaskregardingwhethertheoutputfromthemetricsisappropriate,specificallywhetherthemetricshaveaspecificpurpose,arebasedonanexplicitformulaand/ordatasource,andareobjectiveandnotbasedsolelyonopinion(Neelyetal.,1997).Similarly,Eckerson(2009)laysoutaseriesofquestionsthatcanserveasaqualitycheckondevelopedmetrics,toensurethattheyareofhighquality:
•Doesitlinktostrategy?•Canitbequantified?•Doesitdrivetherightbehavior?•Isitunderstandable?•Isitactionable?•Doesthedataexist?
Regardingthenumberofmetricsnecessary,itisn’tnecessarilythequantityofmetricsthatconstituteasuccessfulimplementation,butwhetherthesemetricsarecollectivelycomprehensiveenoughtoaddresseverythingdeemedimportant(McKayetal.2012).Eckerson(2009)recommendsthatasetofmetricsbesparse,sincewithalimitednumber
20
ofmetricsitiseasiertoanalyzehowmetric‐levelchangesdrivetheperformanceinthesystem,aswellasthepracticalfactthatgathering,synthesizing,andpresentingmultipledatastreamsoftentakesquitesometime.Moregranular,process‐levelmetricsmaystillberequiredhowever,andEckerson(2009)proposesaMAD(monitor,analyze,drill)frameworkforpresentingdifferentlevelsofresolutiontodifferentusersofthatinformation.
Anotherongoingelementofvalidationistraceability,asevidencedintheframeworkpresentedbyNeelyetal.(1997),whichincludesalistofinformation(knownastheperformancemeasurerecordsheet)suchashowoftendataistobecollected,andby
whom,aswellasimportantquestionssuchas“whoactsonthedata?”and“whatdothey
do?”.Ifthesequestionsareconsideredandansweredastheneedarises,itisknownwhoisresponsibleformakingthemeasurementandwhatactionsaretobetakenasaresult.Thiscanrevealinsightintothemetricandhowtheyaremeasuredandbeingutilized,notjustforthecurrentprojectbutforfuturereference.Anitemonthelistaskswhatthemetric
“relatesto.”Thiscanassistinenteringthemindsetofapproachingmetricswithaninterconnectedandgoal‐orientedviewpoint.
Othervalidation‐relatedeffortsincludestandardizingmethodsforICSmetricdevelopmentandimplementation,aswellasinstitutionalizingaclearmeanstointegratemetricswithdecisionanalytictoolstosupporttheriskmanagementprocess.Finally,giventhedynamicnatureofcyberthreats,periodicreviewandupdatingofICSmetricsshouldbeconductedtokeepabreastofthelatestdevelopmentsinthefield.
1.6 Conclusions
Despiteexistingguidelinesandframeworks,designingandmanagingforsecurityincyber‐enabledsystemsremainsdifficult.Thisisinlargepartduetothechallengesassociatedwiththemeasurementofsecurity.Acriticalelementinelicitingameaningfulmetricisingatheringtherelevantinformationaboutone’ssystemandaligningthatmetricwithmeasurablegoalsandstrategicobjectives.ForICSs,time,safetyandcontinuationofservicesfactorconsiderablyintooverallgoals,sincemanysystemsareinapositionwhereafailurecanresultinathreattohumanlives,environmentalsafety,orproductionoutput.Oftenitisnecessarytodevelopaconceptualmodelofthesystemordevelopastandardizedlistofquestionsortopicshelpstoidentifycriticalprocesselements,thefunctionalrelationshipsandcriticalelementsandprocesseswithinasystem.Inthischapter,wediscussindetailtwoapproachesforthegenerationofbroadlyapplicablesecurityandresiliencemetricsandtheirintegrationtoquantifysystemresilience.Thefirstmethodisasemi‐quantitativeapproachinwhichthestagesoftheeventmanagementcycle(plan/prepare,
21
absorb,recover,andadapt)areappliedacrossfourrelevantdomains(physical,information,cognitive,social),formingamatrixofpotentialsecuritymetrics.SecondisaquantitativeapproachbasedonNetworkScience,inwhichfeaturessuchasnetworktopologiescanbemodeledtoassessthemagnitudeandresponsivenessofthecriticalfunctionalitiesofnetworkedsystems.Validationofmetricsisanoftenoverlookedaspectofthemetricdevelopmentprocess;howeveraseriesofquestionscanserveasaqualitycheckondevelopedmetrics,toensurethattheyareofhighquality.
PermissionwasgrantedbytheUSACEChiefofEngineerstopublishthismaterial.TheviewsandopinionsexpressedinthispaperarethoseoftheindividualauthorsandnotthoseoftheUSArmy,orothersponsororganizations.
1.7 References
Alberts,D.S.(2002)Informationagetransformation,gettingtoa21stcenturymilitary.Washington,
DC:DODCommandandControlResearchProgram,Retrievedfromhttp://www.dtic.mil/get‐tr‐doc/pdf?AD=ADA457904.
Allen,J.,&Curtis,P.(2011)MeasuresforManagingOperationalResilience.Pittsburgh,PA:Software
EngineeringInstitute,CarnegieMellonUniversity,Retrievedfromhttp://www.sei.cmu.edu/reports/11tr019.pdf
Beasley,M.S.,Branson,B.C.,&Hancock,B.V.(2010)BuildingKeyRiskIndicatorstoStrengthen
EnterpriseRiskManagement.Durham,NC:TheCommitteeofSponsoringOrganizationsoftheTreadwayCommission(COSO).
Bodeau,D.,&Graubart,R.(2011)MITRECyberResiliencyEngineeringFramework,MTR110237.
Bedford,MA:MITRECorporation,Retrievedfromhttp://www.mitre.org/sites/default/files/pdf/11_4436.pdf
Boyer,W.,&McQueen,M.(2008)IdealBasedCyberSecurityTechnicalMetricsforControlSystems.
Retrievedfromhttp://www.if.uidaho.edu/~amm/faculty/Ideal%20Based%20Cyber%20Security%20Technical%20Metrics%20for%20Control%20Systems.pdf
Black,P.,Scarfone,K.,&Souppaya,M.(2008)Cybersecuritymetricsandmeasures.In:Voeller,J.G.
(Ed.),HandbookofScienceandTechnologyforHomelandSecurity,Vol5,Hoboken,NJ:JohnWileyandSons,Inc.
Buldyrev,S.V.,Parshani,R.,Paul,G.,Stanley,H.E.,&Havlin,S.(2010)Catastrophiccascadeof
failuresininterdependentnetworks.Nature464,1025‐1028CIS(TheCenterforInternetSecurity)(2010)TheCISSecurityMetricsv1.1.0.EastGreenbush,NY:
TheCenterforInternetSecurity,Retrievedfromhttps://benchmarks.cisecurity.org/tools2/metrics/CIS_Security_Metrics_v1.1.0.pdf
Cimellaro,G.P.,Reinhorn,A.M.,&Bruneau,M.(2010)Frameworkforanalyticalquantificationof
disasterresilience.EngineeringStructures,32,3639–3649
22
Collier,Z.A.,Linkov,I.,DiMase,D.,Walters,S.,Tehranipoor,M.,&Lambert,J.H.(2014)Cybersecurity
standards:managingriskandcreatingresilience.Computer,47(9),70‐76Collier,Z.A.,&Linkov,I.(2014)Decisionmakingforresiliencewithinthecontextofnetworkcentric
operations.19thInternationalCommandandControlResearchandTechnologySymposium(ICCRTS),16‐19June,Alexandria,VA,USA.
DefenseScienceBoard(2013)Taskforcereport:resilientmilitarysystemsandtheadvancedcyber
threat.Washington,DC:OfficeoftheUnderSecretaryofDefenseforAcquisition,Technology,andLogistics,Retrievedfromhttp://www.acq.osd.mil/dsb/reports/ResilientMilitarySystems.CyberThreat.pdf
DiMase,D.,Collier,Z.A.,Heffner,K.,&Linkov,I.(2015)Systemsengineeringframeworkforcyber
physicalsecurityandresilience.EnvironmentSystems&Decisions,35(2),291‐300.Eckerson,W.W.(2009)PerformanceManagementStrategies:HowtoCreateandDeployEffective
Metrics.TDWIBestPracticesReport.Renton,WA:TheDataWarehousingInstitute.Retrievedfromhttps://tdwi.org/research/2009/01/bpr‐1q‐performance‐management‐strategies.aspx
ExecutiveOrderNo.13636,ImprovingCriticalInfrastructureCybersecurity,Retrievedfrom
http://www.gpo.gov/fdsys/pkg/FR‐2013‐02‐19/pdf/2013‐03915.pdfGanin,A.A.,Massaro,E.,Gutfraind,A.,Steen,N.,Keisler,J.M.,Kott,A.,Mangoubi,R.,&Linkov,I.
(2015)Resilientcomplexsystemsandnetworks:concepts,design,andanalysis.NatureScientificReports,submitted
Gregory,R.S.,&Keeney,R.L.(2002)Makingsmarterenvironmentalmanagementdecisions.Journal
oftheAmericanWaterResourcesAssociation.38(6):1601‐1612.Kaplan,S.,&Garrick,B.J.(1981)OntheQuantitativeDefinitionofRisk.RiskAnalysis,1(1),11–27Keeney,R.L.,&Gregory,R.S.(2005)Selectingattributestomeasuretheachievementofobjectives.
OperationsResearch53(1),1‐11Igure,V.,Laughter,S.,&Williams,R.(2006)SecurityIssuesinSCADANetworks.Computersand
Society,25(7),498‐506Linkov,I.,Eisenberg,D.A.,Plourde,K.,Seager,T.P.,Allen,J.,&Kott,A.(2013a)Resiliencemetricsfor
cybersystems.EnvironmentSystems&Decisions,33(4),471‐476Linkov,I.,Eisenberg,D.A.,Bates,M.E.,Chang,D.,Convertino,M.,Allen,J.H.,Flynn,S.E.,&Seager,T.P.
(2013b)Measurableresilienceforactionablepolicy.EnvironmentalScience&Technology,47(18),10108–10110
Linkov,I.,Bridges,T.,Creutzig,F.,Decker,J.,Fox‐Lent,C.,Kröger,W.,Lambert,J.H.,Levermann,A.,
Montreuil,B.,Nathwani,J.,Nyer,R.,Renn,O.,Scharte,B.,Scheffler,A.,Schreurs,M.,&Thiel‐Clemen,T.(2014)ChangingtheResilienceParadigm.NatureClimateChange,4,407–409
23
Manadhata,P.K.,&Wing,J.M.(2011)AnAttackSurfaceMetric.IEEETransactionsOnSoftwareEngineering,37(3),371–386.
Marr,B.(2010)HowtodesignKeyPerformanceIndicators.MiltonKeynes,UnitedKingdom:The
AdvancedPerformanceInstitute.Retrievedfromwww.ap‐institute.comMcIntyre,A.,Becker,B.,&Halbgewachs,R.(2007).SecurityMetricsforProcessControlSystems.
SAND2007‐2070P.Albuquerque,NM:SandiaNationalLaboratories,U.S.DepartmentofEnergy.
McKay,S.K.,Linkov,I.,Fischenich,J.C.,Miller,S.J.,&ValverdeJr,L.J.(2012)EcosystemRestoration
ObjectivesandMetrics,ERDCTN‐EMRRP‐EBA‐12‐16.Vicksburg,MS:U.S.ArmyEngineerResearchandDevelopmentCenter
Mell,P.,Scarfone,K.,&Romanosky,S.(2007)ACompleteGuidetotheCommonVulnerabilityScoring
SystemVersion2.0.Morrisville,NC:ForumforIncidentResponseandSecurityTeams.Retrievedfromhttps://www.first.org/cvss/cvss‐guide.pdf
NationalSecurityAgency(NSA).(2010).AFrameworkforAssessingandImprovingtheSecurity
PostureofIndustrialControlSystems(ICS).Retrievedfromhttps://www.nsa.gov/ia/_files/ics/ics_fact_sheet.pdf
Neely,A.,Richards,H.,Mills,J.,Platts,K.,&Bourne,M.(1997)Designingperformancemeasures:a
structuredapproach.InternationalJournalofOperations&ProductionManagement,17(11),1131‐1152
NIST(2014)FrameworkforImprovingCriticalInfrastructureCyberSecurity.Version1.0.
Gaithersburg,MD:NationalInstituteofStandardsandTechnology.Retrievedfromhttp://www.nist.gov/cyberframework/upload/cybersecurity‐framework‐021214‐final.pdfPfleeger,S.L.,&Cunningham,R.K.(2010)Whymeasuringsecurityishard.IEEESecurity&Privacy,
8(4),46‐54Pollet,J.(2002)DevelopingaSolidSCADAStrategy.Sicon/02–SensorsforIndustryConference.
Houston,Texas,USA.19‐21,November2002.Reichert,P.,Borsuk,M.,Hostmann,M.,Schweizer,S.,Sporri,C.,Tockner,K.,&Truffer,B.(2007)
Conceptsofdecisionsupportforriverrehabilitation.EnvironmentalModelingandSoftware22:188‐201.
Roege,P.E.,Collier,Z.A.,Mancillas,J.,McDonagh,J.A.,&Linkov,I.(2014)Metricsforenergy
resilience.EnergyPolicy,72(1),249–256Saltzer,J.H.(1974)ProtectionandthecontrolofinformationsharinginMultics.Communicationsof
theACM,17(7):388‐402.Stouffer,K.,Falco,J.,&Scarfone,K.(2011)GuidetoIndustrialControlSystems(ICS)Security.Special
Publication800‐82.Gaithersburg,MD:NationalInstituteofStandards.Retrievedfromhttp://csrc.nist.gov/publications/nistpubs/800‐82/SP800‐82‐final.pdf
24
USDepartmentofEnergy(2002)21StepstoImproveCyberSecurityofSCADANetworks.Washington,DC:USDepartmentofEnergy.Retrievedfromhttp://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/21_Steps_‐_SCADA.pdf
USDepartmentofEnergy(2009)NationalSCADATestBed:Enhancingcontrolsystemssecurityinthe
energysector.Washington,DC:USDepartmentofEnergy.Retrievedfromhttp://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/NSTB_Fact_Sheet_FINAL_09‐16‐09.pdf
Vespignani,A.(2010)Complexnetworks:thefragilityofinterdependency.Nature,464(7291),984–
985Williamson,R.M.(2006)Whatgetsmeasuredgetsdone:areyoumeasuringwhatreallymatters?
Columbus,NC:StrategicWorkSystems,Inc.Retrievedfromwww.swspitcrew.comZeng,W.,&Chow,M.Y.(2012)OptimalTradeoffBetweenPerformanceandSecurityinNetworked
ControlSystemsBasedonCoevolutionaryAlgorithms.IEEETransactionsonIndustrialElectronics,59(7):3016‐3025.
Zhu,B.,Joseph,A.,&Sastry,S.(2011).AtaxonomyofcyberattacksonSCADAsystems.InInternetof
things(iThings/CPSCom),2011internationalconferenceoncyber,physicalandsocialcomputing(pp.380‐388).
Top Related