Applied R for the quantitative social scientist - Curving Normality
Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk...
Transcript of Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk...
![Page 1: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/1.jpg)
Applied Quantitative Cyber Risk AnalysisMichael Rich, OSCP, CISSPDirector of IT Security, Infrastructure & OperationsMotion Picture Industries Pension & Health Care Plan
![Page 2: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/2.jpg)
| 2 |
Disclaimer for those reading from the ISACA link
My talks are image and slide-build heavy. So they don’t “print” well. Sorry about that.
![Page 3: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/3.jpg)
| 3 |
Agenda
Seek Beyond Your Interest– “Risk Analysis: Don’t Just Trust Your Gut” – Evan Wheeler @
BSidesLA 2016 The Idea:
– What is a Risk?– The Calibration of the Experts– Monte Carlo Risk simulation– A Cyber Risk Model Example The Application:
– Risk Decomposition– Gedanken Experiments– “The SHOCKING truth about probability they don’t want you to
know!!!”– Snowflakes and Monte Carlo– Equivalent Life Event Probabilities Now What?
![Page 4: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/4.jpg)
| 4 |
The Idea
![Page 5: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/5.jpg)
| 5 |
What is a Risk?
An event that has some chance of happening and causes effects we don’t want.
Qualitative Analysis
Source credit: https://www.risklens.com/blog/4-steps-to-a-smarter-risk-heat-map
Quantitative Analysis
![Page 6: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/6.jpg)
| 6 |
What is a Risk?
Probability of Occurrence– Numerically-expressed probability– Can be a range to express uncertainty i.e.: 9-14% chance
Impact (Loss)– Numerically expressed range: Upper bound Lower bound 90% confidence
– Used with a log-normal distribution 5% values are < Lower bound 5% of values are > Upper bound Black Swans!
Log-normal distribution example
![Page 7: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/7.jpg)
| 7 |
Log Normal – In Real Life
Image from Blackline.com
![Page 8: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/8.jpg)
| 8 |
What is a Risk?
Estimated over given time period A basic risk:
– Tomorrow, if a traffic incident occurs during my morning commute, I will be delayed– Probability of occurrence: 30%– Impact (90% confidence): 5 – 60 minute delay from normal commute time
![Page 9: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/9.jpg)
| 9 |
Subjective Range EstimationAKA The Calibration of the Experts
The Equivalent Bet: for 1000 Imperial Credits would you rather– See if the answer is in your interval– Spin the dial?
Win it all
Win nothing
What is the stated capacity of Wembley Stadium in London?
Capacity: 90,000
This slide covered on purpose so we don’t ruin the fun at the event!!
![Page 10: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/10.jpg)
| 10 |
Monte Carlo Simulation
Iterate over probability of occurrence and generate random impactsMany times (100K+)
Probability: 30%Impact, Upper bound: 60Impact, Lower bound: 5Number Trials 10001Trial Delay
1 02 14.552443 17.377024 16.649685 06 07 08 09 0
10 49.68741
Example:
![Page 11: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/11.jpg)
| 11 |
Sim Results and the Loss Exceedance Curve
![Page 12: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/12.jpg)
| 12 |
Reducing Loss Exceedance Curves
Curves are pretty, but I need a number!– Ranking– Comparison– Mitigation effectiveness
In insurance world:– Average Annual Loss = Premium– “Area under the curve”
For Commute:– Average Event Impact– 6.8 minutes…. But…
241 Minute MAX impact
![Page 13: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/13.jpg)
| 13 |
Methodology Demonstration – The Shared Home ComputerCost chosen as impact only for purposes of this example
Banking Trojan
Probability 5%
Max Impact $25,000 ($35,000)
Min Impact $500
Ransomware
Probability 10%
Max Impact $3000
Min Impact $200
Creepy Spyware
Probability 2%
Max Impact $2000 ($5000)
Min Impact $300
Clumsy Cat
Probability 5%
Max Impact $3000
Min Impact $750
Amazon Spree
Probability 30%
Max Impact $750
Min Impact $150
Risks over next 6 months
![Page 14: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/14.jpg)
| 15 |
Simulation Results (100K iterations)Use Case: Ranking Risks
Total Expected Average Loss
$638
Banking Trojan $317
Amazon Spree $112
Ransomware $110
Clumsy Cat $80
Creepy Spyware $19
![Page 15: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/15.jpg)
| 16 |
The Application
![Page 16: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/16.jpg)
| 17 |
Risk Decomposition
Break your risk effects down into chunks– Measureable and observable– Company dependent
Manpower Costs– Business Departments– Leadership
Remediation Costs– IR Retainer– Legal– Hardware– Software
![Page 17: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/17.jpg)
| 18 |
Risk Decomposition
LBUBCapLBUBCap
LBUBCapLBUBCap
LBUBCapLBUBCap
$/Hr
TimeSecurity
Active?
Time
$/Hr
IT Leadership
Active?
Time
$/Hr
IT Ops
Active?
LBUBCapLBUBCap
LBUBCapLBUBCap
LBUBCapLBUBCap
Time
$/Hr
Retirements
Active?
Time
$/Hr
PSC
Active?
Accounting
Active?
Time
$/Hr
LBUBCap
LBUBCap
LBUBCap
LBUBCap
IR Retainer
Active?
Cost
Legal
Active?
Cost
Active?
CostHardware
Software
Active?
Cost
![Page 18: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/18.jpg)
| 19 |
Gedanken Experiments
![Page 19: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/19.jpg)
| 20 |
The ONE SHOCKING Truth About Probability
Aggregate probability is a bitch… 2 times in 120 days, I escalated a security event to the CIOWhat are the odds I have to escalate an issue any given day:
– Odds: 2/120– Probability [Odds/(1+Odds)]: 1.64%
What is the probability (p) I’ll have an event in the next 6 months I have to escalate?Well:
– Probability (p-not) of it not happening [1-p]: 98.4%– Probability of it not happening for 120 consecutive days [(p-not)^120]: 14.4%– Probability of an escalated event in 120 days [1-(not happening)]: 85.6%
![Page 20: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/20.jpg)
| 21 |
Is Monte Carlo a Precious Snowflake?(Sensitivity Analysis)
3 independent variables. How sensitive is the Average Event Loss?Probability Lower Bound Upper Bound
![Page 21: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/21.jpg)
| 22 |
Monte Carlo IS a Precious Snowflake.. Probably
![Page 22: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/22.jpg)
| 23 |
Ooof.. It’s Even Worse Than I Thought
![Page 23: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/23.jpg)
| 24 |
Handling the Snowflake
Must include uncertainty in your probability estimate (i.e. a range) Instead of 1% probability, let’s make it 0.5% - 1.5% (50% error bar)
Test AEL($)
1% Fixed $72
1% +/- .5% $70
![Page 24: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/24.jpg)
| 25 |
Beta Distribution
Single: $71.79Uniform: $71.15Beta: $71.63
Test EAL ($)
1% fixed $71.79
1% +/- 0.5% $71.15
1% Beta $71.63
![Page 25: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/25.jpg)
| 26 |
Some More Experiments
Test EAL ($)
5% fixed $367
5% +/- 4% $355
5% Beta $356
![Page 26: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/26.jpg)
| 27 |
Some More Experiments
Test EAL ($)
5% fixed $350
5% +/- 4% $349
4% +/- 3% $293
4% fixed $277
![Page 27: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/27.jpg)
![Page 28: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/28.jpg)
| 29 |
Statistically Equivalent Probabilities
100% - 50% 50% - 10% 10% 3% 1.5% 1% 0.8% 0.02%
![Page 29: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/29.jpg)
| 30 |
Beta Distribution: Establish Probability from Test Cases
If you have a set of cases, you can get a probability distribution
![Page 30: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/30.jpg)
| 32 |
Using Probability for Complicated Scenarios
Calibrate expertAsk expert to assess probability of the event given no other data
– “What is the probability of an adversary managing to inject code on the target system in the next 6 months?”
Ask expert to re-assess given various conditions– “What if the firewalls are discovered to be misconfigured?”– “What if a Cooperative Vulnerability Inspection team demonstrates code injection?”– “What if a black-box adversarial assessment team demonstrates it?”
Use Log-Odds-Ratio– Statistically valid method for combining the effects of multiple conditions on a final
probability
![Page 31: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/31.jpg)
| 33 |
Log Odds Ratio ExampleUse Case: Using expert knowledge
Initial Prob: P(E) 1.0%
Firewall CVI Finding Adversarial Software Adversary Intel Rogue Network Device Rogue USBP(E|X1) Correct Config Code Injection Code Injection Updated 1 Hop away Detected DetectedP(E|X2) Incorrect Config No Injection No Injection Not Updated 2 Hops away Not detected Not DetectedP(E|X3) 3+ Hops awayP(E|X4)
P(E|X1) 0.5% 5.0% 10.0% 1.0% 10.0% 1.0% 1.0%P(E|X2) 2.0% 0.5% 0.5% 5.0% 3.0% 15.0% 15.0%P(E|X3) 1.0%P(E|X4)
Condition State Which Applies Correct Config No Injection Code Injection Updated 1 Hop away Detected Detected
Conditional Probability 23.2%
Conditions
![Page 32: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/32.jpg)
| 34 |
Now What?
For Me– Solidify my risk decompositions– Identify my events to analyze– Calibrate my team– Model and Simulate– Submit Blackhat ‘18 paper For You
– Go read Hubbard’s book– Go get my code: https://github.com/richmr/QuantitativeRiskSim– Think about your decompositions– Identify your events– Model and Simulate– Come watch my Blackhat ‘18 presentation
![Page 33: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/33.jpg)
| 35 |
Summary
Quantitative risk modeling can be a reality in Cybersecurity– Use Case: Risk ranking and prioritization– Use Case: Assessing control audit results– Use Case: Mitigation comparison– Use Case: Quantifying expert knowledge on complex systems– Use Case: Test planning
Networks can improve its cybersecurity… Measurably! Python Simulation Code available at:
– https://github.com/richmr/QuantitativeRiskSim
![Page 34: Applied Quantitative Cyber Risk Analysis County/IIA OC... · Applied Quantitative Cyber Risk Analysis. Michael Rich, OSCP, CISSP. Director of IT Security, Infrastructure & Operations.](https://reader035.fdocuments.net/reader035/viewer/2022070701/5e5c0e589ccf1d4ad61935ed/html5/thumbnails/34.jpg)
| 36 |