Cyber Measurement Campaign
Transcript of Cyber Measurement Campaign
Cyber Measurement Campaign
Charles Wright, Lee Rossey
Presented at the ITEA Technology ReviewPresented at the ITEA Technology Review
25-27 July 2012
This work is sponsored by the Assistant Secretary of Defense for Research and Engineering under Air Force Contract FA8721-05-C-0002. Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the United States Government.
Goals
• Meaningfully track progress in research to improve cyber security
• Challenge assumptions and encourage new ideas
• Quantitatively measure key aspects of cyber systems– Resiliencyy– Agility– Trustworthiness– Mission Effectiveness
CMC Overview - 2CVW 07/2012
Outline
• Survey of cyber testbeds and ranges
• Emerging cyber range architecture
• Initial experiments with cyber resilient systems
CMC Overview - 3CVW 07/2012
Approaches to Cyber Assessment
Modeling & Simulation Cyber Range Prototype DeploymentAnalysis
• Real code, real apps, • Validates modeling,• Based on first principles • Fidelity/complexity/time
A l i Modeling & C b R Prototype
Real code, real apps,emulated environment
• Repeatable• Provide users with a real time
implementation for evaluation
Validates modeling, simulation, and emulation
• Difficult to repeat• Difficult to obtain ground
truth
• Develops global performance intuition
• Provides bounds that serve as implementation goals
• Provides corner cases to validate modeling, simulation, and
trade-off• Repeatable • Easiest transfer
across organizations
Analysis Modeling & Simulation Cyber Range Prototype
Deployment
Fidelity Low Low Moderate to High High
Scalability High High Moderate Low
Cost Low Low Moderate High
emulation
Cost Low Low Moderate High
Repeatability N/A High Moderate to High Low
Program Phase Early Early Mid-term Mid-term to Late
Selecting an appropriate combination of assessment approaches is critical to a
CMC Overview - 4CVW 07/2012
• Selecting an appropriate combination of assessment approaches is critical to a successful quantitative evaluation
• Study focused on Cyber Range
Core Cyber Range Infrastructure Elements
Sit /R
• What is the site/range purpose, mission?• Who do they primarily support?
Site/Range • Do they have a robust security infrastructure?
• Trained range personnel?• Trained operators (e.g. mission)?• Can they support open-air?
Range AssetsRange Tools • What equipment to they have?
• What level of scale, fidelity?• Is it representative?• Is it realistic?• Is it unique?
• What range tools do they have?• What is their sophistication? • Is it unique?
• Is it cyber targetable?
• All three areas must be considered when assessing cyber ranges
• What is their sophistication?• Are they custom, validated?
CMC Overview - 5CVW 07/2012
All three areas must be considered when assessing cyber ranges• Strength in all areas increases the value of the range
Elements of a Cyber RangeTest Setup:
Specify Configure Validate Baselinep y• Network Layout• Background Traffic• Attack Scenario• Defensive Posture• Data Collection
g• Hosts• Network• Users• Internet
• Network Buildout• Host and site
Reachability
• Traffic generation• Defensive Tools• Data Collection
Test Scenario:
Red Mission Modeler
CommercialCafe Blue Mission Modeler
Traffic Generation
HQ
Protected
Internet
Att k
Traffic Generation
Att k
Situational AwarenessMonitor
MilitaryNet
Tactical Tactical
Military
Power
Gov’tCommercial
Attacks Attacks
AwarenessDefensive Tools
Verify, AssessTactical
Test Environment
CMC Overview - 6CVW 07/2012
Data Analysis:Collect
Data AnalyzeStore Test
Configuration, Data, Results
Report
Cyber Range Inventory Summary
• Dozens of cyber testbeds and ranges across the US– Many small, special-purpose labs– A few large cyber ranges available for general use
• Mostly run by government military and academiaMostly run by government, military, and academia
• Many different tool sets, missions, and focus areas
CMC Overview - 7CVW 07/2012
Overall Cyber Ranges Assessment
• Many ranges and testbeds are currently available to support pure cyber development, experimentation and testing
• Technology advancements are being made to improve range automation and sophistication (e.g. DARPA NCR)– Technology can be transitioned to other ranges
• Traditional kinetic ranges are very mature for traditional missions
• Secure wide area networks (e.g. Joint IO Range) are essential to provide connectivity between users and cyber and kinetic ranges
• No common standards or interfaces exist between cyber ranges orNo common standards or interfaces exist between cyber ranges or range capabilities– Incompatible and fragmented systems and tools
S f f
CMC Overview - 8CVW 07/2012
• Significant investments are still required to fill the capability gaps
Example Academic Testbed: Emulab
• Name: Emulab
• Host Org: University of Utah
• Purpose: Open research in networking and security
• Scale: 13 racks of equipment– 500+ PC servers – 12 network switches– 802.11 wireless– Software-defined radio (GNURadio, USRP)– Programmable network interfaces (NetFPGA)
CMC Overview - 9CVW 07/2012
Outline
• Survey of cyber testbeds and ranges
• Emerging cyber range architecture
• Initial experiments with cyber resilient systems
CMC Overview - 10CVW 07/2012
Cyber Range Architecture Goals
From discussions with the cyber T&E community, it became clear that a vision is emerging for cyber ranges that offer:
• Full end-to-end integration of cyber and kinetic systems– Integrate military, civilian and critical infrastructure
Secure closed loop open air test environment– Secure, closed-loop, open-air test environment– Support integrated testing of cyber and traditional military systems
Support for experimentation development testing training and• Support for experimentation, development, testing, training and exercises across DoD and national agencies
• Cyber range technology that isy g gy– Open, standards based– Interoperable, extensible, robust, scalable– Support varying levels of fidelity
Deplo able to contractors and de elopment laboratories
CMC Overview - 11CVW 07/2012
– Deployable to contractors and development laboratories
Cyber Range Architecture Components
Users
National
Technology Development
Range Services
Army PrototypesAgencies CapabilitiesAir Force Navy
Core Principles: Open Standards
Network
… Field TestsConcepts
Traffic
DefensivePackages
AssetManagement
TestsScheduler
Health &Status
Range
ModelsRangeAutomation
ThreatRange
DistributedOperations
Command &
Visualization Data Collection
Metrics
Data Archival
Instrumentation
D A l i
EnvironmentEmulation
Mission TrafficEmulation
TrafficGeneration
RangeSanitization
EnvironmentRange Setup, Control, Operation
Network
ThreatPackages
RangeValidation
Command &Control
Core Services
Data Analytics
Resources
…
Adapter Adapter Adapter Adapter Adapter Adapter
TargetableAccessible, IntegratedAdapter
CMC Overview - 12CVW 07/2012
WebTransportationMaritimeSatelliteAirborneRadars SCADA Radars Ships/MissilesTelecom
Range Architecture Illustration
Establish a set of major cyber-capable ranges
(Locations shown below are for example only)
Key Technologies:• DARPA NCR• JIOR RSDP
CMC Overview - 13CVW 07/2012
* Notional site selection
• JIOR RSDP
Range Architecture Illustration
Leverage secure wide-area range networks to connect geographically distributed sites for each experiment
Key technologies• Joint IO Range (JIOR)g ( )• Joint Mission Environment Test Capability (JMETC)
Key Technologies:• Joint IO Range• JMETC
Key Technologies:• DARPA NCR• JIOR RSDP
CMC Overview - 14CVW 07/2012
* Notional site selection
• JIOR RSDP
Range Architecture Illustration
Pool resources from all connected ranges to form a unified, distributed virtual testbed
Key technologies• DARPA NCR• TENA
VM Servers, Experiment Nodes Key Technologies:
Data Collection,Analysis, andVisualization
Range Automation
Range Adminand C2
Users
y g• DARPA NCR• TENAOPFOR
Key Technologies:• Joint IO Range• JMETC
Key Technologies:• DARPA NCR• JIOR RSDP
CMC Overview - 15CVW 07/2012
* Notional site selection
• JIOR RSDP
Range Architecture Illustration
Key Technologies:• Traffic Generatio• Emulated
Execute experiments on the testbed
• Emulated Components
VM Servers, Experiment Nodes Key Technologies:
Data Collection,Analysis, andVisualization
Range Automation
Range Adminand C2
Users
y g• DARPA NCR• TENAOPFOR
Key Technologies:• Joint IO Range• JMETC
Key Technologies:• DARPA NCR• JIOR RSDP
CMC Overview - 16CVW 07/2012
* Notional site selection
• JIOR RSDP
Approaches to Connect Cyber with Kinetic Assets
Cyber Range(s)Targetable Kinetic Resources
Integration with Kinetic AssetsTargetable Kinetic Assets
Access to integrated and targetable
Access to integrated kinetic
Resources
Cyber Range(s)
IO Range JMETCIO
Range JMETC
Integrated Cyber Kinetic Cyber Range w/ Access to Kinetic
targetable kinetic resources
resources
Integrated Cyber-Kinetic Cyber Range w/ Access to Kinetic• Full control over use and scheduling of targetable
kinetic resources• Can support experimentation • Can support highly classified, contained cyber tests
on kinetic assets
• Expertise and operations maintained on the kinetic ranges• Targetable assets can be fully integrated with other weapon
systemsPros
on kinetic assets
• Requires experts from many domains to operate suite of systems
• Competition with other ranges for FME/FMA assets
• Traditional ranges may not have cyber expertise to support integrated testing
• Potentially difficult to schedule use of kinetic assets (competing priorities)
• Potentially difficult to integrate targetable kinetic systems
Cons
CMC Overview - 17CVW 07/2012
y g g y• Potential reluctance by traditional ranges to support
experimentation or destructive testing on kinetic assets• Must be able to execute secure distributed testing (e.g. range
control, instrumentation, analysis) across systems
Outline
• Survey of cyber testbeds and ranges
• Emerging cyber range architecture
• Initial experiments with cyber resilient systems
CMC Overview - 18CVW 07/2012
Initial Experiments
• Primary Goals– Demonstrate experiments to measure and quantify cyber resiliency
with mature research prototypes– Measure resulting improvement to cyber security
• Availability, confidentiality, integrity
• Focus is on experimentation, not test & evaluation– This is not a test! No pass or failp– Not (yet) intended to be comprehensive or complete– No attempt (yet) to assess the overall security of the system– No attempt (yet) to assess impact to real missionsp (y ) p
CMC Overview - 19CVW 07/2012
Three Technologies for Initial Experiments
• ARCSYNE• IP hopping IPSec VPN gateway• Protects a closed community of interest from external threats• Protects a closed community of interest from external threats
• LPS: Lightweight Portable SecurityB t bl Li li CD f ti it f tiF f • Bootable Linux live CD for continuity of operations
• Approved by DOD for use in case of pandemic flu, etc.Focus ofthis talk
• TALENT• Dynamically composable platforms• Enables applications to seamlessly migrate from one hardware-
software platform to anothersoftware platform to another
CMC Overview - 20CVW 07/2012
Lightweight Portable Security
• Linux-based OS on a bootable CD-ROM– Includes web browser, VPN, office
software suite– Approved by DOD for use in maintaining
continuity of operations
• Improved security over standard desktop systems• Improved security over standard desktop systems
– Minimal software included
– No persistent storage
CMC Overview - 21CVW 07/2012
Cyber Kill Chain
Attacker seeks to carry out a sequence to steps to achieve his goal
Develop Attack
Recon
to steps to achieve his goal
Launch Attack
Persist & Establish
C2D5 Effects
or Exfil
Assess Eff t
Defender attempts to block his progress at every step
Effects
CMC Overview - 22CVW 07/2012
Cyber Kill Chain
Attacker seeks to carry out a sequence to steps to achieve his goal
Develop Attack
Recon
to steps to achieve his goal
Launch Attack
Persist & Establish
C2D5 Effects
or Exfil
Assess Eff t
LPS makes Attack Development and Persistence more difficult for the EffectsPersistence more difficult for the
attacker
CMC Overview - 23CVW 07/2012
Cyber Kill Chain
Develop Attack
Recon
Launch Attack
Persist & Establish Improvement is very
diffi lt t C2D5 Effects
or Exfil
Assess Eff t
difficult to measure
Underlying science: Theory of Computation
Challenge: EffectsChallenge: Analysis of software is still an unsolved problem Improvement is much
easier to measure
Underlying science:
CMC Overview - 24CVW 07/2012
Underlying science:Queuing Theory, Stochastic Processes
LPS Experiment
Hypothesis: Increasing recovery rate increases the time required for an attacker to penetrate the network
Experiment Outline: Multiple LPS hosts communicate with a remote network, while an attacker attempts to gain a point of presence on a fixed percentage of the LPS clients
Threat model: We assume that the attacker– Can gain a point of presence on a LPS machine though a remote software exploit
– Cannot persist in hardware (BIOS, flash, etc.)
CMC Overview - 25CVW 07/2012
LPS ExperimentVariables of Interest
– Session Length (influences recovery-rate)– Percentage of hosts required for coordinated attack (influences workload)
Measurements: – Workload: Time to gain a point of presence on a fixed percentage of hosts– Resilience: Rate of recovery
acke
r Del
ayIn
crea
se in
Atta
CMC Overview - 26CVW 07/2012
Resilience: Recovery Rate
LPS Experiment Configuration
Custom Firefox extension module simulates
Hypothesis: Increasing resilience (recovery rate) increases attacker’s delay
55 Virtual machines running LPS CD-ROM image40 Web sites on an emulated Internet
Custom Firefox extension module simulates a security vulnerability in the browser
1 Malicious site serving “drive-by” malware downloads
Malicious site includes a special “exploit” header that causes the simulated vulnerability to download and run a program (the “payload”) from the attacker’s site.
4 Dell PowerEdge R410 VM servers
1 D ll k t ti i l ti 55 i t l
Simulated exploit payload spawns a new process on the victim machine and sends a constant stream of packets until the machine is rebooted.
1 Dell workstation simulating 55 virtual users
Each “user” browses the web continuously, andreboots his VM intermittently with Poisson rate R.
Users select web pages uniformly at random. Each user views his selected web page in its
Experimental Methodology• Vary reboot rate R
• Average session length: 2 hr, 1 hr, 30 min, 10 min• Let each scenario run once for 22 hours
• Observe impact on attacker delay time
CMC Overview - 27CVW 07/2012
Each user views his selected web page in its entirety, and stays on a single page for no more than 4 minutes at a time.
• Observe impact on attacker delay time• Observe impact on attacker’s success rate
Results: Increase in Attacker Delay
Average S i
Resulting AvgI f ti
Recovery Rate ( i
16
18
s)Session Length
Infection Length
(recoveries per machine per hour)
2 hr 83 min 0.725
1 hr 49 min 1.22 8
10
12
14
To infect 10%
To infect 20%
er D
elay
(hou
rs
30 min 24 min 2.51
10 min 8 min 7.49
0
2
4
6 To infect 30%
Atta
cke
Increased recovery rate due to frequent reboots does not prevent th tt k f i i i iti l f th ld i t th t k
02 hr 1 hr 30 min 10 min
Average Session Length
the attacker from gaining an initial foothold into the network. (No increase in time required to infect 10%.)
Recovery rate has a profound effect on the workload required to achieve substantial penetration of the network.
CMC Overview - 28CVW 07/2012
p(22x increase in time required to infect 30%.)
Experimental Results (2)
Rebooting more frequently reduced the attacker’s average penetration of the network by a factor of 4.
CMC Overview - 29CVW 07/2012