USE METRICS DASHBOARDS TO MANAGE ENTERPRISE SECURITY … · security metrics as merely measuring...
Transcript of USE METRICS DASHBOARDS TO MANAGE ENTERPRISE SECURITY … · security metrics as merely measuring...
1
USE M ETR I C S DASHBOAR D STO M ANAGE
ENTER P R I SE SEC UR I TY R I SKS
Session 5302
2
S E S S I O N S U P P O R T E D B Y A S I S F O U N D AT I O N A N D A S I S D E F E N S E A N D I N T E L L I G E N C E C O U N C I L 2
PresenterPeter OhlhausenPresidentOhlhausen Research, Inc.
M E T R I C S D A S H B O A R D S
PresenterDaniel McGarveySenior Principal Business Process AnalystAlion Science and Technology
PresenterRichard WeaverChief Security OfficerHead, Security SvcsDept.Johns Hopkins University Applied Physics Laboratory
ModeratorCheryl StoneDirector, Corporate Security & SafetyRAND Corporation
3
O U T L I N E
I. ASIS Foundation metrics research project, Security Metrics Evaluation Tool, and ongoing research
II. Presenting metrics data to C-suite
III. Metrics dashboards for decision making and ROI demonstration
3
M E T R I C S D A S H B O A R D S
4
A S I S F O U N DAT I O N M E T R I C S R E S E A R C H
Persuading Senior Management with Effective, Evaluated Security Metrics (2014)
• Nine criteria for evaluating metrics--Reliability, Validity, Generalizability--Cost, Timeliness, Manipulation--ROI, Org. Relevance, Communication
• Library of evaluated metricshttps://foundation.asisonline.org
• Please contribute your metric at https://www.surveymonkey.com/r/metrics-survey
4
M E T R I C S D A S H B O A R D S
5
S E C U R I T Y M E T R I C S E VA L UAT I O N TO O L ( S E C M E T )
• Discern strong and weak points of a security metric
• Refine metric to optimize its scientific merit, operational reasonableness, and strategic relevance
• More persuasive to senior management
5
M E T R I C S D A S H B O A R D S
6
C R I T E R I O N 5 : T I M E L I N E S S
Extent to which metric data can be gathered in a timely fashion so the results can have an impact.
6
The data for this metric is
out-of-date by the time it
can be gathered and
interpreted; the data
collection process is very
time-consuming; the data is
unlikely to have an impact (as
it does not reflect current
conditions).
2
The data for this metric is
fairly up-to-date by the time
it can be gathered and
interpreted; the data col-
lection process is somewhat
time-consuming; the data is
somewhat likely to have an
impact (as it somewhat
reflects current conditions).
4
The data for this metric is very
up-to-date when gathered
and interpreted; the data
collection process is not time-
consuming; the data is very
likely to have an impact (as it
reflects current conditions).
1 3 5
7
I T ’ S I M P O R TA N T T O T H I N K A B O U T T H E WAY O N E M A K E S D E C I S I O N S
• Halo effect
• Outcome/hindsight bias
• Confirmation bias
• Regression to the mean
• Wet bias
7
M E T R I C S D A S H B O A R D S
A I M F O R L E S S W R O N G N E S S
8
K A R L P O P P E R ,P H I L O S O P H E R O F S C I E N C E
How can we hope to detect and eliminate error? By criticizing the theories or guesses of others and—if we can train ourselves to do so—by criticizing our own theories or guesses.
Conjectures and Refutations: The Growth of Scientific Knowledge, 1963
8
M E T R I C S D A S H B O A R D S
9
O B S E R VAT I O N S O N N E W M E T R I C S
• Metric of completed guard tours: does it discourage stopping to address a problem?
• Metric of driving time saved by conducting investigations long-distance: does it adequately consider quality factors or lean toward speed, convenience, and cost?
9
10
P R E S E N T I N G M E T R I C S TO C - S U I T E
Corporate management tends to view security as overhead (cost center, not production center) and security metrics as merely measuring activity, not value.
Security benefits are difficult to measure compared to the benefits of profit centers. Security professionals often lack the skills or time to create and administer effective metrics.
Thus, current security metrics, in practice, are generally not compelling and are often not taken seriously(Rothke, 2009).
10
11
P R ES E N T I N G M E T R I C S TO C - S U I T E
11
Make Metrics Compelling: an overview
Present metrics that are aligned with the organization’s objectives or risks or that measure the specific issues management is most interested in.
Present metrics that meet measurement standards.
Tell a story.
Use graphics, and keep presentations short.
Present metric data regularly.
12
P R ES E N T I N G M E T R I C S TO C - S U I T E
12
Align with Organizational Objectives and Risks
Risk: Metrics-based approach helps senior management understand the level of risk in site selection and make informed decisions on risk management.
ROI: There is a clear link between reducing shrinkage and saving money. Your metrics must demonstrate that investment in security technology led to reduced losses.
13
13
Risk Vs Return on Investment
14
P R ES E N T I N G M E T R I C S TO C - S U I T E
14
Present Metrics That Meet Measurement Standards
Metrics are quantitative and exude scientific authority. However, if metric is based on invalid or unreliable data, you cannot draw accurate conclusions from it and it will lack external credibility.
A metric that has been properly designed from a scientific point of view and that has been evaluated against a testing tool (such as the Security MET) may appear more valuable and persuasive to senior management.
Using a metric that meets measurement standards also provides an objectivity that aids decision-making.
15
© Pherson associates, llc. all rights reserved. www.pherson.org 15
Risk
Measurement Standards
16
P R ES E N T I N G M E T R I C S TO C - S U I T E
16
Tell a Story
Can be a story about the specific risk that security is attempting to mitigate, as well as the consequences if the event occurs. Be straightforward about risk and uncertainties.
Part of a compelling story is the unfolding of events over time. Metrics can show progress toward meeting a specific strategic goal.
Benchmarking can enrich a story if it is aligned with strategic organizational goals. Benchmarking provides the opportunity to ascertain where company stands on a given metric in relation to its competitors.
17
17
Security Threat
Risk Mitigation
18
P R ES E N T I N G M E T R I C S TO C - S U I T E
18
Use Graphics, and Keep Presentations Short
Keep it simple and clear. Present a few short bullet points—top-level information only, rather than complex charts and graphs.
Less is more.
Pick graphics that get your points across.
One graphic = 1,000 words.
Keep presentation short (but still tell a story).
Present metrics in the style or format management uses.
19
19
20
P R ES E N T I N G M E T R I C S TO C - S U I T E
20
Present Metric Data Regularly
Data ages over time.
Distinguishing metrics that are time-sensitive from those that provide value over time will enhance the overall value of metrics.
Comparing historical data against current data will show trends.
Do not hide painful data from management.
Good metrics are the key to demonstrate ROI.
21
S E C U R I T Y M E T R I C S : W H AT T O M E A S U R E ?S O M E G U I D I N G P R I N C I P L E S
• Be mindful that the process of collecting data and reporting metrics can be extremely time consuming and may unintentionally divert staff from performing work that needs to get done
• Therefore, confine metrics to only those things that provide useful insight into aspects of operations that are actionable and will lead to delivering improved service to customers and/or will reduce security risks
• Make every effort to determine the most critical concerns of senior management, and implement metrics that link to those concerns and that will demonstrate value and return on investment
• To the extent possible, leverage technology and automation to collect and analyze metrics data, thereby avoiding or minimizing manually intensive processes
21
22
T H E $ 6 4 , 0 0 0 Q U E S T I O N : W H AT TO M E A S U R E ?
Example: Enterprise Classified IT Security
- Number of Systems Administrators and ISSOs- Number of Systems (overall)- Number of Networked Systems- Number of WANs- Number of Classified SSP Submissions- Number of Incomplete SSP Submissions Returned - Number of Classified ATOs Received- Number of Users Trained- Number of Authentication Tokens Distributed - Media Write Access Authorized - Number of Privileged Users- Number of Authorized Data Transfer Agents- Number of Classified VTCs Conducted - Number of Mobile Devices- Number of IT related Security Violations and Infractions - Number of Systems Involved in a Classified Spill- Median Number of Days to Receive a Classified ATO- Results of Accreditation/Oversight Inspections- Results of Customer Satisfaction Surveys 22
23
S E L E C T I N G M E T R I C S T H AT B E S T F I T YO U R S E C U R I T Y O P E R AT I O N S ( V 3)
Volume
Numbers (counts) to track and assess level of security activity
Easiest to collect
Useful in defending, adjusting and seeking additional resources
Velocity
Data to capture and assess speed of delivering a security product or service
Useful in evaluating process efficiencies and identifying opportunities for improvement
Helpful in communicating expectations to customers, partners and stakeholders
23
24
S E L E C T I N G M E T R I C S T H AT B E S T F I TYO U R S E C U R I T Y O P E R AT I O N S ( V 3)
Value
Metrics to demonstrate the importance of Security to the overall health and productivity of an organization, capturing key care-abouts of senior management
Harder to identify, develop and measure
Highlights Return on Investment (ROI) by answering “so what” questions
May include Volume and Velocity data but will be outcome oriented
Helpful in providing high-level situational awareness of threats, vulnerabilities and success of mitigating countermeasures
Assists Senior Management in making decisions to accept risk, or to take action to lower risks
Displays/dashboards are useful, and anecdotes (stories, narrative example and explanations) are important to accompany numbers
24
25
S E L E C T I N G M E T R I C S T H AT B E S T F I TYO U R S E C U R I T Y O P E R AT I O N S ( V 3) :
S O M E E X A M P L E SVolume
Visit requests and clearance certifications processed; badges/tokens fabricated and issued; internal access control transactions; security incidents reported; foreign travel and other Security/CI awareness briefings administered
Velocity
Personnel Security clearance cycle time (nomination to indoctrination); IT accreditations (timelines associated with submission of plans to ATO); response time to alarm annunciations and other emergency circumstances
Value
Corporate savings (cost avoidance) attributable to security actions taken; security systems reliability; compliance inspection, audit and red team assessment results; elimination or reduction of undesirable events 25
26
D ATA TO D A S H B O A R DO N E WAY I T ’ S D O N E :
• Data is collected both in real time and on a periodic basis, depending on customer and senior management requirements, and on intended use in security operations, to include adjustment of resources
• Data is collected from a variety of sources: Excel spreadsheets, external SASS (Service Now) and other databases; subsequently using Microsoft SSIS, data is loaded and transformed into Microsoft SQL Server database
• Once data is collected and aggregated in a local data mart, then metrics are calculated and displayed via a SharePoint portal utilizing Power BI
26
27
27
28
28
29
29
30
30
31
31
32
32
33
33
34
34
35
35
36
36
37
37
38
P R E S E N T I N G P I C T O R I A L D I S P L AY S O F M E T R I C S D ATA T O S E N I O R M A N A G E M E N T :
T H E G O L D E N R U L E
CHARTS, GRAPHS, DASHBOARDS, DIAGRAMS, TABLES AND ILLUSTRATIONS SHOULD BE USED ONLY SELECTIVELY
AS A TOOL TO MAKE KEY POINTS
38