Optimizing Performance Management Gina Fisk, LANL Senior Cyber Security Manager gina@lanl
description
Transcript of Optimizing Performance Management Gina Fisk, LANL Senior Cyber Security Manager gina@lanl
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Adaptive Metrics Develop metrics that determine how well
we are adapting to our ever-changing environment.
Fitness Functions Identify dependencies and requirements
for optimum productivity around the Laboratory.
Measure the impact of a localized failure of one entity across the entire organization.
Balanced Score Card Review our program from a balanced
perspective. Provide metrics by which we can manage.
Optimizing Performance Management
Gina Fisk, LANL Senior Cyber Security [email protected]
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Starting Point – Remove the Clutter
Remove metrics that we can’t use to manage our information security program.
How many customers called our help desk. How many connections were deflected by our firewall. How many times our network was scanned, etc.
Bin the remaining metrics into the BSC framework for a Phase I BSC.
Financial. Customer. Internal Processes. Learning and Growth.
2
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Determine Impacts of Failure Conduct IT Impact Analysis
Determine the cost to an organization if various IT services failed for variable lengths of time.
Network, Email, local storage, etc.
Calculate Impact Rating for each IT Service. 1/n, where n is the average number of days until an
organization has lost 100% of productivity. Calculate the Daily Monetary Impact of the
Loss of that IT Service for an organization. Calculate the overall productivity cost for the
Laboratory as a whole based on that loss.
3
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Focus Areas From IT Impact Analysis results, identify IT
Services with largest impacts to productivity. Loss of Accreditation of systems Loss of local network access Loss of Email Loss of Oracle Loss of Internet access
Goals that the CIO and CISO had set for the organization in the Strategy Map.
Develop metrics based on these focus areas and develop Phase II of the BSC.
4
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Information Security Strategy Map
5
INTERNAL AND PROCESS PERSPECTIVE
LEARNING AND GROWTH PERSPECTIVE
LG1. Attract, develop, and retain highly skilled security
professionals
LG2. Develop risk-focused and customer-centric culture
LG3. Align employee training with strategic
initiatives
CUSTOMER PERSPECTIVE
Competency Contribution
IP1. Streamline compliance program to achieve 100% of
scheduled accreditationsIP2. Optimize operations to
reduce KTLO by 10% per BUIP3. Enhance performance through implementation and
management of service agreements
Achieve Operational Excellence
IP4. Mature IT governance processes and increase partner
participationIP5. Build a structured, transparent
and collaborative regulator relationship
IP6. Promote transparency and performance through holistic
metrics program
Create and Support Internal Programs and External Partners
IP7. Propose and deliver business-enabling information
security solutionsIP8. Mature IT risk program to drive security, portfolio, and
governance decisionsIP9. Enhance red network
monitoring and vulnerability management
Deliver Innovative Security Solutions
FINANCIAL PERSPECTIVE
Competitive Advantage Operational Excellence
C1. “Understand and consistently deliver
what I need”
C2. “Keep me out of security and
compliance trouble”
C3. “Establish a positive reputation
which will help me with my customers”
C4. “Become a trusted partner by helping me solve my challenging
problems”
F2. Maximize operational efficiency
F4. Facilitating acquisition of new business through best-in-class IT security execution
F3. Minimize IT enterprise risk
Maximize mission enablementby balancing risk and value (F1)
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Balanced Score Card
Financial (F1-F4)
Security unit costs
On-time rate of accreditations
Enterprise risk rating
Business impact of incidents
Projects on-time/budget
Cyber PBI ratings
Lower unit costs 100% on time Maintain .3
rating <25hrs/Q <10% variance >95% green
Target
Initiative
Customer (C1-C4)
Communication ComplianceCustomer Support
Program InputTime per
accreditationCustomer
Satisfaction
>80% survey scores
>70% survey scores
>80% survey scores
>90% governance participation
>95% CA/avg times
>80% survey scores
Target
Initiative
Internal Processes (IP1-IP7)
AOE: Opex reduction
AOE: SLA performance
CSIPP: unplanned
work
DISS: AOP risk mapping
DISS: BP tied to risk
DISS: Red capabilities
>=2.5% Q/Q <10% variance <=3/Q >=80%>=30% key processes
Positive trend
Target
Initiative
Hits target. Initiative on track
Short of target. Initiative recoverable
Failed process. Initiative not recoverable
Target not defined. No initiative
Learning and Growth (LG1-LG3)
Training roadmap
Planned role rotations
Attrition reduction
Strategic training
X X
<10% schedule variance
>=1/QReduced
attrition rate
>50% training mapped to initiatives
X X
Target
Initiative
Note: BSC target performance scores are represented here for explanatory purposes only
6
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Fitness Functions Fitness functions measure the
overall health of an organization by measuring not only performance, but also the performance of those organizations on which we are dependent to achieve our goals. If the performance of one of the dependencies fails, there are ramifications throughout the entire organization.
Using the fitness scores of dependent organizations, we can measure the impact of a localized failure of one entity across the entire organization, providing valuable measurements of the actual cost of security incidents, network outages, etc.
We can trend these scores to evaluate performance at various levels of the organization.
SystemAdministrati
on
SystemAdministrati
on
NetworkServicesNetworkServices
IdentityManagemen
t
IdentityManagemen
t
Scientific Computing
Core Services
Production Cycles
Production Cycles
Visualization Services
Visualization Services
Backups and Storage
Backups and Storage
Security Infrastructur
e
Science and Engineering
C&A Physical Infrastructure
Publications
ContractsPatents
Dep
en
den
cies
Relia
nt
Org
an
izati
on
s
7
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Example Fitness Function Framework Fiscal Responsibility (weight: 20%)
Milestones and deliverables (quality, timeliness) Expenditures (percentage over budget)
Customer Productivity (weight: 15%) Services maximize productivity around organization (uptime, etc)
Customer Orientation (weight: 15%) Responsiveness to the customer (SLAs, etc)
Improving Security (weight: 15%) Progress made toward improving security against our current threat
environment (hardening tools, etc) Institutional Responsibilities (Weight: 20%)
PBI deliverables and reporting (quality, timeliness) CAP deliverables and reporting (quality, timeliness) Metrics reporting (quality, accuracy)
Goal-Based Initiatives (weight: 10%) Progress made against organizational goals.
8
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Fitness Function Example
Fiscal Responsibility (weight: 20%) .89 Timeliness of deliverables and milestones: .83 % of projects +/- 10% of budget allocation: .95
Laboratory Productivity (weight: 15%) .98 Uptime of service: .98
Customer Orientation (weight: 15%) .89 Customer Satisfaction Rating .89
Improving Security (weight: 15%) ** .56** Progress made toward improving security against our current threat environment
(hardening tools, etc) .56 Institutional Responsibilities (Weight: 25%) .68
PBI deliverables and reporting (quality, timeliness) .90 CAP deliverables and reporting (quality, timeliness) .75 Metrics reporting (quality, accuracy) .40
Goal-Based Initiatives (weight: 10%) .98 Progress made against initiatives. .98
FITNESS SCORE: .806
9
Note: Fitness scores are represented here for explanatory purposes only
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Fitness Score Trends Fitness scores allow us to watch for trends and to manage by our metrics.
See how major changes affect our performance from month to month. Change in Management Change of Platform Change of Vendor, etc.
10
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Adaptive Metrics IT Impact Analysis provide us with costs of the
failures of IT Services. We have the data on our ever changing threat
environment. The fitness functions allow us include “moving
target” metrics, which change each month, to measure our performance against our current threat environment.
11
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
**Adaptive Metric Example
“Improving Security (weight: 15%)” Identify top threats for the month.
Phishing, Windows vulnerability, Oracle vulnerability. Calculate cost of failure of these services across the
organization per month. Email: $200K, Windows: $500K, Oracle: 800K Overall Budget: 10% in jeopardy
Review % of security effort we are placing on these areas ($$$ spent).
Email: 5%, Windows: 31%, Oracle: 20% Weight the fitness function by how responsive we are to
these areas. 56% of our budget is spent on our top threat areas.
12
Note: Threats and budgets represented here for explanatory purposes only
Operated by Los Alamos National Security, LLC for NNSA
U N C L A S S I F I E D
Managing by the Metrics
Our budget, metrics, and initiatives are actionable and directly tied to our goals.
Our use of the Balanced Score Card helps us ensure uniform management of our business.
Our use of the Fitness Functions help us trend our metrics effectively and monitor the major changes.
We can trend our components individually or as a whole, organizationally or institutionally.
Our use of Adaptive Metrics keep our outlook fresh and defendable.
13