Establishing Order in Time of Chaos: IT Incident Management

March 3, 2016 Heather I. Roszkowski, MSIA, CISSP

Chief Information Security Officer

Conflict of Interest • Heather I. Roszkowski

• Has no real or apparent conflicts of interest to report.

Agenda • Why Do We Need IT Incident Management? • What is the Incident Command System • The IT Incident Management Plan

Learning Objectives • Identify the need for organized incident response • Recognize the value of a formal incident management plan • Identify attributes of a strong incident management plan • Apply the knowledge to design an incident management plan for your

organization

Why Do We Need IT Incident Management?

June 2014 - Enterprise Incident • Initial reports described errors accessing an external site • ~1 hr 43 mins later IT Command Center established • ~3 hrs after incident Center Hospital Command Center partially

enabled • 15 hrs after first reported incident service restored

June 2014 – Current Procedures • IT Command Center opened via conference bridge line

– Line remains open for the duration of incident – All leaders remain on the line

• Incident Communications – Primarily done via email and paging – Pop up AlertMe application

• IS Leader on-call designated as Incident Commander – Communications Lead selected during call – Technical Lead selected

June 2014 - Challenges • IT Leadership was on conference call for the 15 hour command

center • Outage impact was not clear • IT outage impacted primary communication tools used to inform

users of the outage – Limited email and paging – Pop-up notification tools were down

• No printing was available because print servers were offline • Hospital Command Center was not familiar with IT incidents • Technical leaders were being asked for updates every 5-10 mins

June 2014 - Results • As a result of the incident the CIO directed the formation of the:

Crisis Incident Management Committee • Deb Dulac, Director – PRISM and Business Systems • Nate Jewett, Manager – Infrastructure Systems • Corey Mercy, Manager – PRISM Inpatient and Systems

(Committee Lead) • Heather Roszkowski, Chief Information Security Officer • Mike Wilson, Interim Chief Technology Officer • Julie Sloma, Communications Strategist

What is an IT Incident “An unplanned interruption to an IT Service or reduction in the

quality of an IT service.” ITIL (2011)

“unplanned interruption to a service, a reduction in the quality of a

service or an event that has not yet impacted the service to the customer”

ISO 20000-1 (2011)

(Incident Management ITSM)

The Cost of IT Downtime Average unplanned downtime costs $7,793 per minute.

Ponemon Institute (2016)

Average company loses $140,003 to $540,358 in revenue per incident.

Avaya Inc. (2014)

Clinical Cost of IT Downtime The deployment of IT systems within Healthcare have established greater dependencies and reliance on these technology services for providing clinical and operational functions. How long can your Pharmacy Department function without IT systems?

Image courtesy of bluebayat FreeDigitalPhotos.net

During an incident – most everyone wants to help… The key is managing these resources so they are effectively utilized and focused where needed.

Image courtesy of bluebay at FreeDigitalPhotos.net

Challenges to Effective IT Incident Management

• Uncoordinated efforts in troubleshooting and resolving the issue • Balancing restoration of service efforts with mitigation and customer

support needs • Inaccurate or misleading information • Keeping focus and using all available resources effectively

Image courtesy of jscreationzs at FreeDigitalPhotos.net

Incident Command System

What is the Incident Command System (ICS) • Developed in the 1970s after catastrophic wildfires in California • Provides a standardized management tool regardless of the size or

complexity of the situation • Symbolizes industry “best practices” and has become the accepted

“standard for emergency management across the country” • Can be used for a various types of events from natural disasters to

terrorism • Provides a common terminology and organizational structure

FEMA (2008)

ICS in Action • Provides organizational structure, regardless of the nature of the

incident • Identifies who is managing the overall incident – the Incident

Commander • Fire Service uses ICS on every call from a minor car accident to

multi-structure fires • Many Hospital Emergency Operations Plans also utilize ICS • Code Blue – structured in similar manner with defined roles and

responsibilities

Image courtesy of Surachai at FreeDigitalPhotos.net

So how does ICS relate to IT Incident Management? The Incident Management Plan

Image courtesy of Stuart Miles at FreeDigitalPhotos.net

IT Incident Management Plan

• Provides direction and guidance in managing an incident

• Defines the roles and structure used in every IT Command Center

• Developed with a Continuous Process Improvement model inherent in the framework.

• Has a Debriefing and Post-Incident Analysis structure to obtain operational feedback for plan assessment and iterative improvements to the plan.

Sections of the Plan • Assessment and Activation • Command and Control Structure • Communications • Debriefing and Post-Incident Analysis • Training • Security Incidents • IT Computer Incident Response Team

Image courtesy of cooldesign at FreeDigitalPhotos.net

Assessment and Activation • Assessment process structured to:

– Give immediate thought to organizational impact – Ensure appropriate resources are engaged – Identify potential security event

• Activation of IT Command Center

– Involves IS Leader On-Call, IS Domain Knowledge Resource and Service Center

– Conference Call numbers dedicated to Incident Management – Includes Administrator On-Call, Physician Leader On-Call and

Administrative Nurse Coordinator when on duty – Primary and Secondary Command Center locations pre-established

Assessment and Activation • At the initial IT Command

Center activation an incident briefing is provided to the group

• Roles will be identified and filled

• Physical Command Center location will be identified if needed

• Frequency and time of next status update will be established

• After incident briefing, resources not identified for activation can drop from the Command Bridge and will dial in for status updates as needed

Command and Control IS Command Roles: Incident Commander Command Scribe • Responsible for the overall coordination and management of the

incident – Command. • IT Incident Commander must be established for every IT Incident

Command Center • Any communications to be distributed outside of the Incident

Management Team must be approved by the Incident Commander.

Command and Control IS Operations Section Roles: Operations Chief Operations Scribe • Focused on the clinical and business operational impact of the

incident, provides timely workflow communications, and coordinates at-the-elbow support staffing.

• Manages the application and operational resources engaged in the incident.

Command and Control IS Technical Section Roles: Technical Chief Technical Scribe • Focused on identifying the “Keystone” incident and its resolution,

along with any associated incidents that arose because of this. • Manages the technical resources engaged in the Incident. • Separate conference bridge line for Section communications

Communications

• Numerous channels are leveraged by Incident Command to ensure timely and consistent communications

• Mixture of services provides resiliency in the event one or several IT services are impacted

• An off-premise IT service is also utilized for mass communications

Communication Templates • Templates are utilized for email and handouts

– Recognizable and Readable – Protects against spoofing

Communication Frequency Incident Activation

• Assess Impact of Incident and Operational Supports Needed • Set Interval for Status Update Calls • Send Out Customer Notifications • Establish Interval for Customer Communication Based on Impact of

Incident Status Updates

• Command Team Updates • Inform Content for Customer Communications • Update to Incident Management Status Site

Incident Resolution • Communication of Incident Resolution and Post-Incident Instructions

Debriefing and Post Incident Analysis

• Debriefing session within two weeks after the incident has been resolved

• The initial draft of the Root Cause Analysis form will be completed for review and can be used to help facilitate the discussion

• The nature of this session will be focused on after-action plans, learning and process improvement

Training • Should include classroom, eLearnings and drills

IS Resource Knowledge Areas

General IS Staff Basic plan overview; drill participation

On-Call Staff Basic plan overview; activation and assessment; drill participation

Service Center Basic plan overview; activation and assessment; drill participation

IS Leaders In-depth plan review; activation and assessment; command and control; basic incident command; drill participation

IS CIRT Team In-depth plan understanding of all aspects; advance incident command and NIMS; drill planning and execution

Security Incident • Management and Strategy of IT Security Incidents is a specialized

focus which requires additional considerations and focus • Should be developed in accordance with your IT Security Team • Should still operate within the IT Command Structure of your IT

Incident Management Plan

IT Computer Incident Response Team • Proactively identify potential threats which could disrupt normal IS

operations. • Maintain and continuously improve the Incident Management Plan. • Leads training in the area of execution of the Incident Management

Plan, Incident Command System and other incident/crisis management areas of knowledge.

• Actively engage in managing an incident when one occurs and capability to perform in any necessary role.

• Leads post-incident debriefing and analysis.

Outcomes • Communication to customers is efficient, meaningful, and with

defined frequencies • IT, clinical and administration leaders are engaged early • Roles and accountabilities are clear • IT resources who need to be engaged in the incident are engaged;

resources not needed are released • Regular status updates occur within the command team so all

involved parties are informed • Resources are focused and productively engaged • Anxieties are reduced • The incident is directed and controlled

Image courtesy of renjith Krishnan at FreeDigitalPhotos.net

Lessons Learned • Not every IT issue requires a Command Center • Iterative role out can be effective when implementing ICS • Empower the Command Team to manage the incident • Continue to improve process • Avoid old habits • Keep communications clear and concise • Know your systems

Questions

Heather Roszkowski, MSIA, CISSP

Information.Security@uvmhealth.org

References and Resources FEMA (2008). ICS Resource Center. Retrieved from http://training.fema.gov/emiweb/is/icsresource/assets/reviewmaterials.pdf

Incident management (ITSM). (n.d.). Retrieved from https://en.wikipedia.org/wiki/Incident_management_(ITSM)

Ponemon Institute. (2016). Cost of Data Center Outage. Retrieved from http://www.emersonnetworkpower.com/en-US/Resources/Market/Data-Center/Latest- Thinking/Ponemon/Documents/2016-Cost-of-Data-Center-Outages-FINAL-2.pdf

Avaya Inc. (2104). Network Downtime Results in Job, Revenue Loss. Retrieved from http://www.avaya.com/usa/about-avaya/newsroom/news-releases/2014/pr-140305/

US-CERT (2016). Incident Definition. Retrieved from https://www.us-cert.gov/government- users/compliance-and-reporting/incident-definition

Training Resource: Welcome to the Emergency Management Institute. (n.d.). Retrieved from https://training.fema.gov/emi.aspx