1

Critical Success Factors in a Complex BCP Development Environment

Don Groth

Disclaimer – Ground Rules

• The views and opinions expressed today do not

necessarily reflect the position of Froedtert and

Community Health or Jefferson Wells

International, Inc.

• Any copyrights/trademarks belong to owners...

• Comments about vendor products or services

are intended for illustrative purposes only.

BCP – Critical Success Factors

• Case Study

– What was done

– How was it done

• Lessons Learned

• Success Factors

2

BCP – Critical Success Factors

• Success

– Ability to survive an actual incident

– Ability to survive a review

• Internal Audit

• Regulators

• Management

Case Study

• The success was due to:

– Hard work and dedication of hospital staff

– Simple “key” factors

– The factors may be called simple because they are easy to describe; putting these

factors in practice may not necessarily be simple.

• Easy to say – tough to do?

Business Continuity Planning at Froedtert & Community Health

• Froedtert & Community Health

– Milwaukee-based regional hospital system

– Combination

– Froedtert Hospital – Milwaukee

– Community Memorial Hospital – Menomonee

Falls

3

Case Study

• Continuity plan development for Froedtert & Community Health

• Development of continuity plans for over 60 key clinical, facility, support, and business

departments throughout the two hospitals.

• Linkages to existing Emergency Management

/ Incident Command Structure and to an IT Disaster Recovery plan.

Froedtert Hospital

414 Bed Academic Medical Center

Staffed by Medical College of Wisconsin Physicians

> 4,500 staff and > 10,000 people on campus

The Only Adult Level I Trauma Center in Eastern Wisconsin

4

Froedtert

• Campus Partners

• Joint Ventures

• Staffing

– Medical College Staff

– Departments

• Off-campus facilities

Community Memorial Hospital

205 bed community hospital >2,000 Staff

Staffed by independent physicians

Community Memorial

• Two large clinics

• Cooperative Ventures

– Free Standing Ambulatory Surgery Center

– Independent community physicians

• Off-campus facilities

5

F&CH Environment

• Recently combined hospitals as F&CH

• Healthcare considerations

– Joint Commission on Accreditation of

Healthcare Organizations – JCAHO– Health Insurance Portability and

Accountability Act (HIPAA)

– Existing Memorandum of Understanding– HEICS / HICS

Environment - continued• Emergency management structures

• Regional disaster drills

• Downtime procedures

IT Environment

• Many initiatives including data center relocation

• A number of “high availability” systems

• A number of systems managed by clinical and

support departments

• Hot site plan – in development

• Downtime procedures

6

Overall Business Environment

Everybody is Busy!

Why BCP at Froedtert & Community Health ?

Why? August 2003: Power Failure

7

Source - U.S./Canada Power Outage Task Force report

F&CH Business Continuity Project

• Began early 2004

• Board directive…

“It’s not a question of if we do this. The only question is how should we do it.”

• Mandate - “Not just an IT plan”

F&CH Business Continuity Project

• Coordinate

– Business Continuity Planning

– Emergency Management

– IT Disaster Recovery

• Consider

– Prior Threat Assessments, Risk Assessments, and Hazard Analyses

8

Project Organization Chart

Timeline2004 20062005

DeOcAuMa Ju J Se No DeOcAuMaJa Fe Ma Ap Ju J Se No Ja Fe

BIA

THREAT ASSESSMENT

MAINTENANCE AND EXERCISING

INITIALPROJECTSCOPING

BUSINESS CONTINUITYPLAN DEVELOPMENT

RECOVERY STRATEGIES

Scoping

• Set project scope

• Determined departments to include /

exclude

• Grouped departments

• Selected department staff

9

Scoping

• Lessons

– Departments

– Combining departments

– About staff

BIA

• Facilitated working sessions

• Groups of departments

• Identified critical

– IT systems

– Resources

• Tangible – Intangible assessment

• Detailed calculations

• RTO & RPO

BIA

• Multiple steps

• Made detailed estimates only for systems and resources with High / Medium Impacts

• Consolidation

10

Step 1 – Identification of Key Systems / Resources

Step 2 – Detailed Estimates

Scales

11

BIA Consolidation

• To provide management with estimated impacts

• To provide IT with system RTOs

• To identify the most significant resources

• Avoid double counting

• Recognize significance of department / process impacted

• Judgment required

BIA

• Lessons

– Grouping of departments

– People will respond differently

– Time

– Intangible vs. Tangible Impact

– Surprises about systems

– Work sessions

Threat Assessment• Timing

• Critical – Global Resources

Electric Power (utility & emergency),

Fire Detection Systems, Medical Gas,

Natural Gas, Steam, Telecom,

Water Supply, Waste Water

• Identified Threats

– Impacts, Probability, Vulnerability

12

Threat Assessment

• Lessons

– Actual incidents

– Water

– Assumptions

Strategy Selection

• Mitigation Strategies – “Global” resources

with vulnerabilities identified in Threat

Assessment

– Hardening Strategies

– Work Around strategies

• Other Resources – (Not Global)

• In-place Strategies – Published

• Recommended Strategies – Budget process

Strategy Selection

• Lessons

– Ownership of solutions

– Tie strategies to budget process

– Executive Support

– And then…

13

Reality Check # 1: July 2005

Brief power failure at Community

Memorial

Renewed enthusiasm

Department Plan Development

• Working sessions to create department

plans

• Remember that everybody is busy – allow

plenty of time

Department Plan Development

• Two sets of working sessions

• Provided sample plan and templates

– Contact information

– BIA

– Strategies

– Vendors

– Forms

– Recovery procedures, and then…

14

Reality Check # 2: December 2005

1 hour power failure at Froedtert

Renewed enthusiasm

Department Plan Development

• Lessons

– Time

– Stories

– Use what you have – but it is probably not

enough

– Executive support

– And then…

Reality Check # 3:

Thursday – March 9, 2006

1,000,000 gallons of water flood the power

plant and steam tunnels at the Milwaukee County Grounds. “It appears to have been a pretty catastrophic blowout,” says George

Torres, County Public Works Director.

15

Reality Check # 3:Initial report

• The basement of the electric power plant that supplies power and steam to the hospital and clinics is filling with water

• The walls may buckle• There is a substantial leak in the water main

• Water pressure is declining – affecting:– drinking water– sanitation water

– central vacuum pressure– steam

Reality Check # 3:Continued

• Maintenance crews have not been able to isolate the leak

• May have to take the plant out of service for an extended period of time

• Could have to evacuate

– approximately 400 inpatients– hundreds of outpatients– all staff

• It could be weeks for the problem to be identified and repaired.

16

Lessons Learned

• Threat Assessment conclusions - reinforced

• Water is a critical resource

• Other organizations were eager to assist

– Hospitals

– Ambulance companies

Lessons Learned

• Communications

– Staff – email used extensively and was

effective, however…

– Media

• Incident Command Center

– ICC established quickly

– Department Command Centers

Plan Exercises• Individual departments

• Tabletop exercise / plan review

• Participants

– Department staff (1 – 10)

– Safety

– Facilitators

• Scenario

• Action Plans

17

Lessons Learned

• Be flexible

• Training opportunity

• Challenge / validate

• Raise the bar

• Stories

– Have participants tell their stories

– Use the stories with others

Current State

• Transitioning from project to program

• Incident command integration

• Strategies for critical resources

• Program expansion

Transition From “Project to Program”

BCP Plan Development BCP Program

• Business Impact Analysis

• Threat Assessment

• Plan Development

• Integrate with Incident Command

• Exercise – Maintain Plans

• Ongoing funding and resources

“Light the Fire” “Keep it Fueled”

18

Success Factors

• Executive Mandate and Executive Support

• Existing Emergency Management Experience

• Steering Committee

• Culture – Patient Care

• Terminology

• Leveraging NIMS Alert - Compliance and day-to-day operations

NIMS Alert - Compliance and day-to-day operations

From the August 17, 2005, NIMS Alert

“The requirement to adopt and implement NIMS and ICS means NIMS and ICS for incident management every day. Those who don’t are not NIMS compliant.”

Success Factors

• Pilot

• “Real incidents”

• Leverage actual Incidents

• “Lessons Learned Sessions” for staff

• Reinforce the need to plan

– They will tell us what to do

– We will do “whatever it takes”

19

Success Factors - Linkages

Emergency

Management

PlansIT Disaster

Recovery

Regional

“Partners”

BCP

Success Factors

• GETS – Government Emergency Telecommunications Service (GETS)

http://gets.ncs.gov/

• Participant profile / skills

– Familiar with department processes

– Department decision maker

– And computer skills

Success Factors

Persistence

“We can do anything we want as long as we stick

to it long enough.” - Helen Keller

“Even if you are on the right track, you will be run

over if you just sit there.” - Will Rodgers

20

Success Factors

Always Serve Good Food!

Final Thoughts

“Just because you’re paranoid, it doesn’t mean that people aren’t out to get you.”

- Unknown

“The reason for time is to avoid doing

everything at once.”

- Albert Einstein

Questions

Jefferson Wells

888-444-5415

or

414-347-2345

don.groth@jeffersonwells.com

www.jeffersonwells.com

21

• Headquartered in Milwaukee, WI

• Founded in 1995

• More than 45 offices

• Over 2,500 employees

• Subsidiary of Manpower Inc.

• Provides services in the areas of:• Internal Audit and Controls• Technology Risk Management• Finance and Accounting

• Tax

Jefferson Wells

• Jefferson Wells - Technology Risk Management Services

• CBCP, CISA, CIA

• Member of BRPASW, IIA, ISACA, Infragard

• 888-444-5415 or 414-347-2345

• Don.Groth@jefffersonwells.com

• www.jeffersonwells.com

Don Groth