The IT FirefighterEveryday Hero and Crisis Manager

Bill EvansSAC HDI CONFERENCE 2015

What is firefighting?• Firefighting is an emergency allocation of resources, required to deal with an

unforeseen problem. In software development, for example, firefighting might involve assigning extra programmers to fix coding bugs that are discovered close to a product's release date; in a security context, it might involve allocating resources to deal with the breach of an information system or the outbreak of a computer virus. At the individual user level, firefighting might involve dealing with hardware or software problems that could have been prevented with basic computer maintenance practices.• Just as in the real world, there's an assumption that "fires" are unpredictable and

that they must be dealt with immediately. However, a too-frequent need for emergency action may reflect poor planning, or a lack or organization, and is likely to tie up resources that are needed elsewhere. To keep firefighting to a minimum, comprehensive disaster recovery planning (DRP) often includes the attempt to foresee, and protect against, such emergencies.• http://whatis.techtarget.com/definition/firefighting

What is Crisis Management?• Crisis management consists of:• Methods used to respond to both the reality and

perception of crises.• Establishing metrics to define what scenarios constitute a

crisis and should consequently trigger the necessaryresponse mechanisms.• Communication that occurs within the response phase of

emergency management scenarios.

Planning

So can planning prevent IT Fires?

Tools

So can tools prevent IT Fires?

Training

So can training prevent IT Fires?

Do you have great IT Firefighters?

• My #1 interview question when hiring for an IT Support position ---

“Let’s say you are working here today. Are you the type of person that before you leave today you want to have tomorrow already planned so you know exactly what you will do hour by hour tomorrow or would you rather just come in and take the day as it comes and handle any challenge that gets thrown at you?”

What IT Processes involve IT Firefighting?

Incident Management/Response• Fire, Police, Ambulance, Environmental

• Desktop Techs, Application Analysts, DBA’s, Network Engineers

• Marketing – 911, Fire Safety and Prevention• SLM, User Training, Self Service, User Knowledge Available

• Procedures• Command and Control

• Disaster Recovery Plan / Business Continuity• Very Organized – No room for error, highly trained, box alarms

• Flow charts, Job Aids, Formal and Informal training

• Roles – For people and apparatus/tools• Same (*make sure everyone knows the roles and how to use the tools)

Fire Response vs. IT ResponsePeople Roles

• Chief• Captain• Lt’s• Sr Engineer• Pumper• Attack• Rescue• Recovery• Mop up• Fire Police• EMS• Police• Dispatcher

Apparatus Roles– Engine– Pumper– Ladder– Rescue– Truck– Tanker– Squad– Brush– Mini Pumper

Protocols– Radio– Box Alarms

• Respond• Stand-by• Relocate

Fire Response

Fire Response vs. IT ResponsePeople Roles

• Service Desk• Situation Manager• Communications

Manager• Business

Continuity• Application

Analyst• DBA• Network Engineer• Desktop Tech• Who Else???

IT Tools– ITSM

• Incident• Change• Event

– IT Alerts– Remote Support– KnowledgeBase– Software

Monitoring– Hardware

Monitoring– What Else?

Protocols– Process Flows– Box Alarms

• Respond• Stand-by• Relocate

– Disaster Recovery Plans

– Business Continuity Plans

IT Response

Hero Examples

Hero Examples

Hero Examples

Hero Examples

Hero Exampleso “That guy really knows what he is doing!” – SJCH Techo “He was very helpful and fixed the issue quickly.” – SC Analysto “Tech was very professional and prompt with helping me get my computer and printer working in sync

again. His help was greatly appreciated. Thank you!” FRH Techo “I am extremely appreciative of the patience Analyst had with me and the willingness to walk me

through the steps on how to link a baby and mother from PIC side. I appreciate the fact that Analyst also emailed me some education not only for myself but for my coworkers on this process to. He was very polite understanding. I look forward to calling in for future help especially if all the Analysts are as knowledge able and willing to work with me as Analyst has. I appreciate all his efforts in helping myself and my coworkers on being able to do a better job to help our patients have a better experience.” – ASA Ancillary Maintenance

o “Tech always goes above and beyond our expectations. Thanks !” – AMCP Techo “Tech always is prompt in fixing our IT problems. He worked with us to get both tracking boards working.

Excellent customer service!!!” SRMC Techo “Excellent resolution of the problem. Having this printer go down impacted our ability to deal with a

very critical STAT situation in Blood Bank. Thanks to Aaron for helping and resolving so quickly.” – SC Analyst

o “Analyst is always prompt and knowledgeable. I very much appreciate her!” – Clinical ASA North Teamo “Analyst was awesome!” Ambulatory Analyst

Zero ExamplesoYes we do get those too

Hero Metrics

Final Thoughts• Know your role• Know everyone else’s role• Be ready to assume higher or lower or lateral roles• Leader and Follower• Crisis can happen anytime• Practice Practice Practice• Always look to get better and improve process• Always evaluate changes in technology, operations, and

practices in your industry• Use HDI – Be a member

The IT FirefighterEveryday Hero and Crisis Manager

The IT FirefighterEveryday Hero and Crisis Manager

Bill EvansSAC HDI CONFERENCE 2015

evansw86@outlook.comLinkedin - www.linkedin.com/in/evansw86