Post Katrina Disaster Post Katrina Disaster Recovery PlansRecovery PlansEducause Western Regional Educause Western Regional
Conference Conference April 26, 2006April 26, 2006
San Francisco, CaliforniaSan Francisco, California
Copyright StatementCopyright Statement
• Copyright Ann Dobson 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.
Panel TopicPanel Topic
• Panel discussion on how Universities have revised Disaster Recovery and Business Continuity plans in light of Hurricane Katrina and the lessons learned by Tulane and other Universities
Panel TopicPanel Topic
• Most Universities have Disaster Recovery and Business Continuity plans for IT. Lessons learned from Hurricane Katrina exposed weaknesses in those plans. For example, most plans don’t include closure for an entire semester. This discussion will center around how Universities are applying the lessons learned from Katrina. We will share ideas, help others get started, answer questions, and brainstorm solutions.
Panel PresentersPanel Presenters
• Kevin Barney, UC San Francisco
• Jacqueline Craig, UC Office of the Pres.
• Ann Dobson, UC Berkeley
• Nina Hundley, UC Berkeley
• Randy Jones, UC San Francisco Med Ctr
• Paul Weiss, UC Office of the Pres.
Information Resources and Communications University of California, Office of the President
Jacqueline CraigDirector of Policy, IR&COffice of the PresidentUniversity of California
Emergency Planning and Disaster Recovery Policy
University of California
Information Resources and Communications University of California, Office of the President
Current IT policy framework• Business and Finance Bulletin, IS-3,
Electronic Information Security– Risk Assessment and criticality
classification• Essential• Required• Deferrable
– Essential services must be included in Disaster Recovery Plans
Emergency Planning and Disaster Recovery Policy
University of California
Information Resources and Communications University of California, Office of the President
Current IT policy framework
Campus planning must include:– provisions for running essential
applications– emergency response procedures– backup of data and software– vendor requirements
Emergency Planning and Disaster Recovery Policy
University of California
Information Resources and Communications University of California, Office of the President
New Policy Framework
Presidential Policy on Safeguards, Security, and Emergency Management– each campus must implement
comprehensive and effective program– conduct risk assessment, risk mitigation,
emergency preparedness and response, and business recovery
Emergency Planning and Disaster Recovery Policy
University of California
Information Resources and Communications University of California, Office of the President
New UC IT Policy• Inventory and classify critical assets
• Continuity Planning at UC includes:– mitigation– preparedness– response– recovery
Emergency Planning and Disaster Recovery Policy
University of California
Information Resources and Communications University of California, Office of the President
Mitigation
• deployment of protective measures appropriate to each classification level
• infrastructure– communications resources– data centers
Emergency Planning and Disaster Recovery Policy
University of California
Information Resources and Communications University of California, Office of the President
Preparedness
• GOAL - reduce risk and minimize disruption of all campus programs
• risk assessment/business impact analysis
• components of continuity planning
Emergency Planning and Disaster Recovery Policy
University of California
Information Resources and Communications University of California, Office of the President
Response
• implement plans, operating procedures, or protocols– priorities for short and long term impacts
• establish processes• build in flexibility – cannot foresee all
potential circumstances
Emergency Planning and Disaster Recovery Policy
University of California
Information Resources and Communications University of California, Office of the President
Recovery• GOAL – re-establish operational
capability identified in planning priorities– people– facilities/property– financial and information technology
systems
Emergency Planning and Disaster Recovery Policy
University of California
UC Wide Disaster Recovery UC Wide Disaster Recovery Before KatrinaBefore Katrina
• Minimal sharing between UC entities
• Each campus & medical center had own plans and approach
• Office of the President had offsite vendor contract for critical systems and, cost if ever used DR site for real, is extremely expensive.
UC Wide Disaster Recovery After UC Wide Disaster Recovery After KatrinaKatrina
• The primary reactions - Increased urgency & enhanced spirit of SHARING across UC system!
• Joint Data Center Managers Group (comprised of all UC entities) begins work to consider sharing opportunities within UC. Two goals:– Enhance IT DR preparedness by eliminating
duplicate efforts– Consider redirecting spend from outside
vendors to within UC system
UC Wide Disaster Recovery After UC Wide Disaster Recovery After KatrinaKatrina
• IT Leadership Council (UC wide CIO working group) sponsors all day meeting on business continuity planning
• Guest speaker John Lawson (CIO at Tulane) speaks at meeting and shares two key lessons from his experience:– Communication is key– Pay your personnel
UC Wide Disaster Recovery After UC Wide Disaster Recovery After KatrinaKatrina
• JDCMG takes Tulane lessons and begins work on two efforts:– Determine if can back up all mainframes w/in
the UC system– Set up shared web site, registration site, and
email emergency site that any entity w/in UC can utilize if their local capabilities are disabled.
UC Berkeley Disaster Recovery UC Berkeley Disaster Recovery Before KatrinaBefore Katrina
• Emergency Web Site
• Emergency Purchasing Procedures
• On-site Backups
• Off-site Backups
• Off-site Vital Records
UC Berkeley Disaster Recovery UC Berkeley Disaster Recovery Before KatrinaBefore Katrina
• Business Impact Analysis
• IBM Hot Site Contract
• Annual Hot Site Tests
• New Applications Added Each Year
UC Berkeley Disaster Recovery UC Berkeley Disaster Recovery After KatrinaAfter Katrina
• Servers for redundancy at sister campus– CalNet authentication and directory services– Web communications tools (home page)– People locator– Distance learning (Sakai, Webcast)– Other applications, e.g. student registration
UC Berkeley Disaster Recovery UC Berkeley Disaster Recovery After KatrinaAfter Katrina
• Additional satellite phones
• Laptops, printers for response teams
• Data backup pilot for researchers
• Portable generators
• Planning for Wide-spread Regional Impact• Multi-pronged approach to Business Continuity
Risk Assessment - Analysis Prevention and Mitigation – New Data Center Emergency Preparedness – Emergency Operations Plan Emergency Communications – Immediate Information Business Resumption – Resume Business Functions Disaster Recovery – Recover Technical Environment
Administrative Computing
Planning for Business Continuity: “Lessons Learned from Katrina”
Identify critical business functions Determine priorities Establish recovery point objective (RPO) Establish recovery time objective (RTO) Determine disaster recovery strategy Develop disaster recovery procedures Contract recovery services
Business Continuity Governance
Align IT and business activities around business resumption requirements
Ensure responsibility and accountability are accepted by key stakeholders
Joint development of technical and business recovery plans
Integrate planning and testing Validate capabilities and compliance without
crippling productivity
Specific Challenges
Establish priorities Validate plan Justify funding and staffing Coordinate multiple plans
Information Technology Services• Associate Vice Chancellor, ITS• Director of Administrative
Computing• Administrative Computing Chief
Technology Officer• Administrative Computing
Operations Manager Controller’s Office
• Associate Vice Chancellor & Controller
• Asst. Controller, Fin. & Admin. Systems
• Asst. Controller, Disbursements• Asst. Controller, Acctng. / Reptng.
Business Continuity Committee
Develop plans Test & maintain procedures Conduct the recovery
Information Technology Services
• ITS Application Manager
• ITS Application Programmer/Analyst
• Open System Administrator
• MVS System Programmer Controller’s Office
• Manager Functional Unit
• Analyst Functional Unit
Business Continuity Team
Payroll / Personnel• Resident on the UCOP system • UCOP hot site contract covers 5 campuses on the base payroll
system + UCSF payroll• Last recovery test performed Fall 2005• RPO = 24 to 48 hours before disaster• RTO = 3 days after disaster
• Other Mainframe Applications• Need to acquire UCSF hot site contract• Develop recovery procedures for system and remaining
applications• Test and maintain procedures
Mainframe Recovery
Establish Hot Site Contact
• Recovery of critical business functions within five days
Emergency Communications – University-wide Solution
• Immediate availability
• Emergency communications website
• Emergency email
• Employee Registration
Establish Cold Site recovery at other UC campus
• Longer term recovery (Hitching-post site)
Second Site at UCSF
• Renovate former Data Center as a recovery site for limited localized outages
Backup Printing
• Backup printers at UCSF second site
• Backup printing at other UC campus
Open Systems Recovery Proposal
UCSF Medical Center Disaster Recovery Plan before Katrina
Overview of UCSF Medical CenterOverview of UCSF Medical Center Install generator.Install generator. Hire DRP Coordinator in 2002.Hire DRP Coordinator in 2002. Analysis of all application backups. Analysis of all application backups.
Make changes.Make changes. Internal Business Impact Analysis.Internal Business Impact Analysis. Recovery Plans for Specific ApplicationsRecovery Plans for Specific Applications Recover applications locally, Document Recover applications locally, Document
recoveries.recoveries. IBM BCRS Contract for 5 critical systems.IBM BCRS Contract for 5 critical systems.
UCSF Medical Center Disaster Recovery Plan before Katrina
Completed 5 Recovery Tests since Completed 5 Recovery Tests since December 2002.December 2002.
Recovery Plans for Specific Applications.Recovery Plans for Specific Applications. Cross train staff.Cross train staff. Iron Mountain 7 days a week.Iron Mountain 7 days a week. During Analysis Phase of an Application, During Analysis Phase of an Application,
complete high level assessment of DR complete high level assessment of DR implications, including Senior implications, including Senior Management Ranking.Management Ranking.
Examine hot site options.Examine hot site options.
UCSF Medical Center Disaster Recovery Plan after Katrina
Increase of Senior Management Increase of Senior Management awareness and concern.awareness and concern.
IBM Consultancy for facilities IBM Consultancy for facilities analysis, business impact analysis, business impact analysis, and recommendations analysis, and recommendations and plan going forward.and plan going forward.
Begin re-examine hot site Begin re-examine hot site options.options.
Top Related