Post on 12-Jan-2016
description
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Realizing the Need
In June, 2004, the Department of State (CDOS) experienced a significant interruption in its data center services. Following an extensive outage, testing revealed the presence of “zinc whiskers”, zinc fibers that can be extruded over time by electroplated metal coatings, such as the coating applied on the electrostatic-dissipating raised floor tiles used in the CDOS data center.
The facility was rendered unusable for computer operations. The Department’s equipment had been internally contaminated and as a result was subject to additional disruptions. This catastrophic event spotlighted a paramount need for an achievable business continuity and disaster recovery capability.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Disaster Recovery Survey
In March of 2005 the Colorado Information Management Commission (IMC) through the Governors Office of Innovation and Technology (OIT) conducted a survey of the State Agency disaster recovery preparedness. The survey found that:
Of all 23 state agencies surveyed, none reported complete IT disaster recovery capability.
Risk was high for all State agencies because disaster recovery capabilities were low.
Threats were rising & the risk profile was getting worse.
Majority of agencies were planning and/or seeking individualized rather than collaborative disaster recovery investments.
An overwhelmingly common misconception about disaster recovery planning existed where it was thought that having a backup strategy, doing backups everyday and running anti-virus software was the same as having a Disaster Recovery Plan.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Strategic SolutionThe Colorado Legislature approved a $3.6 million appropriation with $2.3 million in continuation funds from the CDOS cash fund to build and maintain a disaster recovery facility.
Colorado Department of State, with the support of the Governor’s Office of Innovation and Technology successfully implemented a strategic solution for disaster recovery known as the enterprise Facility for Operational Recovery/Readiness/Response and Transition services, ( “e-FOR3T”.)
The e-FOR3T initiative established a statewide disaster recovery facility for use by state agencies and government entities. The facility is operated under the provisions of a cooperative agreement between the Colorado Secretary of State's Office and participating agencies. The facility allows agencies to recover critical functionality in the event of a disaster or an event resulting in a significant interruption of service by:
Providing a tier 3+ IT facility for operational recovery
Promoting economies of scale through uniformity of service & infrastructure
Providing participating member agencies with a cost effective & reliable level of expected service
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
E-For3t Disaster Recovery Facility
Enterprise Solution – Uniform application of best practices for operational recovery, readiness & response
Funding Strategy – Colorado Department of State utilizing its cash funds to maintain portions of the disaster recovery infrastructure that are common in facility infrastructure & operations
Participating Member Agency Flexibility – Allows the flexibility to architect specific required recovery solutions
All State of Colorado Government Entities Invited – Participation is voluntary for any of these entities
Facility Infrastructure Components – Power, connectivity, and 12,167 sq ft of space to house equipment and personnel
Enterprise Facility for Operational Recovery Readiness Response & Transition services
Addresses the following strategic objectives by providing
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
e-FOR3T Specifications
Data Center Space 6,549 sq ft raised floor
Individually locking (front and rear) cabinets
Private cages as required
Migration & Transition Support Staffing and resources for physical migration
Documented processes and procedures
Relocation and standup of state agency hardware
Review post-transition analysis internally and externally
Off-site tape vaulting
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
e-FOR3T Specifications
Office Space 5,618 sq ft fully furnished (desks,
workstations, phones, etc)
Security 24 x 7 magnetic card key access with
secondary pin code
Digital motion activated security cameras and intercoms
Cable Management Provide all cabling for each agency
deployment
Cabling tagged and documented
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
e-FOR3T Specifications
Network Operations Center –Telecom & Network Support MNT network services provided on dual path
system with redundant IP router & LAN switches
20 remote access internet accounts with static IP addresses for each participating member agency (10 DSL & 10 dial up)
Remote hands services allowing State to use facility personnel to perform basic tasks as instructed by State engineers
Insurance coverage for State assets
MNT network failover to redundant IP network
Private Network Transit (PNT) via backbone connection between Denver and alternate site
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
e-FOR3T Specifications
Power Facility power, delivery, infrastructure,
generator and fuel
UPS systems and megawatt diesel generators
Cooling HVAC service to support requirements
Anti-static raised floor with cooled air delivery
Fire Detection & Suppression Certified incipient smoke detection (VESDA)
Dual-Interlock Pre-Action dry pipe sprinkler system
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Continuity of Operation Plan
The Governor’s Office of Information Technology (OIT) provides the leadership in planning for recovery of State of Colorado functions after a man-made or natural disasters through their Continuity of Operation/Continuity of Government Program. The Program Manager assists State of Colorado agencies in developing COOP plans using Federal standards for content per FPC 65. Internet-based LBL Contingency Planner software is used to create and update COOP documentation.
The templates included in the Contingency Planner software include suggested procedures for carrying out a disaster recovery plan. Following the federal model, these plans will address identification and prioritization of agency essential functions, succession, delegation of authority, interoperable communications, identification of alternate facilities, identification of vital records and databases, human capital aspects, reconstitution of facilities and the testing, training and exercising of these plans.
Each program within CDOS was tasked with providing specific details regarding their operations. By creating documents with possible scenarios agencies were able to determine how specific needs would change based on the circumstances.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Scenario I – SOS and Other Agencies Affected/Internet Functional
A. Assumptions:
1. Activation of the COOP is the result of a disaster that affects more than the Secretary of State.
2. The Secretary of State’s office is uninhabitable.
3. Other affected agencies will want to file emergency rules with Secretary of State.
4. Internet and SOS systems operational at e-For3t within 36 to 48 hours after COOP activation.
5. IT has implemented VPN so that employees can work from home rather than at e-For3t.
B. Critical functions:
In a normal work environment rulemaking agencies make all filings electronically, paper filings are not accepted. With the current procedures, all electronically filed rules need to be reviewed and the formatting needs to be modified by the Administrative Rules Staff in order to be published online. This process requires extensive review and formatting changes by the Administrative Rules Staff to ensure that documents are compatible with the online publishing system.
Normally, the rulemaking process is spread out over a number of months and many agencies file infrequently. Only a handful of agencies file every month. In an emergency situation, many more agencies may be forced to adopt rules to accommodate the situation. The additional load on the system may create problems and the total volume of rules will necessarily increase but the amount is unknown.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Scenario I – Continued
C. Minimum personnel:
1. Program Mgr
2. Program Asst II
3. Administrative Asst III
D. Resource Requirements:
1. Personal desktops, laptops or laptops provided by SOS.
2. Access to high speed Internet lines (May require subsidization of additional costs to employee).
3. VPN connectivity.
4. Telephone.
5. Scanner.
E. Space Requirements:
1. None – employees work from home.
F. Travel costs:
1. Program Mgr may be required to travel from home to eFort on occasional basis.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Scenario II – SOS and Other Agencies Affected/Internet Nonfunctional
A. Assumptions:
1. Activation of the COOP is the result of a disaster that affects more than the Secretary of State.
2. The Secretary of State’s office is uninhabitable.
3. Other affected agencies will want to file emergency rules with Secretary of State.
4. Internet is NOT functional but e-For3t is operational with an Intranet capability.
5. IT has NOT implemented VPN. Employees cannot work from home.
B. Critical functions:
The normal process of electronic filing of administrative rules is disrupted. Agencies must default to the previous procedure of delivering rules directly to the Secretary of State. Although the Internet is not working, the Intranet will allow for rules to be entered into the database for use when the Internet becomes available. Emergency rules will be the major portion of the workload with normal rules making up the balance.
In this scenario, mail or hand carried documents must be delivered to a single location. The Administrative Rules program will have to receive the documents (either in electronic or paper format, or both), date/time stamp them and then process them into the system. For an interim period the official register will be published in print rather than electronically. Materials will be sent electronically or mailed to the outside vendor that produces the Colorado Register in print.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Scenario II – Continued
C. Minimum Personnel:
1. Program Mgr.
2. Program Asst. I
3. Admin. Asst. III
D. Resource Requirements:
1. Computer workstations with Intranet access to the database for three individuals.
2. Telephone (at least one).
3. Copy machine.
4. Fax machine.
5. Scanner.
E. Space Requirements:
1. Three workstations at eFort. One option would be to coordinate with the Arapahoe County Clerk’s office and use it as the mail reception point and initial processing site. That could eliminate the need for two of the three workstations at eFort.
F. Travel Costs:
1. Staff will be required to work from an alternate location.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Scenario III – Only SOS Affected/Internet Functional
A. Assumptions:
1. Activation of the COOP is the result of a disaster that only affects the Secretary of State.
2. The Secretary of State’s office is uninhabitable.
3. Agencies will only need to file rules in the normal course. These may be regular or emergency rules but the situation is no different than normal operations.
4. Internet and SOS systems operational at e-For3t within 36 to 48 hours after COOP activation.
5. IT has implemented VPN so that employees can work from home rather than at e-For3t.
B. Critical functions:
In a normal work environment rulemaking agencies make all filings electronically, paper filings are not accepted. With the current procedures, all electronically filed rules need to be reviewed and the formatting needs to be modified by the Administrative Rules Staff in order to be published online. This process requires extensive review and formatting changes by the Administrative Rules Staff to ensure that documents are compatible with the online publishing system.
In this circumstance, it will more or less be business as usual for the Administrative Rules Program. The only thing different will be that staff will either work from home or work at e-For3t.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Scenario III – Continued
C. Minimum Personnel:
1. Program Mgr.
2. Program Asst. I
3. Admin. Asst. III
D. Resource Requirements:
1. Personal desktops, laptops or laptops provided by SOS.
2. Access to high speed Internet lines (May require subsidization of additional costs to employee).
3. VPN connectivity.
4. Telephone.
5. Scanner.
E. Space Requirements:
1. Personnel could work from home per Scenario I or work at eFort per Scenario II.
F. Travel Costs:
1. Staff may need to work from an alternate location.
ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE
Links to COOP Info Continuity of Operation Planning/ Continuity of Government
http://www.colorado.gov/cs/Satellite/OIT-New/OITX/1188204067666 enterprise Facility for Operational Recovery Readiness Response & Transition (e-FOR3T)
http://www.sos.state.co.us/pubs/business/efort.htm LBL Technology Partners Contingency Planner software
http://www.lbltech.com/ContingencyPlanner.htm