Realizing the Need

ColoradoSecretary of State EMERGENCY PREPARATION AND RESPONSE


Realizing the Need

In June, 2004, the Department of State (CDOS) experienced a significant interruption in its data center services. Following an extensive outage, testing revealed the presence of “zinc whiskers”, zinc fibers that can be extruded over time by electroplated metal coatings, such as the coating applied on the electrostatic-dissipating raised floor tiles used in the CDOS data center.

The facility was rendered unusable for computer operations. The Department’s equipment had been internally contaminated and as a result was subject to additional disruptions. This catastrophic event spotlighted a paramount need for an achievable business continuity and disaster recovery capability.


Disaster Recovery Survey

In March of 2005 the Colorado Information Management Commission (IMC) through the Governors Office of Innovation and Technology (OIT) conducted a survey of the State Agency disaster recovery preparedness. The survey found that:

Of all 23 state agencies surveyed, none reported complete IT disaster recovery capability.

Risk was high for all State agencies because disaster recovery capabilities were low.

Threats were rising & the risk profile was getting worse.

Majority of agencies were planning and/or seeking individualized rather than collaborative disaster recovery investments.

An overwhelmingly common misconception about disaster recovery planning existed where it was thought that having a backup strategy, doing backups everyday and running anti-virus software was the same as having a Disaster Recovery Plan.


Strategic SolutionThe Colorado Legislature approved a $3.6 million appropriation with $2.3 million in continuation funds from the CDOS cash fund to build and maintain a disaster recovery facility.

Colorado Department of State, with the support of the Governor’s Office of Innovation and Technology successfully implemented a strategic solution for disaster recovery known as the enterprise Facility for Operational Recovery/Readiness/Response and Transition services, ( “e-FOR3T”.)

The e-FOR3T initiative established a statewide disaster recovery facility for use by state agencies and government entities. The facility is operated under the provisions of a cooperative agreement between the Colorado Secretary of State's Office and participating agencies. The facility allows agencies to recover critical functionality in the event of a disaster or an event resulting in a significant interruption of service by:

Providing a tier 3+ IT facility for operational recovery

Promoting economies of scale through uniformity of service & infrastructure

Providing participating member agencies with a cost effective & reliable level of expected service


E-For3t Disaster Recovery Facility

Enterprise Solution – Uniform application of best practices for operational recovery, readiness & response

Funding Strategy – Colorado Department of State utilizing its cash funds to maintain portions of the disaster recovery infrastructure that are common in facility infrastructure & operations

Participating Member Agency Flexibility – Allows the flexibility to architect specific required recovery solutions

All State of Colorado Government Entities Invited – Participation is voluntary for any of these entities

Facility Infrastructure Components – Power, connectivity, and 12,167 sq ft of space to house equipment and personnel

Enterprise Facility for Operational Recovery Readiness Response & Transition services

Addresses the following strategic objectives by providing


e-FOR3T Specifications

Data Center Space 6,549 sq ft raised floor

Individually locking (front and rear) cabinets

Private cages as required

Migration & Transition Support Staffing and resources for physical migration

Documented processes and procedures

Relocation and standup of state agency hardware

Review post-transition analysis internally and externally

Off-site tape vaulting



Office Space 5,618 sq ft fully furnished (desks,

workstations, phones, etc)

Security 24 x 7 magnetic card key access with

secondary pin code

Digital motion activated security cameras and intercoms

Cable Management Provide all cabling for each agency

deployment

Cabling tagged and documented



Network Operations Center –Telecom & Network Support MNT network services provided on dual path

system with redundant IP router & LAN switches

20 remote access internet accounts with static IP addresses for each participating member agency (10 DSL & 10 dial up)

Remote hands services allowing State to use facility personnel to perform basic tasks as instructed by State engineers

Insurance coverage for State assets

MNT network failover to redundant IP network

Private Network Transit (PNT) via backbone connection between Denver and alternate site



Power Facility power, delivery, infrastructure,

generator and fuel

UPS systems and megawatt diesel generators

Cooling HVAC service to support requirements

Anti-static raised floor with cooled air delivery

Fire Detection & Suppression Certified incipient smoke detection (VESDA)

Dual-Interlock Pre-Action dry pipe sprinkler system


Continuity of Operation Plan

The Governor’s Office of Information Technology (OIT) provides the leadership in planning for recovery of State of Colorado functions after a man-made or natural disasters through their Continuity of Operation/Continuity of Government Program. The Program Manager assists State of Colorado agencies in developing COOP plans using Federal standards for content per FPC 65. Internet-based LBL Contingency Planner software is used to create and update COOP documentation.

The templates included in the Contingency Planner software include suggested procedures for carrying out a disaster recovery plan. Following the federal model, these plans will address identification and prioritization of agency essential functions, succession, delegation of authority, interoperable communications, identification of alternate facilities, identification of vital records and databases, human capital aspects, reconstitution of facilities and the testing, training and exercising of these plans.

Each program within CDOS was tasked with providing specific details regarding their operations. By creating documents with possible scenarios agencies were able to determine how specific needs would change based on the circumstances.


Scenario I – SOS and Other Agencies Affected/Internet Functional

A. Assumptions:

1. Activation of the COOP is the result of a disaster that affects more than the Secretary of State.

2. The Secretary of State’s office is uninhabitable.

3. Other affected agencies will want to file emergency rules with Secretary of State.

4. Internet and SOS systems operational at e-For3t within 36 to 48 hours after COOP activation.

5. IT has implemented VPN so that employees can work from home rather than at e-For3t.

B. Critical functions:

In a normal work environment rulemaking agencies make all filings electronically, paper filings are not accepted. With the current procedures, all electronically filed rules need to be reviewed and the formatting needs to be modified by the Administrative Rules Staff in order to be published online. This process requires extensive review and formatting changes by the Administrative Rules Staff to ensure that documents are compatible with the online publishing system.

Normally, the rulemaking process is spread out over a number of months and many agencies file infrequently. Only a handful of agencies file every month. In an emergency situation, many more agencies may be forced to adopt rules to accommodate the situation. The additional load on the system may create problems and the total volume of rules will necessarily increase but the amount is unknown.


Scenario I – Continued

C. Minimum personnel:

1. Program Mgr

2. Program Asst II

3. Administrative Asst III

D. Resource Requirements:

1. Personal desktops, laptops or laptops provided by SOS.

2. Access to high speed Internet lines (May require subsidization of additional costs to employee).

3. VPN connectivity.

4. Telephone.

5. Scanner.

E. Space Requirements:

1. None – employees work from home.

F. Travel costs:

1. Program Mgr may be required to travel from home to eFort on occasional basis.


Scenario II – SOS and Other Agencies Affected/Internet Nonfunctional

A. Assumptions:

1. Activation of the COOP is the result of a disaster that affects more than the Secretary of State.


3. Other affected agencies will want to file emergency rules with Secretary of State.

4. Internet is NOT functional but e-For3t is operational with an Intranet capability.

5. IT has NOT implemented VPN. Employees cannot work from home.


The normal process of electronic filing of administrative rules is disrupted. Agencies must default to the previous procedure of delivering rules directly to the Secretary of State. Although the Internet is not working, the Intranet will allow for rules to be entered into the database for use when the Internet becomes available. Emergency rules will be the major portion of the workload with normal rules making up the balance.

In this scenario, mail or hand carried documents must be delivered to a single location. The Administrative Rules program will have to receive the documents (either in electronic or paper format, or both), date/time stamp them and then process them into the system. For an interim period the official register will be published in print rather than electronically. Materials will be sent electronically or mailed to the outside vendor that produces the Colorado Register in print.


Scenario II – Continued

C. Minimum Personnel:

1. Program Mgr.

2. Program Asst. I

3. Admin. Asst. III


1. Computer workstations with Intranet access to the database for three individuals.

2. Telephone (at least one).

3. Copy machine.

4. Fax machine.

5. Scanner.


1. Three workstations at eFort. One option would be to coordinate with the Arapahoe County Clerk’s office and use it as the mail reception point and initial processing site. That could eliminate the need for two of the three workstations at eFort.

F. Travel Costs:

1. Staff will be required to work from an alternate location.


Scenario III – Only SOS Affected/Internet Functional

A. Assumptions:

1. Activation of the COOP is the result of a disaster that only affects the Secretary of State.


3. Agencies will only need to file rules in the normal course. These may be regular or emergency rules but the situation is no different than normal operations.

4. Internet and SOS systems operational at e-For3t within 36 to 48 hours after COOP activation.

5. IT has implemented VPN so that employees can work from home rather than at e-For3t.


In a normal work environment rulemaking agencies make all filings electronically, paper filings are not accepted. With the current procedures, all electronically filed rules need to be reviewed and the formatting needs to be modified by the Administrative Rules Staff in order to be published online. This process requires extensive review and formatting changes by the Administrative Rules Staff to ensure that documents are compatible with the online publishing system.

In this circumstance, it will more or less be business as usual for the Administrative Rules Program. The only thing different will be that staff will either work from home or work at e-For3t.


Scenario III – Continued

C. Minimum Personnel:

1. Program Mgr.

2. Program Asst. I

3. Admin. Asst. III


1. Personal desktops, laptops or laptops provided by SOS.

2. Access to high speed Internet lines (May require subsidization of additional costs to employee).

3. VPN connectivity.

4. Telephone.

5. Scanner.


1. Personnel could work from home per Scenario I or work at eFort per Scenario II.

F. Travel Costs:

1. Staff may need to work from an alternate location.


Links to COOP Info Continuity of Operation Planning/ Continuity of Government

http://www.colorado.gov/cs/Satellite/OIT-New/OITX/1188204067666 enterprise Facility for Operational Recovery Readiness Response & Transition (e-FOR3T)

http://www.sos.state.co.us/pubs/business/efort.htm LBL Technology Partners Contingency Planner software

http://www.lbltech.com/ContingencyPlanner.htm

Realizing the Need

Documents

Transcript of Realizing the Need