Post on 26-Dec-2015
Pre-Disaster Recovery Plans
Presentation to:NACRC Legislative Conference
March 3, 2007Carol Foglesong, Orange County Comptroller’s
Office,Orlando, FL
andPaul Ketz, Broward County Records Division,
Ft. Lauderdale, FL
Agenda Introductions Purpose Unpredictability of Disasters Business Continuity Planning Definitions Reasons for a Pre-Need Contract What do you need to consider? Orange County’s Process to Contract Other Resources Questions
Purpose We are not here because we are
experts in the field of Business Continuity/Disaster Planning.
We are here because all of us are struggling with this topic.
We would like to share some of our experiences with you, and learn from the experiences of others here today.
So, what’s a disaster? Disasters come in all sizes and shapes
Big events: Hurricanes, Tornadoes, Earthquakes, Blizzards, Floods, Vandalism, Terrorism
Smaller events: Fires, Water Pipes, Roof Leaks
Disasters’ impact A large geographic area A localized area (county or city) A particular office operation/location
Hurricane Wilma Exposed office
suites are seen in the damaged Regents Bank Plaza on Tuesday, October 25, 2005 in Fort Lauderdale. (Sun-Sentinel/Joe Cavaretta)
Hurricane Wilma The Broward
County School Board building as seen from the 30th floor of the AutoNation Building in downtown Fort Lauderdale. (Brian Hekman, Coconut Creek)
Hurricane Wilma Virtually every
window on the west side of the Broward School Board building in downtown Fort Lauderdale was blown out by Hurricane Wilma. (Sun-Sentinel /Carl Seibert)
Hurricane Wilma Repairs are
underway at the Broward County Courthouse offices and parking garage in downtown Fort Lauderdale. (Sun-Sentinel.com)
What’s a common theme? Disasters are typically unexpected Disasters happen when you have
or are making other plans Disaster size or impact can’t be
readily predicted Disasters aren’t all BIG events; it
can be the little ones that cause the most concern and frustration
Disaster Readiness Business Continuity Plan
Staffing plans How to get up and running again
Disaster Recovery Plan Setting priorities on what has to be tackled first Papers versus computers
Forecast Disaster Plan Before the event planning Office preparations when a disaster is predicted
But who do you call for help? You’ve made a list
Businesses with addresses, names, phone numbers, faxes, and email addresses
Services offered and needed You’ve identified responsibilities of staff
Who calls whom when Who’s in charge
Do you and your staff keep a copy of your plan/phone lists with you? It won’t help if your list is at the office!
How to guarantee availability?
If a large area is impacted, there will be many organizations needing help; you may need to “get in line”
If a small area is impacted, will you get the specific help you need quickly?
If it’s a particular building or location impacted, what’s the best solution?
One answer A PRE-NEED CONTRACT A commitment to your organization
by a service provider You have a spot on the top of their
list You’ve pre-negotiated prices and
services You’ve already checked out who the
provider is and their references
Getting the Pre-Need Contract
You will have to convince your purchasing/contract group that a pre-need contract is not only possible, but logical
You knowingly commit to going through the purchasing process of your organization even with no known disaster on the horizon
Cost Be clear: no services = no dollars But money has to be available when
services are, in fact, necessary Does your organization require a “set aside”? Does your organization have a “reserve fund”?
Contract has to be renewed Annually? Every 3 years? What are your renewal options?
Who’s in charge of the procurement process? You may be the designated Records
Management Officer for your organization, but you will need: The purchasing/contract group Likely users must “buy in” Agreement on who sets the priorities for
service Recognition that the “disaster” may be small
or large scale Approval by your governing body
Who will negotiate the contract? Do they have a solid understanding of your business needs to secure the best vendor?
Other Considerations Can you opt for higher priority by
negotiating a retainer? Will the retainer be applied against services rendered?
If something happened to your materials, how long would it take to access them? Do you have 24/7 access? How do you get someone to the facility?
Is HIPAA a consideration in the custody/restoration of your records?
Response Time
Do you have a Continuity of Operations Plan (COOP)?
What kind of time will it take to respond to an event? For Emergency Services
Fire, Plumber, Repairmen For you and your staff For the Paper-Recovery Vendor
What events are importantbased on your geography?
Where are you located and what kinds of events do you expect every year? Florida – Hurricanes California – Earthquakes Northeast – Snow Low lying – Flooding What is it for you???
Vendor Capabilities How many linear feet of paper can be
recovered per hour per piece of equipment? Is the Vendor’s equipment wholly owned by
company, or is some subcontracted? Does the Vendor have a schedule of who
will respond after they are first notified? Process for chain of custody of records Where will the vendor set up “base camp”? What kind of turn-around time
Vendor Experience How much experience will you
require? What kinds of references do you
want to see? Past Performance
Program Name Owner / Architect Contract Amount / Percent Complete Completion Date
Be clear about the scope What media do you want used to
recreate your records? Film Electronic Back to paper?!?
Who can activate the contract Do you have other agencies/business units
who need access to this contract? If multiple agencies/business units can
access this contract, have you told the vendor who has priority?
Whose budget is used to pay for services?
Writing the RFP Plagiarize! Don’t reinvent the wheel
unnecessarily Each organization has their own
standard language and format Stick with the assistance particulars
that you might actually need Have a way to compare apples and
oranges between vendors/proposers
Computers vs Papers? Typically, there’s already a recovery plan
for computers, servers, databases, etc. Communication is key – talk to your IT
people and be sure they have a plan too! Computer recovery involves not only the
data, but also the equipment (machines, hardware)
There are specialized companies who handle this specialized service
Make sure your organization has an active and realistic plan in place
Paper or Paper Plus? Paper in a warehouse, awaiting its
retention time or its microfilming Paper in all those office filing cabinets
and individual desk drawers Microfilm? Microfiche? CDs? Maps or big drawings? Other media – besides computers?
Which documents first?
Are your documents prioritized? Do you store records for multiple
agencies? What gets saved first? Are your boxes segregated?
All things being equal, whose records are most important?
Does it make sense to secure your truly irreplaceable records elsewhere?
What type of paper recovery process will be used?
There are multiple types of document recovery/restoration techniques Vacuum Freeze Drying Molecular Sieve treatment Desiccant dehumidification HVAC decontamination Gamma Radiation (Remove Mold/Bacteria)
Are there techniques that cannot/should not be used on your records?
Paper Contamination Factors
Debris Sewage Silt Haz-Mat materials
Asbestos PCB Other petroleum products
Extras Re-filing, Re-indexing, Re-shelving Daily equipment rentals Small equipment purchases (shovels) Will they provide their own generators, or
does the vendor require electricity (water)?
Freight / Transportation Costs Travel / Per Diem or subsistence costs General Debris removal at your site to
gain building access Warranty on services
performed/document reproduction quality
Who does what? Inventory Packing Sorting/Identifying Prioritizing Labeling Loading/Unloading Re-shelving Security
Price per each item Audio tapes Blueprints Maps Compact disks Diskettes Manuscripts Laser disks Photographs
Slides Microfiche Microfilm Negatives X-rays Pack out boxes Pack in boxes
Price per hour
Will any of the vendor’s services be provided per hour in consultation fashion?
Does the vendor have per hour charges for anything not specifically covered in your contract?
Where will the actual restoration and recovery take place?
Where will the vendor set up “base camp”? Onsite Remote Location Vendor’s Site
How will the materials be transported?
What kind of turn-around time does the vendor give for recovery onsite? Offsite?
Before you go back in… Who declares whether the building is
safe to re-enter? Fire Marshal Building Inspector Facilities Management Some other authority?
Where are your emergency triage supplies? What will you provide? What will the vendor provide? Where will the
vendor set up “base camp”?
Before you go back in…(cont’d) Will you need assistance to secure
the structure itself? Water removal Temporary Power Temporary building repairs
Securing Walls Roof Repairs
What is your plan if your staff balk at going back in?
One solution
Set up scenarios and make pricing those responses a part of the evaluation process
Pick 3 events of varying significance
Mix up the media to be recovered
Orange County FL Sample Full RFP and evaluation grading sheets
being lifted to NACRC document library Understand that we decided our biggest
problem was going to be water damage We don’t get snow or many tornadoes Substitute your own boiler plate front
end
Scope of Work Location for services (Orange County FL; mostly mid-
Orange County) Volume of work undetermined (who knows what’s going
to happen?) Not for computer systems What we asked for
There shall be no retainer paid in order to keep the contract in effect. The resulting contract will be on an as-needed basis and used only in case of a disaster or emergency as so deemed by the County. The Contractor shall provide 24 hours per day, 7 days per week emergency response service. Response by phone after the first notification shall be within 2 hours. The Contractor shall have a representative on site after first notification within six (6) hours with the necessary equipment to be transferred within twelve (12) hours.
Selection Criteria and PointsCRITERIA
Qualifications of StaffQualifications of FirmTechnical ApproachM/WBE UtilizationLocationFee Proposal
TOTALWelfare Recipient Hires
WEIGHT
10 20 30 10 (minimum allowed) 10 (minimum allowed) 20
100 5 bonus points
Qualifications of Staff “Include a listing of all staff to be assigned to provide
the required services and resumes for each describing experience, training and education in the required consulting services.”
Reasoning and/or recommended changes Tell them to make the bios BRIEF
You don’t need or want 5 pages per person! Clarify the kind of training or certification staff has
WHEN did they get that training or certification? “Identify staff experience working with governmental
entities and list those projects.” Reasoning and/or recommended changes
Hold the government entities and projects for the firm’s qualifications
You just get the same information in two places
Qualifications of Firm #1 “Provide a description and history of the firm focusing
on previous governmental experience. Provide the number of years in business showing proof of a minimum of 5 years in document recovery, the current number of full-time employees and the excess labor that can be provided in case of an emergency.”
Reasoning and/or recommended changes Does governmental experience actually matter? Five years in business seemed a logical threshold so that
we weren’t getting a firm that sprung up in response to our 2004 hurricanes
Full-time employees seemed important to evaluate whether the firm could respond to more than one disaster
Excess labor? Probably would just hire locally and teach/train while on the job
Qualifications of Firm #2 “Identify your equipment inventory available
to Orange County within the Southeastern United States.
Identify the capacity to process in linear feet per hour for each piece of equipment at each location.
Identify equipment wholly owned by your firm, or which is available through subcontracting.”
Reasoning and/or recommended changes Making up the equipment list was VERY interesting We decided on Southeastern US as a logical staging
area for us We learned you can lease/rent anything!
Qualifications of Firm #3 “List at least five references, with a minimum
of two from governmental entity experience, for which the firm has performed similar work including the contact name, address, telephone number, email, and date of the contract.”
Reasoning and/or recommended changes 5 references seems about the standard 2 government entities seemed to make purchasing
people happier Make sure to ask for email so you can readily contact
the references Send the reference organizations questions and then
call to interview them
Technical Approach #1 “Provide a brief description of the firm’s
approach to the project. Identify the equipment and recovery methodologies you employ to recover water-damaged paper documents, microfilm, CDs, maps and plans, etc.
Provide a schedule (not later than) of arrival of your equipment in hours after first notification.
Identify how you ensure records are properly inventoried and identified as well as the chain of custodianship.”
Technical Approach #2 “Identify the methods and instruments used
to assess moisture content of records. Provide an operational plan for rapid
deployment of required supervisors to provide direction to County employee “first responders” to assess damage and take interim steps to minimize loss. Include number of people, and number of hours for arrival.
Should no power be available, contractor shall be responsible for providing all power required to operate equipment.”
Technical Approach #3 “Provide a plan for how your clients are
prioritized in the event of a widespread emergency in which there is multiple-client demand.
Identify your permanent dehumidifying equipment locations in the Southeastern United States and the capacity in pounds of water extracted per hour for each piece of equipment at each location.
Present options available to provide dehumidifying of buildings and mold and mildew treatment.”
Technical Approach #4 “Submit your warranty or
guarantee of your services. Confirm the firm’s agreement to
meet the minimum requirements of this Request for Proposal.
Proposers may offer alternative solutions/options to achieve successful completion of the scope of work herein.”
Technical Approach Notes This area provided us with the greatest
flexibility in evaluating proposals and the largest points to award/withhold.
We don’t know if we got this area “right” as we haven’t had to use the contract.
But we did get it in writing in advance from the firm with which we contracted.
We drew upon notes from prior NACRC and disaster recovery sessions. We tried to imagine many/most scenarios.
Fee Schedule #1 “Submit a rate schedule for equipment, supplies, and
staff services. Travel, lodging, and meals shall be at rates allowed to
public employees under state law per state statute 112.061
Identify the cost per cubic foot, separately, for each of the elements as listed in item below. Provide a cost for all recovery services.
freeze conventionally blast freeze freeze dry rapid freeze treat for mold and mildew packing and unpacking treatment for soot”
Fee Schedule #2 “Identify mobilization and demobilization
rates for equipment and staff. Identify your markup for any equipment
rented, and supplies purchased. Identify freight cost per mile by type of
equipment. Provide the cost to implement the recovery
process. Provide the cost of the Recovery Service. Describe, if offered, your pre-registration
service and cost associated for such service.”
Fee Schedule #3 “In order to adequately evaluate and
compare services provided by proposers, three (3) hypothetical scenarios have been created for which proposers must provide a written response using prices submitted on Attachment A, the scope of work provided, and the technical approach of your firm. These scenarios are located in Attachment B.”
Scenario #1: Roof Leak It has just been discovered that the roof of the County’s
Records Center has been leaking for some time now. It is estimated from the buckling ceiling tiles that this has occurred over the past six (6) months.
When the actual damage was assessed, 200 boxes (dimensions: 10x12x15 = 1.2 cubic feet) containing paper documents and microfilm had been severely damaged by water, resulting in mold and mildew.
Contents of the boxes are as follows: 7/8 copier and bond paper and file folders, and 1/8 microfilm.
Based on the limited information in this scenario, the scope of work, and the services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the 240 cubic feet of documents. It is understood that the type or method of restoration will vary depending upon the type of records involved.
Scenario #2: Water Pipe Burst A water pipe has just burst in a County building,
which houses permits and application forms of the County. Standing water, approximately 8” deep has seeped into the records (i.e., CDs, microfilm, paper, maps, and plans, etc.). The damaged records consist of 1/2 copier and bond paper and file folders, 1/8 CDs, 1/8 microfilm, and 1/4 building maps and plans.
The room size for this County office is approximately 20,000 sq. ft. Housed in this office space are records contained in 10 four-drawer vertical letter-size filing cabinets (@ 6.0 cubic feet per unit for approximately 60.0 cubic feet total); 10 five-drawer lateral, letter-size filing cabinets (@ 13.0 cubic feet per unit) for approximately 130.0 cubic feet of records.
Scenario #2 (continued) This space also contains approximately 50
workstation cubicles at 100 sq. ft. each, which is equivalent to an estimated 4 to 5 cubic feet of records housed at each workstation. The 50 workstations contain approximately 250 cubic feet of water-damaged records.
Based on the limited information in this scenario, scope of work, and services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the records. It is understood that the type or method of restoration will vary depending upon the type of records involved.
Scenario #3: Small Fire A coffee pot with a potential electrical shortage problem
was left plugged into an outlet too long and sparked a small fire. Smoke from the fire triggered the sprinkler system, which caused water damage to some of the records.
When the actual damage was assessed, 300 boxes (dimensions: 10x12x15 = 1.2 cubic feet) containing paper documents, microfilm, CDs, and maps had been moderately damaged by both water from the sprinklers and soot from the fire. The damaged records consist of ¾ copier and bond paper and file folders, 1/8 CDs, and 1/8 microfilm.
Based on the limited information in this scenario, scope of work, and services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the records. It is understood that the type or method of restoration will vary depending upon the type of records involved.
Fee Schedule Notes
Having the scenarios was very important
We had trouble figuring out how to evaluate all the “standard” information the proposers submitted
The scenarios gave us a window into comparing apples to apples side-by-side
Checking References You really have to do this part We designed a series of 10
questions (lifted to the NACRC document library) which we sent to the reference people in advance of our call to them
We had at least 2 people and usually 3 people on each call – and then we compared notes
Doing the Evaluation We created spreadsheets and allocated
points among the criteria for each category (lifted to the NACRC Document Library)
Each of the evaluators completed their own spreadsheet separately
We then met together and compared scores; we were very close to each other
What one person couldn’t find or figure out, another person did
The Contract We selected our finalist just as
Hurricanes Katrina and Rita plowed in Getting our contract fully executed
dragged out for 3 months – and we only needed their properly executed insurance form!
Since the contract was finally and fully signed in January 2006, we have not had to use the contract
But we’re sure sleeping better!
Additional Thoughts
Start now; don’t wait for your “season”
Anticipate delays along the way From first concept to actual
completion was 2+ years for us Writing the scenarios and considering
all the possibilities requires multiple brains and various styles
Lessons Learned from Katrina Having a pre-need contract crosses the
“T’s” and dots the “I’s” in advance. Remember to look at the whole process New Orleans Parish experience with their
vendor, “The work that they did was excellent but the collection department was awful. They relied on a technical line in the work authorization to keep our books in Chicago until they were paid eliminating the opportunity for an audit of their bill. I suggest that any pre arrangements include reviewing the billing and collection methods.”
Use your NACRC Contacts Use NACRC as a resource to find
vendors Talk to other jurisdictions about
their experiences Conferences Listserv Email Conference Calls
Questions & Answers What experiences have you had? Good
or bad? When will you have a pre-need
contract? How will you develop your potential
proposer list to which to send the RFP? Can you join with other organizations in
your area? Can you “tag along” to an existing
contract someone else has?
Additional Resources NACRC www.nacrc.org Florida DOS State Library &
Archives http://dlis.dos.state.fl.us/disasterrecovery/#HVAC
National Parks Service – Wet Collections Recovery http://www.nps.gov/hfc/products/cons/wet-recovery.htm#
Thanks and for any follow-up Carol Foglesong, Assistant Comptroller,
Orange County Comptroller’s Office, PO Box 38, Orlando, FL 32802; 407.836.5982; carol.foglesong@occompt.com
Paul Ketz, Broward County Records Division, 115 S. Andrews Ave., Room 120,
Ft. Lauderdale, FL 33301; 954.357.7272; pketz@broward.org