Tropical Storm Allison & its impact on Harris County Binbin Chen.
Tropical Storm Allison Recovery
description
Transcript of Tropical Storm Allison Recovery
![Page 1: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/1.jpg)
Allison 2001
Charles Chambers
Information Technology
2001
Tropical Storm AllisonRecovery
Tropical Storm AllisonRecovery
![Page 2: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/2.jpg)
Allison 2001Time LineTime Line
• Saturday– flooding begins (1 A.M.) – IT team on campus
– Additional IT staff arrive – pumping and power are critical
– Institutional priorities are focused – Non-IT
• Sunday– Pumping of network node facilities begin
– Power restored to UHS data center
– Internet connection reestablished
– 713-743-1000 call center established
– Systems and servers returned to service
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22nd
![Page 3: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/3.jpg)
Allison 2001Time LineTime Line
• Monday, June 11th– WAN services restored except UHV– IP telephones activated in critical areas– Relocated some servers to Data Center
• Tuesday, June 12th
– Internet connectivity unstable due to ISP– Servers in Data Center grow by 20– UHV network connection restored– Desktop assessment team inventories campus– Network teams inspect buildings for damage
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22nd
![Page 4: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/4.jpg)
Allison 2001Time LineTime Line
• Wednesday, June 13th
– UH-CVN services restored, classes held.– Payroll processed– 50% of phone system returned to service
• Thursday, June 14th – Data backbone restored– Data connectivity to opened buildings– Order Equipment for Storm Solutions for
Hofheinz Pavillion
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22nd
![Page 5: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/5.jpg)
Allison 2001Time LineTime Line
• Friday, June 15th
– Majority of phone system restored
• Saturday, June 16th – Equipment for Hofheinz Pavillion arrives,
installed
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22nd
![Page 6: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/6.jpg)
Allison 2001Time LineTime Line
• Sunday, June 17th – Hofheinz Pavillion available
• 270 IP telephones
• 314 data ports
• IT support booth created
• Monday, June 18th – University Re-Opens– First occupants moves into Hofheinz Pavillion
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22nd
![Page 7: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/7.jpg)
Allison 2001Time LineTime Line
• Tuesday– Voice Mail system restored– Touch Tone Registration restored– Remainder of phone system restored
• Wednesday– Majority of SWB trunking problems corrected
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22nd
![Page 8: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/8.jpg)
Allison 2001Time LineTime Line
• Thursday• Friday
– Reconfiguration of Hofheinz Pavillion network
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22nd
![Page 9: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/9.jpg)
Allison 2001Technology PrioritiesTechnology Priorities
• Phone System• Internet Connectivity• Inter-campus Network• Campus Data Network• Interactive Video Network• Data Center Operation
Infrastructure
![Page 10: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/10.jpg)
Allison 2001Technology PrioritiesTechnology Priorities
• E-mail• Web Site• WebCT• IVR for Registration/Records• Payroll• Financial Application
Services
![Page 11: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/11.jpg)
Allison 2001Technology PrioritiesTechnology Priorities
• Inventory• Repair• Replacement Computers• Data restores
Desktop Computers
![Page 12: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/12.jpg)
Allison 2001Technology PrioritiesTechnology Priorities
• Used basketball pavilion• Boxes became offices• Basketball floor
– became labs and cubicles
Temporary Space - Holfheinz
![Page 13: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/13.jpg)
Allison 2001Holfheinz PavillionHolfheinz Pavillion
• 27 boxes• Cisco IP telephony – 270 phones• Switched data network
– 270 ports in offices– 144 ports on basketball court– Over 100,000 feet of copper and fiber
• IT support booth– Voice and data network support– Desktop computer support
From Basketball to Academics
![Page 14: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/14.jpg)
Allison 2001Recovery ResourcesRecovery Resources
• Over xxx person hours (UH Staff)• Loss and Business Resumption
– Costs estimated in millions
• Over 250 IT related staff involved
![Page 15: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/15.jpg)
Allison 2001Lessons LearnedLessons Learned
• Move IT facilities above ground• Move generators above ground• Flood doors on underground entrances• Established plans helped . . .
– The right people at the right places, making the right decisions was the key.
• Mobilize support efforts early– Food
– Sleeping Quarters
![Page 16: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/16.jpg)
Allison 2001Heroic EffortsHeroic Efforts
• Pumps/Pumping• Saving the Animals• Getting Primary DID in service• Internet Connectivity• Holfheinz Pavillion
![Page 17: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/17.jpg)
Allison 2001How the Speedy RecoveryHow the Speedy Recovery
• Temporary fixes• Great minds and enormous goals• Dedicated staff• Very long hours• Multiple plans• Quick vendor response• Elimination of bureaucracy• Communication of status and goals• Central coordination point
![Page 18: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/18.jpg)
Allison 2001Starting RecoveryStarting Recovery
• Pumping and Power• Starting pumping PGH on Saturday• Moved pumps to animal lab (SR2)• Sunday power restored to
Computing Center• Internet Connection
– Reconfigured to computing center– Web and e-mail up first
![Page 19: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/19.jpg)
Allison 2001Recovery PartnersRecovery Partners
• Herculean efforts by all!– Pumping water– Removing damaged equipment– Designing and installing new systems– Power restoration
• UH Support– College volunteers– Facilities
• Vendor response was great!– Staff on site (over 100)– Rapid deployment of equipment
![Page 20: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/20.jpg)
Allison 2001Initial ImpactInitial Impact
• Primary Technology Areas Affected– UHS Data Center– Network Node in PGH– Network Node in UCU– Headend Facility in Moody
![Page 21: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/21.jpg)
Allison 2001UHS Data CenterUHS Data Center
• Lost power• No flooding or other damage
![Page 22: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/22.jpg)
Allison 2001Network Node (PGH)Network Node (PGH)
• Services lost– Voice, data, video for UH– Internet and interactive video for all
UHS
• Damage– Water 18 inches above raised floor– Power systems total loss
![Page 23: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/23.jpg)
Allison 2001Network Node (PGH)Network Node (PGH)
Telephone and Voice Mail Systems
![Page 24: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/24.jpg)
Allison 2001Network Node (UCU)Network Node (UCU)
• Services lost– Voice network for 1/3 of campus– Trunk lines from SWB– High capacity circuits from SWB
• Damage– Telephone Switch corroded– Wire center destroyed
• 3600 stations
![Page 25: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/25.jpg)
Allison 2001Network Node (UCU)Network Node (UCU)
Telephone system
![Page 26: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/26.jpg)
Allison 2001Network Node (UCU)Network Node (UCU)
SWB
Interface
![Page 27: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/27.jpg)
Allison 2001Network Node (UCU)Network Node (UCU)
Wire Center Frame
![Page 28: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/28.jpg)
Allison 2001Headend Facility (Moody)Headend Facility (Moody)
• Services Lost– CATV Network
![Page 29: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/29.jpg)
Allison 2001Hofheinz PavillionHofheinz PavillionHome of “Phi Slamma Jama”
![Page 30: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/30.jpg)
Allison 2001Hofheinz PavillionHofheinz Pavillion
Wiring was pretty open…
![Page 31: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/31.jpg)
Allison 2001
An wide shot view of the inner-city of Houston, Texas
![Page 32: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/32.jpg)
Allison 2001
Flood waters invade the City with no Mercy
![Page 33: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/33.jpg)
Allison 2001
Freeway Entrance ramp into downtown from the
North
![Page 34: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/34.jpg)
Allison 2001
Freeway Exit ramp
![Page 35: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/35.jpg)
Allison 2001
A view of Allen-Parkway
![Page 36: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/36.jpg)
Allison 2001
Raging waters in the Tunnel System Downtown
![Page 37: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/37.jpg)
Allison 2001
Another view of the Tunnel System Downtown
![Page 38: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/38.jpg)
Allison 2001
And...Another view of the Tunnel System Downtown
![Page 39: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/39.jpg)
Allison 2001
Canoeing down HWY 59 near the Compaq Center
![Page 40: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/40.jpg)
Allison 2001
A view of 1 of 3 of the Suspension Bridges that crosses over HWY 59 near the Compaq
Center
![Page 41: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/41.jpg)
Allison 2001
A view of the Houston’s Channel 11 - KHOU - Television Studio
![Page 42: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/42.jpg)
Allison 2001
A view of Interstate
Highway I-10
![Page 43: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/43.jpg)
Allison 2001
Jet Sking in the flood waters on Interstate Highway I-10
![Page 44: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/44.jpg)
Allison 2001
A view of a 3-way Intersection located at Allen Parkway and Sanford Street
![Page 45: Tropical Storm Allison Recovery](https://reader035.fdocuments.net/reader035/viewer/2022062323/56815881550346895dc5e32e/html5/thumbnails/45.jpg)
Allison 2001Here’s Who did it!Here’s Who did it!
CTS “Crew” at Storm Allison Luncheon