I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective...
-
Upload
hilary-clarke -
Category
Documents
-
view
221 -
download
0
Transcript of I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective...
I C S Integrated Communication Services
The Network During the Crisis- A New York City Perspective
Doug Carlson, Director of Network ServicesTim Lance, President and Board Chair, NYSERNet
New York University
N Largest private university in US Located in Greenwich Village area of
New York City Concentrated North of Houston
(NoHo) up through 14th Street School of Medicine and NYU Hospital
near 21st Street Residence Halls extend down to
blocks away from Ground Zero
ew York University
Abilene
CA*NetBuffalo
Rochester
Syracuse
Albany
New York
Topology of the NYSERNet Network.
OC-3OC-12
legend
NYSERNet 2000 PoP
DS-3
Gateway Network
Abilene
vBNS+
Research Network
Connected Institutions (R&E Network) Amer. Museum of Natural History Columbia University Cornell University Hauptman-Woodward Institute * Marist College * New York University Pace University * Rensselaer Polytechnic Institute
Rochester Institute of Technology Rockefeller University SUNY Albany SUNY Binghamton SUNY Buffalo SUNY Stony Brook Syracuse University University of Rochester Weill Medical College
* Approved for Funding.
Manhattan Project Participants – Geographic Map
Before September 11th
Photograph by Doug Carlson – 7/2001
Impact of September 11th
Photograph by Doug Carlson – 9/23/2001
Relationship between 7 World Trade Center and 140 West Street prior to
September 11th.
60 Hudson Street
140 West Street
World Trade Center
Images courtesy www.spaceim
ages.com.
Image of Manhattan from IKONOS Satellite taken
Summer 2000.
140 West Street
60 Hudson Street
World Trade Center
Images courtesy www.spaceim
ages.com.
Image of Manhattan from IKONOS Satellite taken
shortly after the attack on the World Trade Center.
World Trade Center complex showing
relationship between 7 World Trade Center and
140 West Street.
Exterior view of 140 West Street showing debris and damage from collapse of
7 World Trade Center.
Photographs Copyright 2001 Verizon Comm
unications. All rights reserved.
An interior view of 140 West Street showing
damage from collapse of 7 World Trade Center.
Photographs Copyright 2001 Verizon Comm
unications. All rights reserved.
N As a result of the attack…
CO at 140 West Street severely damaged. NYSERNet OC-48 SONET ring operational but no
longer a ring. NYSERNet R&E Network operational. NYSERNet POP at 60 Hudson Street operational. Abilene connections operational. vBNS+ connection down. R&E service for SUNY Stony Brook and AMNH is
down.
etwork Status - NYSERNet
N As a result of the attack…
CO at 140 West Street severely damaged. AppliedTheory equipment largely undamaged. AppliedTheory Network operational. AppliedTheory POP at 60 Hudson Street
operational. Customers connecting to 60 Hudson
operational. Customers connecting at 140 West are down. Customers connecting via Garden City, Deer
Park, and White Plains are down.
etwork Status - Commodity
RRestoring Service to Mt. Sinai & School of MedicinePrepared by Doug Carlson, New York University
estoring Service
Photo of 140 West Street. Copyright 2001 Verizon Communications. All rights reserved.
Restoring Service
Post Attack
Phone systems immediately get overloadedLines by all payphones in NYU areaManhattan phone and cell systems overwhelmed
Access to Lower Manhattan impossible
Pedestrian access limited below 14th Steet (Union Square)
Pedestrians required to have reason to go beyond checkpoint and be able to show ID
Some students and others left ID behind
Restoring Service
Post Attack
Initially, no traffic below 14th Street unless emergency or government vehicle
NYU worked with Mayor’s office to get some critical deliveries through to NYU locations
Most businesses and restaurants closed below 14th Street
NYU Administration establishes a Crisis Command Center
Initially, did not have sufficient phone or data service
Restoring Service
Post Attack Campus phone system becomes overloaded for a period of
time (no dial-tone)
Internet connections provide only reliable information links to family, friends, co-workers:
Email Web Instant Messaging
VoIP (Cisco) phones used to communicate with staff unable to come into Manhattan and unable to get through on traditional telephone/cell systems
Restoring Service
Post Attack
Information Technology Services (ITS) takes point in keeping NYU community informed via Web, mass-mailings via email, voice mail announcements and Help Desk services.
Mt. Sinai Hospital traffic routed through NYU.
Phone banks set up to allow students to call home for free.
Temporary shelter set up for students and staff who were evacuated.
Restoring Service
Post Attack
Evacuated students and staff are relocated to hotels around the city:
Students given some money for essentials.
Arrangements are made to get new sets of books for students.
Loans of laptops and desktop computers offered.
Preparations made to increase dial-in capacity via ISP. (Eventually not needed due to students returning to their residence halls.)
RColumbia, IP Telephony & The New York Academy of Medicine.Prepared by Alan Crosswell, Columbia University
estoring Service
Photo of 140 West Street and WTC 7 courtesy of the Westchester Emergency Communications Association, www.weca.org
Restoring Service
Early Morning, September 11th
Dealing with start of semester file server performance issues after a failed Summer file server upgrade project.
Trying to get a handle on filtering Code Red with our Catalyst 6509’s.
Turning off many user network ports for compromised hosts day in and day out.
Restoring Service
Post Attack, September 11th
Trying to find out where all missing staff are. Discover phone trunks are overloaded. Send email to friends asking them to call our families and
let them know we are OK.
Restoring Service
Post Attack, September 11th
Administration establishes response team. They request that we:
Set up large lecture halls with CNN; Students are crowding around TV monitors in lounge areas.
Establish net2phone-like functionality so students can phone home.
Send a mass e-mail to the Columbia community from the President.
Restoring Service
Getting CNN Out There, September 11th
Start planning to drag an IPTV encoder over to a cable box in a dorm.
Staff drag some coax from a lounge TV monitor to a large lecture hall.
Read on wg-multicast list that Northwestern has started multicasting CNN, but find our multicast connectivity is broken.
Restoring Service
Getting Phones Up, September 11th
One staff member tests net2phone…. Send email to Videnet & others + CS SIP group requesting
use of PSTN gateways to route around local congestion. ~11am.
End up configuring H.323 Polycoms via UNC Chapel Hill, and
Cisco IP phones via Penn State, and SIP phones via 4 sites (Yale, Dynamicsoft, Nortel, Clarent).
Restoring Service
Getting Phones Up, September 11th
Net2phone works but it’s not easy & requires a credit card….
Polycoms work but also not so easy…. CS puts 4 SIP phones in CS conference room. 2:20pm We put 2 Cisco IP phones via Penn State in student center
lobby. Just dial 8 and get a PSU dial tone and call home! 4:00pm.
Restoring Service
Getting Phones Up, September 11th
By the time we had the IP phones in place trunk congestion had declined.
We had many many other Videnet sites offer their H.323 – H.320 PSTN gateways.
Thanks to all of you!
Restoring Service
September 11th Onward
One of our dialup modem pools is out (and still out). A major expensive carrier.
Our free T1’s from a small carrier are up on our main pool. Weekly Email volume increased by 40% over prior week. Daily volume on Thursday 9/13 was 800,000 messages:
double last year’s.
Restoring Service
September 11th Onward
Filtering SIRCAM, nimda, Code Red, WTC viruses. Turning off lots of ports of compromised hosts and
attempting to deal with helping people reformat and reinstall their systems.
Established outbound P2P traffic shaping.
Restoring Service
September 18th
After a week, NYAM’s T1 service via 140 West St is still down and low priority for restoration.
Set up an 802.11b link across Central Park and turn it up Wednesday 9/19.
RRestoring Service to Rockefeller, Weill, & HSS.Prepared by Armand Gazes, The Rockefeller University
estoring Service
Photo of new cable being placed on West Street. Copyright 2001 Verizon Communications. All rights reserved.
Hospital for Special Surgery
The Rockefeller University
Weill Medical Center of CU
AppliedTheory
AppliedTheory Router
Internet
NYSERNet Switch
NYSERNet
Internet2
Network Connectivity Prior to September 11th.
Rockefeller Router
AppliedTheory Router
Rockefeller
Commodity Path
R&E Path
key
Hospital for Special Surgery
The Rockefeller University
Weill Medical Center of CU
AppliedTheory
AppliedTheory Router
Internet
NYSERNet Switch
NYSERNet
Internet2
Network Connectivity Subsequent to September 11th.
Rockefeller Router
AppliedTheory Router
Rockefeller
Commodity Path
R&E Path
key
Hospital for Special Surgery
The Rockefeller University
Weill Medical Center of CU
AppliedTheory
AppliedTheory Router
Internet
NYSERNet Switch
NYSERNet
Internet2
Commodity Service Restored.
NYSERNet Router
AppliedTheory Router
Buffalo
Rockefeller Router
AppliedTheory Router
Rockefeller
Commodity Path
R&E Path
key
R Restoring commodity service…
To Rockefeller University, Weill Medical Center, and the Hospital for Special Surgery.
Service restored through Rockefeller’s R&E connection and emergency installation of a jumper between NYSERNet and AppliedTheory in Buffalo.
Key - cooperation among campus personnel, commercial service providers and NYSERNet.
Service restored by Wednesday afternoon, the 12th.
estoring Service
Restoring ServiceRestoring Commodity and R&E
Services.Tim Lance, NYSERNet.
New
York S
tock Exchange – P
hoto by Doug C
arlson
AppliedTheory
AppliedTheoryNYSERNet
AppliedTheoryLong Island
NYSERNet Metro Ring & AppliedTheory Prior to September 11th.
NYSERNet
Verizon
60 Hudson St.
AppliedTheory
Verizon
140 West St.
Verizon
Broad St.
NYSERNet Metropolitan OC-48 SONET Ring
AppliedTheory
NYSERNet
AppliedTheoryLong Island
NYSERNet Metro Ring & AppliedTheory After September 11th.
NYSERNet
60 Hudson St.
AppliedThory
Verizon
NYSERNet Metropolitan OC-48 SONET Ring
Verizon
Broad St.
Verizon
AppliedTheory
140 West St.
AppliedTheory
NYSERNet
AppliedTheoryLong Island
Service Restored Utilizing NYSERNet SONET Ring.
NYSERNet
60 Hudson St.
AppliedThory
Verizon
NYSERNet Metropolitan OC-48 SONET Ring
Verizon
Broad St.
Verizon
AppliedTheory
140 West St.
N Restoring commodity service…
Polytechnic University - commodity service restored on 10/1. Long Island – commodity traffic still transiting NYSERNet SONET
Ring. White Plains – connected to Garden City and then onto NYSERNet
Ring. Rockefeller Solution - evolving into standby service for all
customers having both NYSERNet R&E service and AppliedTheory commodity service.
West Street Service – a few commodity T1’s still down.
etwork Status - Commodity
Network Status – NYSERNet
Restoring R&E service…
American Museum of Natural History – service restored on 9/20.
SUNY Stony Brook – service restored on 9/25. vBNS+ – service restored almost immediately.
O New York University extended service to Mt.Sinai and School
of Medicine. Hofstra University offered NYSERNet bandwidth on its OC3 if it
could be utilized to serve any impacted institutions. City University of New York hosted the New York City
Board of Education Web site to ensure that parents could remain informed on the status of their children’s schools.
American Museum of Natural History staff offered to serve as remote hands for any institution needing onsite service.
Columbia University provided New York Academy of Medicine with access via wireless service.
NYSERNet offered to allow AppliedTheory to utilize its Manhattan SONET ring to restore commodity service, if feasible.
ther Stories
Conclusions
C Lessons learned…
The value of networks in sustaining communications.
The survivability of the technology. How much can be accomplished quickly when
the urgency is clear. The value in multiple independent paths to the
Internet. The value of government investment in science
and the often unanticipated nature of the return on that investment.
The value of developing an Emergency Response Plan
onclusions
Challenges Ahead
Networks are necessities, not luxuries…
Networks are now mission critical. Diverse capabilities must back each other up. Our constituents are flexible, we must be too. Merging technologies can lead to redundancy.
Challenges Ahead
Industry Contraction and Consolidation
Mergers, acquisitions, business failures. Continued geographic concentration. Reduced capital investment.
Challenges Ahead
Shifting Public Policy
H.R. 3162, The Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism (USA PATRIOT) Act of 2001.
Lack of controlling authority viewed as a deficiency.
House Science Committee considering Cyber security bill.
Senate Commerce Committee considering expanded government role.
Challenges Ahead
Prioritization of Limited Resources
Attracting, retaining and training staff. Funding. Don’t chase the bubble. Base next generation design on controlling the
transport medium, with physical and logical redundancy and ever more intelligence at the edge.
Responding
6
9
3
2
4
5
14
15
16
Brooklyn
New Jersey
Bronx
Redundant Fibers
FiberTrunk
Loop 1A
Redundant Fibers
New School
Arts4All
CooperUnion
NYU (ACF)
NYU (MC)
CUNY(CIS)
NYPL
AMNH
Rockefeller
MSKCC (HQ)
MSKCC(DC)
Columbia
NYPH (CPC)
Fordham U.Yeshiva U.
Polytechnic
1
15C15BFordham U.TarrytownFordham U.
MarymountTarrytown
14C
16A
NYPH (AP)
Yeshiva(Uptown)
CUNY(City College)
NYSBC
NYPH (WMC)
11B
11D
MSKCC (64th)
14E NYPH (E61)
NYPL (PA)
Fordham (LC)
13AMSKCC (53rd)
11A
14A
NYPL(Science/DC)
16CYeshiva(Midtown-1)
16B Yeshiva (Law)
8A7C
16D
NYPH (PFS)
Yeshiva(Midtown-2)
Upstate NY
11C
14B
15A
8B
7A
10
12
814D
7B
MSKCC (PCB)
13
11
7
NYPL (Annex)
ColumbiaController
CUNY (Grad.Center)
NYPH (DC)
MSKCC(Admin)
NYSERNet POP
NYSERNet’s Manhattan Project.Developing a Dark Fiber Network in the Manhattan area.
Manhattan Project
Advancing Technology for our Members
NYSERNet as catalyst, project manager, business manager.
Seeking to serve needs of the entire community for: redundancy, security, flexibility, performance, cost-control.
Developing a model for future projects. Urgency has risen since September 11th.
Manhattan Project
Goals
Implement a network that provides economical, high-performance access to:
the NYSERNet network, commodity Internet providers, the public switched telephone network, plus the ability to implement private intra- and inter-
institutional networks, and unlimited potential for performance upgrades.
Manhattan Project
Participating Institutions (initial list) Amer. Museum of Natural History Arts4All Columbia University Cooper Union City University of New York Fordham University Memorial Sloan Kettering New School University
New York Polytechnic Institute New York Presbyterian Hospital New York Public Library New York University The Rockefeller University Weill Medical College Yeshiva University
Manhattan Project
Project Objectives
Survey potential participants about their requirements.
Create network design based upon the survey. Preliminary costs determination based upon
design. Review costs with participants and secure
commitment.
Network and colocation construction commences.
First participant online.
2001
2003
2002
Actions at NYU In negotiations with vendor for second
commodity Internet link with diverse routing within Manhattan
Seeking funding for distributing infrastructure in multiple locations (e.g., network core, key servers, etc.)
Working with FEMA to determine appropriate infrastructure upgrades
Risk assessment and response planning
Washington Square Park – 9/23/2001 Photo by D
oug Carlson
NY
Harbor – 2/2/2002 Photo by D
oug Carlson