I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective...

59
I C S Integrated Communication Servic The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance, President and Board Chair, NYSERNet

Transcript of I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective...

Page 1: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

I C S Integrated Communication Services

The Network During the Crisis- A New York City Perspective

Doug Carlson, Director of Network ServicesTim Lance, President and Board Chair, NYSERNet

Page 2: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

New York University

Page 3: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

N Largest private university in US Located in Greenwich Village area of

New York City Concentrated North of Houston

(NoHo) up through 14th Street School of Medicine and NYU Hospital

near 21st Street Residence Halls extend down to

blocks away from Ground Zero

ew York University

Page 4: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Abilene

CA*NetBuffalo

Rochester

Syracuse

Albany

New York

Topology of the NYSERNet Network.

OC-3OC-12

legend

NYSERNet 2000 PoP

DS-3

Gateway Network

Abilene

vBNS+

Page 5: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Research Network

Connected Institutions (R&E Network) Amer. Museum of Natural History Columbia University Cornell University Hauptman-Woodward Institute * Marist College * New York University Pace University * Rensselaer Polytechnic Institute

Rochester Institute of Technology Rockefeller University SUNY Albany SUNY Binghamton SUNY Buffalo SUNY Stony Brook Syracuse University University of Rochester Weill Medical College

* Approved for Funding.

Page 6: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Manhattan Project Participants – Geographic Map

Page 7: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Before September 11th

Photograph by Doug Carlson – 7/2001

Page 8: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Impact of September 11th

Photograph by Doug Carlson – 9/23/2001

Page 9: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Relationship between 7 World Trade Center and 140 West Street prior to

September 11th.

Page 10: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

60 Hudson Street

140 West Street

World Trade Center

Images courtesy www.spaceim

ages.com.

Image of Manhattan from IKONOS Satellite taken

Summer 2000.

Page 11: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

140 West Street

60 Hudson Street

World Trade Center

Images courtesy www.spaceim

ages.com.

Image of Manhattan from IKONOS Satellite taken

shortly after the attack on the World Trade Center.

Page 12: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

World Trade Center complex showing

relationship between 7 World Trade Center and

140 West Street.

Page 13: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Exterior view of 140 West Street showing debris and damage from collapse of

7 World Trade Center.

Photographs Copyright 2001 Verizon Comm

unications. All rights reserved.

Page 14: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

An interior view of 140 West Street showing

damage from collapse of 7 World Trade Center.

Photographs Copyright 2001 Verizon Comm

unications. All rights reserved.

Page 15: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

N As a result of the attack…

CO at 140 West Street severely damaged. NYSERNet OC-48 SONET ring operational but no

longer a ring. NYSERNet R&E Network operational. NYSERNet POP at 60 Hudson Street operational. Abilene connections operational. vBNS+ connection down. R&E service for SUNY Stony Brook and AMNH is

down.

etwork Status - NYSERNet

Page 16: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

N As a result of the attack…

CO at 140 West Street severely damaged. AppliedTheory equipment largely undamaged. AppliedTheory Network operational. AppliedTheory POP at 60 Hudson Street

operational. Customers connecting to 60 Hudson

operational. Customers connecting at 140 West are down. Customers connecting via Garden City, Deer

Park, and White Plains are down.

etwork Status - Commodity

Page 17: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

RRestoring Service to Mt. Sinai & School of MedicinePrepared by Doug Carlson, New York University

estoring Service

Photo of 140 West Street. Copyright 2001 Verizon Communications. All rights reserved.

Page 18: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Post Attack

Phone systems immediately get overloadedLines by all payphones in NYU areaManhattan phone and cell systems overwhelmed

Access to Lower Manhattan impossible

Pedestrian access limited below 14th Steet (Union Square)

Pedestrians required to have reason to go beyond checkpoint and be able to show ID

Some students and others left ID behind

Page 19: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Post Attack

Initially, no traffic below 14th Street unless emergency or government vehicle

NYU worked with Mayor’s office to get some critical deliveries through to NYU locations

Most businesses and restaurants closed below 14th Street

NYU Administration establishes a Crisis Command Center

Initially, did not have sufficient phone or data service

Page 20: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Post Attack Campus phone system becomes overloaded for a period of

time (no dial-tone)

Internet connections provide only reliable information links to family, friends, co-workers:

Email Web Instant Messaging

VoIP (Cisco) phones used to communicate with staff unable to come into Manhattan and unable to get through on traditional telephone/cell systems

Page 21: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Post Attack

Information Technology Services (ITS) takes point in keeping NYU community informed via Web, mass-mailings via email, voice mail announcements and Help Desk services.

Mt. Sinai Hospital traffic routed through NYU.

Phone banks set up to allow students to call home for free.

Temporary shelter set up for students and staff who were evacuated.

Page 22: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Post Attack

Evacuated students and staff are relocated to hotels around the city:

Students given some money for essentials.

Arrangements are made to get new sets of books for students.

Loans of laptops and desktop computers offered.

Preparations made to increase dial-in capacity via ISP. (Eventually not needed due to students returning to their residence halls.)

Page 23: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

RColumbia, IP Telephony & The New York Academy of Medicine.Prepared by Alan Crosswell, Columbia University

estoring Service

Photo of 140 West Street and WTC 7 courtesy of the Westchester Emergency Communications Association, www.weca.org

Page 24: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Early Morning, September 11th

Dealing with start of semester file server performance issues after a failed Summer file server upgrade project.

Trying to get a handle on filtering Code Red with our Catalyst 6509’s.

Turning off many user network ports for compromised hosts day in and day out.

Page 25: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Post Attack, September 11th

Trying to find out where all missing staff are. Discover phone trunks are overloaded. Send email to friends asking them to call our families and

let them know we are OK.

Page 26: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Post Attack, September 11th

Administration establishes response team. They request that we:

Set up large lecture halls with CNN; Students are crowding around TV monitors in lounge areas.

Establish net2phone-like functionality so students can phone home.

Send a mass e-mail to the Columbia community from the President.

Page 27: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Getting CNN Out There, September 11th

Start planning to drag an IPTV encoder over to a cable box in a dorm.

Staff drag some coax from a lounge TV monitor to a large lecture hall.

Read on wg-multicast list that Northwestern has started multicasting CNN, but find our multicast connectivity is broken.

Page 28: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Getting Phones Up, September 11th

One staff member tests net2phone…. Send email to Videnet & others + CS SIP group requesting

use of PSTN gateways to route around local congestion. ~11am.

End up configuring H.323 Polycoms via UNC Chapel Hill, and

Cisco IP phones via Penn State, and SIP phones via 4 sites (Yale, Dynamicsoft, Nortel, Clarent).

Page 29: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Getting Phones Up, September 11th

Net2phone works but it’s not easy & requires a credit card….

Polycoms work but also not so easy…. CS puts 4 SIP phones in CS conference room. 2:20pm We put 2 Cisco IP phones via Penn State in student center

lobby. Just dial 8 and get a PSU dial tone and call home! 4:00pm.

Page 30: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

Getting Phones Up, September 11th

By the time we had the IP phones in place trunk congestion had declined.

We had many many other Videnet sites offer their H.323 – H.320 PSTN gateways.

Thanks to all of you!

Page 31: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

September 11th Onward

One of our dialup modem pools is out (and still out). A major expensive carrier.

Our free T1’s from a small carrier are up on our main pool. Weekly Email volume increased by 40% over prior week. Daily volume on Thursday 9/13 was 800,000 messages:

double last year’s.

Page 32: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

September 11th Onward

Filtering SIRCAM, nimda, Code Red, WTC viruses. Turning off lots of ports of compromised hosts and

attempting to deal with helping people reformat and reinstall their systems.

Established outbound P2P traffic shaping.

Page 33: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring Service

September 18th

After a week, NYAM’s T1 service via 140 West St is still down and low priority for restoration.

Set up an 802.11b link across Central Park and turn it up Wednesday 9/19.

Page 34: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

RRestoring Service to Rockefeller, Weill, & HSS.Prepared by Armand Gazes, The Rockefeller University

estoring Service

Photo of new cable being placed on West Street. Copyright 2001 Verizon Communications. All rights reserved.

Page 35: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Hospital for Special Surgery

The Rockefeller University

Weill Medical Center of CU

AppliedTheory

AppliedTheory Router

Internet

NYSERNet Switch

NYSERNet

Internet2

Network Connectivity Prior to September 11th.

Rockefeller Router

AppliedTheory Router

Rockefeller

Commodity Path

R&E Path

key

Page 36: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Hospital for Special Surgery

The Rockefeller University

Weill Medical Center of CU

AppliedTheory

AppliedTheory Router

Internet

NYSERNet Switch

NYSERNet

Internet2

Network Connectivity Subsequent to September 11th.

Rockefeller Router

AppliedTheory Router

Rockefeller

Commodity Path

R&E Path

key

Page 37: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Hospital for Special Surgery

The Rockefeller University

Weill Medical Center of CU

AppliedTheory

AppliedTheory Router

Internet

NYSERNet Switch

NYSERNet

Internet2

Commodity Service Restored.

NYSERNet Router

AppliedTheory Router

Buffalo

Rockefeller Router

AppliedTheory Router

Rockefeller

Commodity Path

R&E Path

key

Page 38: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

R Restoring commodity service…

To Rockefeller University, Weill Medical Center, and the Hospital for Special Surgery.

Service restored through Rockefeller’s R&E connection and emergency installation of a jumper between NYSERNet and AppliedTheory in Buffalo.

Key - cooperation among campus personnel, commercial service providers and NYSERNet.

Service restored by Wednesday afternoon, the 12th.

estoring Service

Page 39: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Restoring ServiceRestoring Commodity and R&E

Services.Tim Lance, NYSERNet.

New

York S

tock Exchange – P

hoto by Doug C

arlson

Page 40: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

AppliedTheory

AppliedTheoryNYSERNet

AppliedTheoryLong Island

NYSERNet Metro Ring & AppliedTheory Prior to September 11th.

NYSERNet

Verizon

60 Hudson St.

AppliedTheory

Verizon

140 West St.

Verizon

Broad St.

NYSERNet Metropolitan OC-48 SONET Ring

Page 41: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

AppliedTheory

NYSERNet

AppliedTheoryLong Island

NYSERNet Metro Ring & AppliedTheory After September 11th.

NYSERNet

60 Hudson St.

AppliedThory

Verizon

NYSERNet Metropolitan OC-48 SONET Ring

Verizon

Broad St.

Verizon

AppliedTheory

140 West St.

Page 42: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

AppliedTheory

NYSERNet

AppliedTheoryLong Island

Service Restored Utilizing NYSERNet SONET Ring.

NYSERNet

60 Hudson St.

AppliedThory

Verizon

NYSERNet Metropolitan OC-48 SONET Ring

Verizon

Broad St.

Verizon

AppliedTheory

140 West St.

Page 43: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

N Restoring commodity service…

Polytechnic University - commodity service restored on 10/1. Long Island – commodity traffic still transiting NYSERNet SONET

Ring. White Plains – connected to Garden City and then onto NYSERNet

Ring. Rockefeller Solution - evolving into standby service for all

customers having both NYSERNet R&E service and AppliedTheory commodity service.

West Street Service – a few commodity T1’s still down.

etwork Status - Commodity

Page 44: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Network Status – NYSERNet

Restoring R&E service…

American Museum of Natural History – service restored on 9/20.

SUNY Stony Brook – service restored on 9/25. vBNS+ – service restored almost immediately.

Page 45: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

O New York University extended service to Mt.Sinai and School

of Medicine. Hofstra University offered NYSERNet bandwidth on its OC3 if it

could be utilized to serve any impacted institutions. City University of New York hosted the New York City

Board of Education Web site to ensure that parents could remain informed on the status of their children’s schools.

American Museum of Natural History staff offered to serve as remote hands for any institution needing onsite service.

Columbia University provided New York Academy of Medicine with access via wireless service.

NYSERNet offered to allow AppliedTheory to utilize its Manhattan SONET ring to restore commodity service, if feasible.

ther Stories

Page 46: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Conclusions

Page 47: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

C Lessons learned…

The value of networks in sustaining communications.

The survivability of the technology. How much can be accomplished quickly when

the urgency is clear. The value in multiple independent paths to the

Internet. The value of government investment in science

and the often unanticipated nature of the return on that investment.

The value of developing an Emergency Response Plan

onclusions

Page 48: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Challenges Ahead

Networks are necessities, not luxuries…

Networks are now mission critical. Diverse capabilities must back each other up. Our constituents are flexible, we must be too. Merging technologies can lead to redundancy.

Page 49: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Challenges Ahead

Industry Contraction and Consolidation

Mergers, acquisitions, business failures. Continued geographic concentration. Reduced capital investment.

Page 50: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Challenges Ahead

Shifting Public Policy

H.R. 3162, The Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism (USA PATRIOT) Act of 2001.

Lack of controlling authority viewed as a deficiency.

House Science Committee considering Cyber security bill.

Senate Commerce Committee considering expanded government role.

Page 51: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Challenges Ahead

Prioritization of Limited Resources

Attracting, retaining and training staff. Funding. Don’t chase the bubble. Base next generation design on controlling the

transport medium, with physical and logical redundancy and ever more intelligence at the edge.

Page 52: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Responding

6

9

3

2

4

5

14

15

16

Brooklyn

New Jersey

Bronx

Redundant Fibers

FiberTrunk

Loop 1A

Redundant Fibers

New School

Arts4All

CooperUnion

NYU (ACF)

NYU (MC)

CUNY(CIS)

NYPL

AMNH

Rockefeller

MSKCC (HQ)

MSKCC(DC)

Columbia

NYPH (CPC)

Fordham U.Yeshiva U.

Polytechnic

1

15C15BFordham U.TarrytownFordham U.

MarymountTarrytown

14C

16A

NYPH (AP)

Yeshiva(Uptown)

CUNY(City College)

NYSBC

NYPH (WMC)

11B

11D

MSKCC (64th)

14E NYPH (E61)

NYPL (PA)

Fordham (LC)

13AMSKCC (53rd)

11A

14A

NYPL(Science/DC)

16CYeshiva(Midtown-1)

16B Yeshiva (Law)

8A7C

16D

NYPH (PFS)

Yeshiva(Midtown-2)

Upstate NY

11C

14B

15A

8B

7A

10

12

814D

7B

MSKCC (PCB)

13

11

7

NYPL (Annex)

ColumbiaController

CUNY (Grad.Center)

NYPH (DC)

MSKCC(Admin)

NYSERNet POP

NYSERNet’s Manhattan Project.Developing a Dark Fiber Network in the Manhattan area.

Page 53: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Manhattan Project

Advancing Technology for our Members

NYSERNet as catalyst, project manager, business manager.

Seeking to serve needs of the entire community for: redundancy, security, flexibility, performance, cost-control.

Developing a model for future projects. Urgency has risen since September 11th.

Page 54: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Manhattan Project

Goals

Implement a network that provides economical, high-performance access to:

the NYSERNet network, commodity Internet providers, the public switched telephone network, plus the ability to implement private intra- and inter-

institutional networks, and unlimited potential for performance upgrades.

Page 55: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Manhattan Project

Participating Institutions (initial list) Amer. Museum of Natural History Arts4All Columbia University Cooper Union City University of New York Fordham University Memorial Sloan Kettering New School University

New York Polytechnic Institute New York Presbyterian Hospital New York Public Library New York University The Rockefeller University Weill Medical College Yeshiva University

Page 56: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Manhattan Project

Project Objectives

Survey potential participants about their requirements.

Create network design based upon the survey. Preliminary costs determination based upon

design. Review costs with participants and secure

commitment.

Network and colocation construction commences.

First participant online.

2001

2003

2002

Page 57: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Actions at NYU In negotiations with vendor for second

commodity Internet link with diverse routing within Manhattan

Seeking funding for distributing infrastructure in multiple locations (e.g., network core, key servers, etc.)

Working with FEMA to determine appropriate infrastructure upgrades

Risk assessment and response planning

Page 58: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

Washington Square Park – 9/23/2001 Photo by D

oug Carlson

Page 59: I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance,

NY

Harbor – 2/2/2002 Photo by D

oug Carlson