NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data...

38
NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations [email protected] / 206.667.5063

Transcript of NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data...

Page 1: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Stress Testing of Research Systems

Ron Hood, Manager Voice/Data Operations

[email protected] / 206.667.5063

Page 2: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Recent Painful Experiences&

How I Hope to Avoid Repeating Them in the

Future

Ron Hood, Manager Voice/Data Operations

[email protected] / 206.667.5063

Page 3: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Hutch Background

• Seattle, Washington

• ~2,500 Faculty & Staff

• 12 buildings on a 13 acre campus

• Stand-alone institution (not part of a university or hospital)

• Research only – no clinical care

Page 4: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Hutch IT Background

• Decentralized – 30 different IT groups.• “Center IT” is the largest @ ~60 FTE.• Housed in Administration division.• Desktop, File/Print, and Application support

for Admin & Clinical Research divisions.• Project Management support for Center IT.• Network, Information Security, and email for

entire center.

Page 5: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Ron’s Background

• Has-been network geek.• Newbie @ information security.• Minimal knowledge of servers, desktops,

software development, project management, and scientific applications.

• Responsible for voice, network, information security, and physical infrastructure.

Page 6: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Server Rooms: HVAC

Page 7: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Page 8: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Servers

Page 9: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

N

Servers

N

Page 10: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

UPS

N

UPS

Servers

N

Page 11: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

UPS

N

CRAC

CRAC

UPS

Servers

N

Page 12: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

UPS

N

CRAC

CRAC

UPS

Servers

N

Page 13: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

UPS

N

CRAC

CRAC

UPS

Servers: 50 Cabinets

N

Page 14: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

125 kW

N

CRAC

CRAC

125 kW

Servers: 50 Cabinets

N

Page 15: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

125 kW

N

CRAC

CRAC

125 kW

Servers: 50 CabinetsPower: 100 kW => 2 kW/Cabinet

N

Page 16: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

125 kW

N

20

T

20

T

125 kW

Servers: 50 CabinetsPower: 100 kW => 2 kW/Cabinet

N

Page 17: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

125 kW

N

20

T

20

T

125 kW

Servers: 50 CabinetsPower: 100 kW => 2 kW/Cabinet

Cooling: 56 kW => ~ 1 kW/Cabinet

N

Page 18: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Take-Away Lessons:

• Do the math.

Page 19: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Server Rooms: Power

Page 20: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Page 21: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Page 22: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Take-Away Lessons:

• Do the math.

• Understand the architecture.

Page 23: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Storage

Page 24: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Clinical Tray

Clinical

SAN in a Can

Page 25: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Clinical Tray Admin Tray

ClinicalAdmin

SAN in a Can

Page 26: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Clinical Tray Admin Tray

ClinicalAdmin

SAN in a Can

Page 27: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Clinical Tray Admin Tray

NVRAM

ClinicalAdmin

SAN in a Can

Page 28: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Take-Away Lessons:

• Do the math.

• Understand the architecture.

Page 29: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Take-Away Lessons:

• Do the math.

• Understand the architecture.

Page 30: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Networks

Page 31: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Page 32: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Take-Away Lessons:

• Do the math.

• Understand the architecture.

• Test regularly.

Page 33: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Page 34: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Take-Away Lessons:

• Do the math.

• Understand the architecture.

• Test regularly.

• Don’t touch it.

Page 35: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Where do we go from here?

Page 36: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Where do we go from here?

• We’ve learned some of the right things to do.

Page 37: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Where do we go from here?

• We’ve learned some of the right things to do.

• It remains to be seen if we will have the discipline to do them.

Page 38: NYC BRIITE – May 4, 2006 Stress Testing of Research Systems Ron Hood, Manager Voice/Data Operations rhood@fhcrc.orgrhood@fhcrc.org / 206.667.5063.

NYC BRIITE – May 4, 2006

Questions?