7/28/2019 APW President BICSI-MAIT Sept09 Rev1
1/55
Datacenter 'Last Mile' Challenges
- A Rack Perspective
S.Venkatraman
10 th September09
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
2/55
Power Management
ThermalManagement
Level 1
Securitymanagement Cable
ManagementEnvironmentMonitoring
Spacemanagement
Thermal planning &Consultancy
Services - CFD
ThermalManagement
Level 2
DATA CENTER LAST MILE RACK PERSPECTIVE
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
3/55
Thermal Management & Planning
Cooling /Air flow in the Data Center
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
4/55
Thermal&
Vapour Barrier
Q1
Q1
Q1 Q1Q2C1
R1 R2 R3
R1, R2, R3 Racks
Q1 Outside Heat
C1 PAC Cooling
C1 = Q1 + Q2
THERMAL EQUILIBRIUM
Eliminate Q1 using thermal & vapour barrier to save on Opex
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
5/55
THERMAL EQUILIBRIUM
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
6/55
THERMAL MANAGEMENT - BASICS
Q = M Cp T
T = (T l - Te ) Q = 1.08 x cfm x T
Limitations
Precision AC T across evaporator coil cannot
exceed 10C (std products)
Better heat removal
By improving air flow
Effective utilization of cold air
Key factor Air quantity & Distribution
Te Tl
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
7/55
THERMAL MANAGEMENT - BASICS
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
8/55
LETS BIFURCATE THE PROBLEM
Low & medium density cooling Focus on Air flow management
High density cooling
Focus on using chilled water as themedium of cooling
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
9/55
AIR FLOW MANAGEMENT
Under floor air management
Above floor air management (includingabove the false ceiling)
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
10/55
THE P ROBLEM : Subfloor Air Velocity Air Handlers generate a great deal of air velocity in
the subfloor, causing a reduction in static pressure,which results in low-flowing and negative-flowing tiles.
In many cases, simply adding additional Air Handlers
creates more velocity, causing more hot spots tooccur.
As additional or hotter-running equipment is added to
the room, the hot spots spread and cause the entireroom to run warmer
The Goal - Slow down the subfloor velocity, therebyincreasing static pressure
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
11/55
CURRENT SITUATION
This typically requires a lower air handler temperature set pointand/or additional air handlers to keep the equipment cool, which
requires increasing the energy consumption being used.
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
12/55
DID YOU KNOW The average data center has 2x the number of Air
Handlers actually needed to cool the equipment
Approximately 60% of all cool air produced by Air Handlers never actually hits the heat load (Servers,Hubs, Routers, Switches, etc)
ASHRAE (American Society of Heating, Refrigerationand Air- Conditioning Engineers), is encouraging datacenter managers to raise the temperature set points tolower energy costs
95% of all data centers are paying a large energypenalty caused by poor subfloor air distribution
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
13/55
Eliminate hot spots
Direct more cool air to the heat load
Completely balance your subfloor air
Allow you to raise the temperature set pointson your air handlers
Reduce your energy costs from 23% - 42%
WHY SUBFLOOR AIR MANAGEMENT
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
14/55
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
15/55
Below-Floor CFDs show us howthe air flows from the Computer Room Air Conditioner (CRAC)through the subfloor plenum upinto the ambient room.
STEP ONE A Below-Floor CFD AnalysisIncludes Before and After: CFM Readings
Cable Cutout Leakage Lowest/Highest Flowing Tiles Velocity/Vector/Pressure Readings Deliverable
Before
After
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
16/55
STEP TWO Air Balancers
By balancing the subfloor pressure,
we manage it and can properly deliver air to the heat load.
Air balancers are strategically placed,using our intellectual properties, toenhance and balance static pressure
RESULTS FROM INSTALLED Air Balancers
Every perforated tile within the data
center will produce a significantlyhigher plume of air.
Each perforated tile will providegreater cooling.
Helps eliminate hot spots Increased air stratification
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
17/55
STEP THREE
Eliminate Bypass
Airflow
Air Plumes Before FlowLogix
Air Plumes After FlowLogix
Between 100 600 CFM is leakedthrough a cable cutout opening; thiscauses loss of CFM and a decrease instatic pressure
The mixing of hot and cold air increases energy consumption.
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
18/55
With Riser AddedThe performance of that same 30-tonunit with improved air stratificationand a return air temperature of 31C
the unit will put out more than 30tons.
At the higher return air temperatures,the Air Handler capacity is increasedand the chiller plant operates moreefficiently.
Increase Return Air Temperature
A 30 ton nominal, down-flow, chilledwater Air Handler might have a sensible
capacity of 27 tons when operated at22C but only 23 tons when operated at20C. Eighteen percent more capacity atthe higher operating temperature! Froma cost viewpoint, it might be possible toeliminate one in five Air Handlers.
Please view ASHRAE TC9.9 & ambient conditions
STEP FOUR
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
19/55
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
20/55
Based on CFD analysis and corrective actionthereof
Average reduction in data center energy costs rangingfrom 23% - 42%
Drops of cold aisle temperature ranging from 3 11degrees
Elimination of hot spots
Increased static pressure in the subfloor
Moving 95% +/- of available conditioned air delivered to
the heat load
RESULTS
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
21/55
AIR FLOW MANAGEMENT
Under floor air management
Above floor air management (includingabove the false ceiling)
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
22/55
DATA CENTER - AIRFLOW ISSUES Over 60% of cooling energy is wasted in the form of
bypass air
Re-circulated air from IT equipment exhaust canreduce server performance
Air stratification forces set points of precision coolingequipment to be lower than recommended
Air remixing drives down precision cooling efficiencies
Most data centers produce significantly more
cold air then is required by the IT devices
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
23/55
DC COOLING ISSUE DEFINITIONS Bypass Airflow
Air supplied from precision air conditioning units
that bypasses the rack-mounted equipment
Recirculation
Exhaust air from rack-mounted devices that makesits way back to the device inlets
Air Temperature Stratification Layer effect of temperature gradients from the
bottom to the top of the enclosure
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
24/55
BYPASS AIRFLOW A comprehensive study
performed by Upsite
Technologies Inc. concludedthat 60% of the air suppliedin traditional data centers iswasted due to bypassairflow
Significant efficiencies andcost savings can be realizedby eliminating bypassairflow.
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
25/55
RECIRCULATION Hot air exhaust circulating
back into its own intake cancause device thermaloverload.
Typical manufacturer inlettemperature threshold for
device operations is 35C.
Exceeding manufacturersoperating device thresholdcan lead to unplanned
computing system outagesand data loss.
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
26/55
TEMPERATURE STRATIFICATION
Significant gradient of air temperatures beyond ASHRAE TC9.9 placesdevices at risk of thermaloverload
Maintaining inlettemperature gradients
within the ASHRAErecommended rangesignificantly saves energy
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
27/55
TEMPERATURE BOUNDARIES ASHRAE Technical Committee 9.9 has provided a standard as a
statement of reliability
Manufacturers specification is an allowable range within which theproducts can run effectively
ASHRAE TC9.9Manufacturer Specification
Device InletTemperature
Range (64.4 - 80.6)F (50 - 95)F
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
28/55
HCS optimizes the use of existingprecision cooling capacity within thedata center to deliver effective cooling
to IT loads- Balance cool air supply with demand
HCS is a scalable, flexible and provendesign that can be adapted to existinginfrastructures as capacity demandsincrease- No stranded supplemental cooling
assets
HEAT CONTAINMENT SYSTEM
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
29/55
FLEXIBILITY AND HCS CHIMNEY The HCS chimney allows both
active and passive ducting options
Using the passive solution willreduce energy usage when theserver exhaust within theenclosure is substantial.
However, for enclosures withpotential backflow, active hotswappable fans can mediate therisk of thermal overload
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
30/55
BACKFLOW AND STAGNATION Low density racks with passive
chimneys can be susceptible tobackflow
Backflow can lead to hightemperature air leaking throughundesirable regions of theenclosure
Backflow increases pressurebuildup which may lead to server exhaust fan stagnation andultimately damage to computingequipment
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
31/55
MODELED RESULTS - 3D View Traditional
Data Center containing all airflow issues
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
32/55
MODELED RESULTS - 3D View HCS Data
Center optimized results of full containment
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
33/55
COLD AISLE CONTAINMENT
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
34/55
BEFORE & AFTER COLD AISLE CONTAINMENT
27 oc 21 oc
16o
c
27 oc 17 oc
16oc
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
35/55
ARCHITECTURAL VIEWWarm return air
Cold supply air Cold supply air
Warm return air
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
36/55
COLD AISLE CONTAINMENT - Schematic
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
37/55
WHAT IS THE BENEFIT OF CONTAINMENT All energy used to move or produce cool air is used to cool IT equipment
IT equipment will only receive conditioned air, maintaining ideal operating
environment leading to better performance
Air conditioner set points can be tuned to deliver ideal inlet air temp to ITequipment and eliminate the energy from overcooling
Warm air is sent directly back to conditioning systems increasing overallcooling performance
Predictable air management is obtained
Yields a savings greater than 30% reduction in the energyconsumed in a typical legacy datacenter CRAC cooling
system
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
38/55
DATA CENTER FLEXIBILITY VS.
COOLING PERFORMANCE Demand for Flexibi lity
Ability to support any Vendor IT Rack of any depth or height Ability to support any density anywhere on the white space
Available Rack space unencumbered by cooling equipment in the rackrows
Adds, Moves and change happens everyday
Demand for cooling Performance Efficient cooling systems cost less to operate Optimized designs that match cooling capacity and demand cost less to
purchase Predictable performance in a dynamic environment drives reliability
Cooling Performance and Flexibility can co-exist
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
39/55
LEGACY DATACENTER COOLING STYLE
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
40/55
LEGACY DATACENTER COOLING STYLE
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
41/55
SOLUTION: CREATE COLD AND HOT AISLE
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
42/55
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
43/55
COMMON MISTAKES
Enclosure without Blanking PanelEnclosure without BlankingPanel and common outlet for Hot and Cold Air through Top
Mounted Fan
DONT DONT
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
44/55
LEGACY DATACENTER COOLING STYLE
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
45/55
NO INSULATION BETWEEN TWO ENCLOSURES
HOT AIR /COLD AIR MIXESBETWEEN TWOENCLOSURES
INSULATED SIDE PANELS
WRONG WAY RIGHT WAY
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
46/55
IN SUMMARY Focus on air flow for addressing thermal issues in DC
Follow air containment options for enhancingperformance of existing & upcoming DC
Thermal planning using Computational FluidDynamics (CFD) & deploy corrective actions and do aperiodic audit Energy saving proposition
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
47/55
HIGH DENSITY COOLING Water has 3500 times higher density than air
Cooling using water has become common incritical spaces and the traditional beliefs are beingbroken by use of appropriate technology
50%+ energy saving with major space savings withuse of chilled water based cooling for the DC
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
48/55
Front of Enclosure
Rear of Enclosure
How Does It Work? Rear Door Heat Exchanger replaces existing rear door of IT enclosure Rear door has chilled water Supply & Return quick connections at bottom or top Chilled water circulates through fin+tube coil from Supply connection Equipment exhaust air passes through coil and is cooled before re-entering the
room Heat is rejected from room through Return water connection
Tube + FinHeat exchanger
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
49/55
Chilled Water Cooling System Rear door provides 100% sensible cooling No condensation, no need for reheat or humidification Chilled water distributor creates a fully isolated, temperature controlled
Secondary Loop Chilled water source - city water, building chilled water, packaged chiller
~65o
F
~45 oF
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
50/55
50
Thermal Image - Before & After
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
51/55
IN SUMMARY
For higher density racks use chilled water
cooling solution
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
52/55
"You can manage what you canmonitor"
RACK ENVIRONMENT MONITORING
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
53/55
IN SUMMARY What is measured is managed
Monitor environment parameters for correctiveaction & resource optimization
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
54/55
CONCLUSION Proper Air flow management below and above false
floor for effective cooling & hence optimization of energy & resources for low & medium density DCs
Plan thermal management through audit & simulationusing CFD techniques
Use Chilled water based system for high densitycooling requirement in DCs
Monitor environment parameters for corrective action& resource optimization
7/28/2019 APW President BICSI-MAIT Sept09 Rev1
55/55
S.VenkatramanVP Sales
APW President Systems LimitedMob: 09324235923Email: [email protected]