How Can Downtime Ruin Your Budget

Carrie Higbie, The Siemon Company

Small Catastrophe DowntimeHardware failurePower outages Infrastructure failuresNoiseBandwidth issuesPoor convergence planning#$%^&* HackersVirus attacksBubba stuffUser error

BIG Catastrophe DowntimeFire@#$%^ HackersVirus attacksPower outages with inadequate UPS/generatorsCut WAN links or WAN failuresFloodNatural disasterFunding cutsSabotageUser error

Biggest Keys to ReliabilityPlanning

Redundant sites

Geographically dispersed

TEST

Did I say TEST?

Proper Provisioning

Today’s latest buzzword – Compartmentalization

Look back to look forward

Think outside the box – check outside the box

What does downtime cost?

Fortune 1000 Published DataIndustry Rev $ Millions Revenue Per

Hour (2080)

Motor Vehicles and Parts 523,222 251,548,798.10

General Merchandisers 471,419 226,643,798.10

Petroleum Refining 432,627 207,993,605.80

Commercial Banks 414,970 199,504,711.50

Specialty Retailers 352,989 169,706,442.30

Diversified Financials 315,214 151,545,288.50

Telecommunications 309,455 148,776,250.00

Insurance: P&C Stock 298,206 143,368,076.90

Utilities: Gas and Electric 277,869 133,590,865.40

Health Care 261,366 125,656,682.70

Annual revenue / 2080 hour per year = revenue assumptions per hour

Employee Costs (National Avg. Wage $33,252.09 *1.4 / 2080 = weighted hour rate)

# of Employees * Weighted average Wage / 2080 Hours per year

Average hourly wage based on figures supplied by the National Bureau of Labor Statistics

# EmployeesHourly Salary

Cost 15% down/one hour INDUSTRY

$ 2,915,457 $ 65,251,474 $ 9,787,721 General Merchandisers

$ 1,945,251 $ 43,537,056 $ 6,530,563 Specialty Retailers

$ 1,706,609 $ 38,195,968 $ 5,729,398 Motor Vehicles and Parts

$ 1,293,584 $ 28,951,983 $ 4,342,798 Commercial Banks

$ 1,242,655 $ 27,812,130 $ 4,171,820 Food and Drug Stores

$ 1,176,568 $ 26,333,023 $ 3,949,953 Food Services

$ 997,801 $ 22,332,000 $ 3,349,800 Health Care

$ 976,678 $ 21,859,242 $ 3,278,886 Telecommunications

$ 960,200 $ 21,490,444 $ 3,223,567 Aerospace and Defense

$ 721,848 $ 16,155,836 $ 2,423,375 Food Consumer Products

Revenue Per Hour Per Employee

Annual Rev / 2080 / Number of Employees = Revenue Per hour Per Employee

Rev/hour # EmployeesRev / hour

/emp INDUSTRY

$ 49,845,865.4 50,489 $ 987 Insurance: Life, Health (mutual)

$ 11,484,855.8 12,528 $ 917 Pipelines

$ 207,993,605.8

287,698 $ 723 Petroleum Refining

$ 76,945,528.8

107,612 $ 715 Wholesalers: Health Care

$ 48,083,653.8

116,666 $ 412 Energy

$ 41,628,557.7

114,474 $ 364 Wholesalers: Electronics/Office Equipment

$ 26,844,567.3 80,836 $ 332 Homebuilders

$ 9,290,288.5 28,751 $ 323 Real estate

$ 66,264,038.5

211,534 $ 313 Insurance: Life, Health (stock)

$ 20,634,903.8 67,105 $ 308 Mining, Crude-oil production

Salary and Revenues Combined 15% of Workforce Down for One Hour

Salary + Rev - 15% Down INDUSTRY

$ 43,784,291 General Merchandisers

$ 43,461,717 Motor vehicles and Parts

$ 34,268,504 Commercial Banks

$ 32,164,896 Petroleum Refining

$ 31,986,529 Specialty Retailers

$ 25,595,324 Telecommunications

$ 24,800,090 Diversified Financials

$ 23,493,798 Insurance: P&C (stock)

$ 22,198,302 Health Care

$ 21,711,759 Utilities: Gas and Electric

Average hourly wage based on figures supplied by the National Bureau of Labor Statistics

FormulasRevenue per hour • Total revenue / 2080 hour work week

Revenue per employee per hour• Total Revenue / Number of Employees / 2080

Salary expense per hour (weighted)• Average hourly wage * 1.4 (to include overhead) /

2080

Salary expense plus lost revenue• Total revenue per hour + weighted salary expense

per hour * % of workforce down at any given time (we used 15%)

How Much Uptime?

UP Hours Down

90% 876 hours

95% 438 hours

98% 175 hours 40 minutes

99% 87 hours 36 minutes

99.9% 8 hours 45 minutes

99.99 52 minutes

99.999 33.6 seconds

Downtime Costs Per Component

Combined -15% Industry Hardware - 25% Software - 25% Cabling - 21% Other

$43,784,291 General Merchandisers $ 10,946,072.75 $ 10,946,072.75 $ 9,194,701.11 $ 12,697,444.39

$43,461,717 Motor vehicles &Parts $ 10,865,429.25 $ 10,865,429.25 $ 9,126,960.57 $ 12,603,897.93

$34,268,504 Commercial Banks $ 8,567,126.00 $ 8,567,126.00 $ 7,196,385.84 $ 9,937,866.16

$32,164,896 Petroleum Refining $ 8,041,224.00 $ 8,041,224.00 $ 6,754,628.16 $ 9,327,819.84

$31,986,529 Specialty Retailers $ 7,996,632.25 $ 7,996,632.25 $ 6,717,171.09 $ 9,276,093.41

$25,595,324 Telecommunications $ 6,398,831.00 $ 6,398,831.00 $ 5,375,018.04 $ 7,422,643.96

$24,800,090 Diversified Financials $ 6,200,022.50 $ 6,200,022.50 $ 5,208,018.90 $ 7,192,026.10

$23,493,798 Insurance: P&C $ 5,873,449.50 $ 5,873,449.50 $ 4,933,697.58 $ 6,813,201.42

$22,198,302 Health Care $ 5,549,575.50 $ 5,549,575.50 $ 4,661,643.42 $ 6,437,507.58

$21,711,759 Gas and Electric $ 5,427,939.75 $ 5,427,939.75 $ 4,559,469.39 $ 6,296,410.11

Cost Over Life of SystemCombined Industry

3 Years Hardware 5 Years Software 10 Years Cabling

$ 43,784,291.00 General Merchandisers $ 32,838,218.25 $ 54,730,363.75 $ 91,947,011.10

$ 43,461,717.00 Motor Vehicles/ Parts $ 32,596,287.75 $ 54,327,146.25 $ 91,269,605.70

$ 34,268,504.00 Commercial Banks $ 25,701,378.00 $ 42,835,630.00 $ 71,963,858.40

$ 32,164,896.00 Petroleum Refining $ 24,123,672.00 $ 40,206,120.00 $ 67,546,281.60

$ 31,986,529.00 Specialty Retailers $ 23,989,896.75 $ 39,983,161.25 $ 67,171,710.90

$ 25,595,324.00 Tele-communications $ 19,196,493.00 $ 31,994,155.00 $ 53,750,180.40

$ 24,800,090.00 Diversified Financials $ 18,600,067.50 $ 31,000,112.50 $ 52,080,189.00

$ 23,493,798.00 Insurance: P&C (stock) $ 17,620,348.50 $ 29,367,247.50 $ 49,336,975.80

$ 22,198,302.00 Health Care $ 16,648,726.50 $ 27,747,877.50 $ 46,616,434.20

$ 21,711,759.00 Utilities $ 16,283,819.25 $ 27,139,698.75 $ 45,594,693.90

System Lifetime Graph

$-

$10,000,000.00

$20,000,000.00

$30,000,000.00

$40,000,000.00

$50,000,000.00

$60,000,000.00

$70,000,000.00

$80,000,000.00

$90,000,000.00

$100,000,000.00

Ge

ne

ral M

erc

h.

Mo

tor

veh

icle

s/P

art

s

Co

mm

erc

ial B

an

ks

Pe

tro

leu

m R

efin

ing

Sp

eci

alty

Re

taile

rs

Te

leco

mm

un

ica

tion

s

Div

ers

ifie

dF

ina

nci

als

Insu

ran

ce:

P&

C

He

alth

Ca

re

Util

itie

s:

3 Years Hardware

5 Years Software

10 Years Cabling

The Cost of a Slow Network – Partially Down

Examples:

Company A:Number of Employees: 500Average Hourly Wage: $15.00Hours of Productivity Lost per Year: 30Network Slow Cost = $225,000.00

Company B:Number of Employees: 1,000Average Hourly Wage: $18.00Hours of Productivity Lost per Year: 52Network Slow Cost = $936,000.00

Company C:Number of Employees: 5,000Average Hourly Wage: $20.00Hours of Productivity Lost per Year: 20Network Slow Cost = $2,000,000.00

Calculate network slow cost:

Cost = P x W x E

P = Total Number of hours lost Productivity per year (weekly minutes/60 x 52)

W = Average hourly Wage E = Number of Employees

on the network

Environmental conditions• Temperature and humidity variations• EM and RF interference

High network traffic

Outdated, slow PCs, NICs

Poor installation

Inferior network cabling

What causes slow response?

Other CostsRecuperation

Verification

Overtime

Redesign

Re-qualification

Second guessing and justifications

Rework of disaster recovery plan

Rework of business continuity plan

Mind altering medications

The Cost of Data lossPCs in Use 72.0 million

PCs Experiencing Data Loss

Hardware Failure $1,921,300

Human Error $1,397,300

Software Corruption $611,300

Computer Viruses $305,700

Theft $234,400

Total $4,601,000

Avg. Cost of Each Data Loss Incident

Technical Services $380.00

Lost Productivity $177.00

Value of the lost data $2,000

Total $2,557

Total US Data Loss Costs 1998 $11.8 billionAn incident of data loss is counted if it is significant enough to warrant a recovery effort. Estimates made from “1998 Safeware Loss Study,” Safeware, The Insurance Agency Inc., http://www.safeware.com, 1999 and “Understanding Data Loss,” ONTRACK Data International, Inc., http://www.ontrack.com, 1999.

Hidden Costs Freight

Package insurance

Costs to undo workarounds

Costs to re-route cables / links

Heavy equipment

Overtime for security personnel

Payment for personnel tracking downtime and creating downtime reports

Fees/penalties

Lost return business

Employee sabotage

Notification procedures and costs

Infonetics 2003 – Greatest Threats

Network products

Security products

Cables and connectors

Servers

Applications

WAN and Internet connectivity

E-commerce

How to Prevent DowntimePreventative maintenancewww.cert.org - security subscriptionsPlanningTESTINGLogging activitiesSecurity measuresContingency and business continuity planningDocumentationREVISIONSPay attention to codes

Example 1 – Catastrophic Failure

FIRE

The Tip of the Iceberg Obvious Costs of Fires

Cost of downtime Data loss Equipment loss

Hourly Cost of Downtime and Lost DataEnergy $2.8 milTelecommunications $2.0 milManufacturing $1.6 milFinancial Institution $1.4 milInformation Technology $1.3 milInsurance $1.2 milRetail $1.1 milPharmaceuticals $1.0 milBanking $1.0 mil

Source: IT Performance Engineering & Measurement Strategies, Meta Group, 2000

Most Companies are Able to See the Obvious Costs of a Fire…

Source: IT Performance Engineering & Measurement Strategies: Quantifying Performance Loss, Meta Group, October 2000.

The Bulk of the IcebergHidden Costs of Fires

Productivity/efficiency Loss

Employee absenteeismClean upMedical costsDamage to customer relationships

Loss of competitive advantage

LitigationDamage to public imageRegulatory requirements

Contractual obligations

…however, the hidden costs of fires that are not seen can be significant.

According to

Each day, 323 non-residential structure fires will occur . . . .

Every 5 days, 1 person will die

Each day, 4 people will be injured

Each day, $7.4 million in property damage occurs

Fire Loss in the US during 2002, Michael Kartner, Jr., NFPA Fire Analysis and Research Division, August 2003

NFPA codes and standards represent a set of minimum fire safety requirements for the protection of buildings.

NFPA 90A is responsible for plenum spaces in buildings.

NFPA 90A - Standard for Air Conditioning and Ventilation Equipment:

• Sets requirements for flame, smoke and fuel load

• 4.3.10.2.6 -- “All materials exposed to the airflow shall be non-combustible or limited combustible and have a maximum smoke developed index of 50…”

• Combustible cables allowed as exception (CMP, etc.)

• Requires listing of limited combustible cable

NFPA 70 - National Electrical Code (NEC):

• Does not currently include LCC

• Should correlate with NFPA 90A requirements

• Article 645 permits type CM cable under raised floors in computer rooms under certain conditions

• Requires removal of abandoned cable

NPFA 70 is responsible for plenum cable products and applications.

NFPA 13 Standard for the Installation of Sprinkler Systems

• In sprinklered buildings, use of combustible cables in concealed spaces, including plenums, requires installation of sprinklers in these spaces.

• Use of limited combustible cable does not require sprinklers in these spaces.

• Options: 1. Sprinklers in concealed space

2. Cable in conduit

3. LCC cable (cost effective)

NPFA 13 sets the requirements for sprinklered buildings.

However, in the years that followed, many high profile cable fires still occurred.

Alexis Nihon Plaza - Montreal, ON

“The fire spread horizontally on the tenth floor and traveled through non-fire stopped penetrations for communications wiring …”

October 1986

Fire spread rapidly to 6 floors11 fatalities occurred due to smoke inhalation

“… heat and smoke escaped through the communications wiring openings that had not been fire stopped.”

-- Fire Journal, January/February 1988

However, in the years that followed, many high profile cable fires still occurred. (Cont’d.)

Bell Central Office - Hinsdale, IL

“Fueled by the insulation, the fire quickly spread into the groups of cables in the cable tray and eventually emerged at the top of the cables. The fire was able to travel horizontally both in the confined spaces between cables in the trays and in an open space between the top layer of cables and the ceiling.”

-- NFPA Fire Investigation

30’ by 40’ Fire area$90 Million in damages

17 Day outageCalls disrupted for 4

weeks

May 1988“The smoke and combustion by-products from burning cable insulation penetrated the entire building . . . . irreparably contaminating much of the communications equipment.”

-- From report of Illinois Office of the State Fire Marshall and Illinois Commerce Commission

October 1996

Rockefeller Center - New York, NY

“The fire, reported at 3:59 AM, jumped helter-skelter from floorto floor following the snaking utility products, feeding oninsulation and thick cable-housing and breaking out into office hereand there, fire officials said.”

-- New York Times

However, in the years that followed, many high profile cable fires still occurred. (Cont’d.)

However, in the years that followed, many high profile cable fires still occurred. (Cont’d.)

United Airlines Operations Control Center - Elk Grove, IL

August 1999

Years of investigation and research has yielded key lessons learned about the link between fires and cabling. . . .

Ignition source is often electrical failure

Cables can add significant fuel load and spread fire rapidly

Mechanism of fire spread follows cable pathways and spaces

Small fires can create extensive smoke and non-thermal damage

Passive fire protection essentialNew cabling technologies can significantly

reduce vulnerability to fire

Fire Investigations: Lessons Learned

. . . . and has highlighted the leading source of fatalities and damage in a fire -- SMOKE.

Low visibility in an evacuation

Dangerous carbon monoxide

Smoke permeation results in:• Fine particulate carbon and moisture• Current leakage (shorts) which damage or

destroy sensitive electronic equipment• Data loss• Early component failure

The Effects of Smoke

“95% of the fire damage . . . . is attributed to the smoke products and only 5% is caused by the thermal effects of fire.”

-- Network Reliability: A Report to the Nation, FCC 1993

Recuperation CostsHardware replacement

Notification costs (internal and external)

Data restoration

Hot site utilization fees

Overtime

Emergency maintenance call fees

Replacement of onsite mirrored components

Smoke damage

ETC.

What’s in a Contingency Plan?

Contingency PlanningDetection and reaction• Identification of the problem and notification of

authorities Emergency services Environmental agencies Physical security

• Reduction of exposure HVAC failures Fire alarms procedures Electrical failure procedures Flood and water damage

• Evacuation

• Emergency management team notification

• Flow of procedures

Back-up Site Procedures

Establish control center

Begin disaster recovery

LOG EVERYTHING

Timed events• Set up things to happen within time blocks

Communications

Verify all connectivity

Disaster recovery checklists

Maintain the FollowingUp to date IS organizational charts with contact info

Disaster planning coordinator

Emergency management team

Operations team• Computer operations

• Facilities and replacements

• Cold site preparation

• Misc. support equipment and supplies

Data entry team• Data input and control

Maintain

Special projects team

Technical support personnel

Data restoration integrity and verification team

Programming assurance team

Insurance team

Internal audit team

Requirements Should be Updated QuarterlyCommunications links

Cable pathways, labeling and terminations

Power and HVAC requirements

Computer equipment, vendors, and serial numbers

Teleprocessing systems and requirements

Terminal configuration charts

Supplier Lists

Offsite paper copies should be maintained

Hardware

Software

Additional systems

Custom form suppliers

Utility vendors

Prioritize Applications

Assign risk assessment number from 1-5

Assign responsibilities for each

Update data dictionaries

Update software revision levels

Custom programming should be updated

Source code should be backed up each time there is a revision

Don’t forget gateways and other applications that use any one application

Offsite StorageVirtualization

Live redundant offsite storage

Cold-site

Hot-site

Personnel backup files

Custom forms at offsite

Other CostsRecuperation

Verification

Overtime

Redesign

Re-qualification

Second Guessing and Justifications

Rework of Disaster Recovery Plan

Rework of Business Continuity Plan

Mind altering medications

Hidden Costs Freight

Package Insurance

Costs to undo workarounds

Costs to retroute cables / links

Heavy equipment

Overtime for Security personnel

Payment for personnel tracking downtime and creating downtime reports

Fees/penalties

Lost Return Business

Employee Sabotage

Notification Procedures and costs

Document all Procedures

IPL or power up

Power down

Schedules

Operations run procedures

Disk drive and file layouts

Security databases

Listing of all purchased software and serial numbers

Physical Security and Access ControlStaff

Service personnel

Consultants and outside personnel

Access control

Vault access

Non office hours and after hours contacts

Security personnel

Office security

Backup Facilities

Layout

Hardware

Software

Communications

Contracts

Testing

Mock disasters

Program compilations and complications

Most Important

TEST

RETEST

REVISIT

UPDATE

AUDIT

Practice your notification procedures

TIA 942 – Data Center StandardSets up “hot zones” for equipment

All horizontal cables should be run and terminated accommodating growth so that it does not have to be revisited

Fire, Life, Safety, Power and Lighting considerations

Distribution areas and Telecommunications Rooms

Equipment Placement

Cabling Systems, Cabling Pathways and spaces

Security and other included systems

TR 942 Design Considerations

TIA 942 – Data Center Standard

Redundancy• N – Base requirement• N+1 Redundancy• N+2 Redundancy• 2N• 2(N+1)

Tiers• Tier 1 – Basic data center• Tier 2 – Redundant components• Tier 3 – Concurrently maintainable• Tier 4 – Fault tolerant

Tiers – What do they Mean?TIER 1 TIER II TIER III TIER IV

Delivery paths 1 1 1 Active/1 Passive

2 Active

Redundant components N N+1 N+1 2(N+1) or S+S

Support space to floor ratio 20% 30% 80-90% 100%

Ultimate watts/ft2 20-30 40-50 100-150 150+

First year deployed 1965 1970 1985 1995

Annual IT downtime due to Site

28.8 hrs 22.0 hrs 1.6 hrs 0.4 hrs

Site availability 99.671% 99.749% 99.982% 99.995%

Power support UPS UPS + Gen UPS + Gen UPS + Gen

Critical path support requires

Shutdown Shutdown Auto Auto

Redundant components Maybe None

Systems Systems and Power some others

All

Cost per square foot $450 $600 $900 $1,100+

Based on information from the Uptime Institute

10 Point InspectionAdherence to StandardsCloset Clean-upDocumentationSNMP or other monitoring and testingIdentification of weak linksRe-certify questionable links LabelingSpeed reports IP Address listings Monitor reports (bandwidth and throughput)Replace home-made patch cables

Top Offenders

•Improperly terminated cables•Improperly terminated patch cords•Lengths exceeded specified maximum•Cabling was improperly or not labeled (troubleshooting problems)•Cables run over fluorescent lighting causing interference•Electronics and closets in poor locations (humidity, EM, RF)•Cables bunched to tightly causing the pairs to be flattened•Cables tied to electrical conduits or run too close to Power panels•Cabling that did not pass testing due to various issues•Closet spaghetti•CAT3 cables terminated to CAT 5 – 100M switches•Bent fiber exceeding bend radius•Cables not to spec•Racks not grounded

Top Protocol Issues

• Extra protocols/all protocols being activated on the workstations

• Ports forced to 10Mb half duplex no longer needed

• Packet over-runs• Unknown protocols• Retransmissions• Routing loops or VLAN errors• Mystery devices or devices that should not be

there• High bit error rates

Data Center Solutions

Blade patch

Grid ready cabling• XGLO™• TERA™• 10G6™

Wire management

Specialty trunk cable assemblies

IP video and CCTP products

Don’t be DENSE!

Shared Media ApplicationsWiFi/WLAN • Shared channel to switch• Supports 10-20 users per access point

Network Jack• Provides additional connectivity where another cable

can’t be run• Shared channel from switch to switch• Adds some management features

VoIP or IP Telephony• Some implementations use switch inside phone• Shared media to network switch• HIGH bandwidth demands due to quality needed for

voice

Powered (PoE) Applications (IEEE 802.3af)

Power provided over cabling channel

Can be mid-span (injected power within channel) or end-span (in switch)

Gigabit / end span applications allow for power to transmit on data pairs

10/100 mid span allow power to be provided on non-data pairs

500mA absolute limit, peak allowable current 450mA with a 50mA safety margin (guardbands)

Port voltage of 44V – Maximum 48V

Most resistive allowable cable (20 ohms round trip)

Cable drops an additional 7V when maximum current is flowing and arrives at powered equipment as 37V

37V*350mA = 12.95W (maximum power to a powered device (PD)

PSE (Power Source Equipment) must detect if attached device is standard or powered and drop power if not needed

802.3af (continued)

Endspan PSE (at Switch) can operate at 10/100/1000 alternative A or B or Both

Midspan 10/100 only – alternative B Only

Conductor Alt. A (MDI-X) Alt. A (MDI) Alt. B (All)

1 - V port / Data + V port / Data Data

2 - V port / Data + V port / Data Data

3 + V port / Data - V port / Data Data

4 + V port

5 + V port

6 + V port / Data - V port / Data Data

7 - V port

8 - V port

WiFi (Wireless Fidelity) WLAN (Wireless LANs)

802.11a 802.11b 802.11g HiperLAN/1 HiperLAN/2 Wi-MAX (802.16)

Still in development

54Mb/s 11Mb/s* 6,12,24 or 54Mb/s*

20Mb/s Up to 54Mb/s

Up to 70Mb/s

5-6GHz 2.4GHz 2.4GHz 5GHz RF 5GHz RF 2-11GHz or 10-66GHz

Ultrawide broadband (802.15.3a) 480Mb/s up to 30’ (still in development)

10GBT Objectives Support operation over 4-connector structured 4-pair,

twisted-pair copper cabling for all supported distances and classes

Define a single 10 Gb/s PHY that would support links of:– At least 100 m on four-pair Class F balanced copper cabling

– At least 55 m to 100 m on four-pair Class E balanced copper cabling

Support a BER of 10-12 on all supported distances and Classes

The equipment technology is there. Is the cabling?

Other 10G Copper Standard

10GBASE-CX4• Infiniband connector

• 10G over Twinax to 15m

Already doubled with TERA™• Same chip with no modifications

• Second generation chips will go full 100meters

10Gig Ethernet Over Fiber

IEEE 802.3ae published June 2002

www.10gea.org

Multiple interfaces designed to support MM & SM media

Allows limited use of existing TIA/ISO based MMF

New 50/125µm MMF created for 10GbE

Estimated 10x cost increase over 1GbE

No auto-negotiation

New application for legacy MMF in process

Optical GbE Distances

ApplicationWavelengt

h62.5

160/500

62.5

200/500

50

500/500

50

2000/500SMF

100BASE-SX 850nm 300m 300m 300m 300m -

1000BASE-SX

850nm 220m 275m 550m 550m -

1000BASE-LX

1300nm 550m 550m 550m 550m 5km

10GBASE-SX 850nm 28m 35m 86m 300m -

10GBASE-LX 1310nm - - - - 10km

10GBASE-EX 1550nm - - - - 40km

10GBASE-LX4

1310nm 300m 300m 300m 300m 10km

Considerations

Gartner says…..• 20% of all IT purchases are for things that

DON’T work

• Most companies UNDERestimate labor by 50%