Energy-efficient Ethernet Networks - Calit2 · Energy Efficient Ethernet Networks UCI – Broadcom...

www.broadcom.com

Nicholas IlyadisVice President and Enterprise CTOBroadcom Office of the CTO

Energy Efficient Ethernet Networks

UCI – Broadcom Collaboration

January 12 2010

2

Agenda

Overview of Energy Efficient NetworksMotivation, Savings and End-to-End System Interaction

Energy Efficient Ethernet (802.3az)Standard Objectives, Status and Solutions

EEN – Energy-efficient Ethernet NetworksBroadcom’s End-to-End Philosophy to Energy Savings in Networks

Overview of EEE’s Layer 2 Mechanism and Features

Open Discussion on Collaboration Opportunities

3

Broadcom EEE Vision and Mission

Traditional Networks

Next Generation Networks

Enable end-to-end networks that allow Broadcom’s customers to lead the industry in performance and features in a cost- and

energy-competitive framework

Performance Cost

Energy

4

Industry and Regulatory Trends• Government and Industry Recognition

– April 19, 2006 “Green Grid” formed

– December 20, 2006 House Resolution 5646 signed into law

– European code of conduct

– Japanese government initiative “Top Runner”

• IEEE P802.3az – Energy Efficient Ethernet– Broadcom heavily involved in launching the project in 802.3

• Energy Star– EEE requirements for Servers planned in future draft (2010) once P802.3az is ratified

– EEE requirements for PCs planned in future draft (2010) once P802.3az is ratified

– Historically, EU and other countries will follow suit

– Energy Star has kicked off an enterprise storage elements specification, gaming console etc.

– Specification for SNE (Small Networking Equipment) launched Nov 2009. LNE will be next

• Lower energy usage means lower operating costs

5

Motivation – A Macro View

• All electronics– IT equipment, consumer electronics, telephony

• Residential, commercial, industrial– At least 250 TWh/year– $20 billion/year

• Based on .08$/kWh; rates are rising

– Over 180 million tonsof CO2 per year

• Roughly equivalent to 35 million cars!

• IT equipment about half of this– PCs, displays, printers, servers,

network equipment

One central baseloadpower plant (about 7 TWh/yr)

PCs etc. are digitallynetworked now — Consumer Electronics (CE) will be soon

Numbers represent U.S. only

6

Motivation – A Micro View

Source: Luiz André Barroso, (Google) “The Price of Performance,” ACM Queue, Vol. 2, No. 7, pp. 48-53, September 2005.

(Modified with permission.)

Unrestrained IT power consumption could eclipse hardware costs and put great pressure on affordability, data center infrastructure, and the environment.

7

Motivation – Multiplier Effect of the Micro View

Source: IEEE Spectrum

8

Motivation – A Link Perspective

• High port count triple speed switches

– Linear relationship of power consumption to number of active links

– Aggregate savings attractive in putting inactive links in LPI

• Low port count 10G systems

– Idle power savings on a single link attractive

Typical switch with 24 ports 10/100/1000 Mb/s

Various computer NICs averaged

Results from 1st order (rough) measurements – all incremental AC power

9

Application – Wiring Closet

EEE EnabledEEE EnabledGig SwitchGig Switch

EEE EnabledEEE EnabledClientsClients

Wiring Closet

Wireless LAN

EEEEnabled GigE

Links

EEEEnabled GigE or 10G

Links (for <100Meter Links)

10

Desktop links have low utilization• Snapshot of a typical 100 Mb Ethernet link

– Shows time versus utilization (trace from Portland State Univ.)

0%

20%

40%

60%

80%

100%

0 1000 2000 3000 4000 5000 6000 7000Time (s)

Util

izat

ion

Typical bursty usage(utilization = 1.0 %)

Fig1.xls

11

Why EEE?

• 48 Port Triple Speed Switch (PHY Only Savings)

– Non-EEE Phy Power = 48*440 mW+ = 21.1W– EEE Phy Power = 48*220 mW = 10.6W – Power Savings on idle switch = 10.6W– 8 High Stack power Savings = 84.5W– 20 Wiring Closets (Large Enterprise) = 1690W– Assuming 100% attached Desktops

= additional 1690W– Total Enterprise Power savings = 3379W

• Assume average traffic is 5% …– $0.10/kWhr * 0.95 * 3.379 KW * 24 hrs

= $7.70 per day– Total Savings = $2,812 per year

+BRCM GPHY power/port

Power Savings Case – Wiring Closet Electricity Costs

100%

50%

Without EEE BRCM EEE PHYsavings*

Lowers Power by50%

*Additional power savings possible with BRCM

12

Application – Data Center and TOR

EEE Gphy EnabledEEE Gphy EnabledServer ControllerServer Controller

Data Center Rack

EEEEnabled 1 and 10 GBASET

Links

EEEEnabled GigE or 10G

Links (for <100Meter Links)

http://image.dell.com/image.asp?ImageID=600







13

Ethernet Traffic Profiles• Snapshot of a File Server with 1 Gb Ethernet link

– Shows time versus utilization (trace from LBNL)

utilization <=1.0 %Start time 12:33 PM 2/8/2007 (30 min)

14

Agenda






15

IEEE 802.3 (Ethernet) Working Group Organization802.3 (Ethernet)Working Group

Chair: David J. Law

Vice-Chair: Wael William Diab

IEEE 802.3atMDI Power

Enhancements(PoEP)Chair:

Mike McCormack

IEEE 802.3av10G-EPONTask Force

Chair:Glen Kramer

IEEE 802.3azEnergy EfficientEthernet (EEE)

Task ForceChair:

Mike Bennett

IEEE 802.3baHigh Speed

Ethernet 40G/100G Task Force

Chair:John D’Ambrosia

IEEE 802.3TSSG – Time Sync

Study Group

Chair: Steve Carlson

IEEE 802.3Maintenance

Standing Task ForceMaint. Requests &Current Projects:

IEEE 802.3bb,IEEE 802.3bc

Chair: Wael Diab

IEEE 802.3.1Ethernet MIBs

Task Force

Chair:Howard Frazier

Optics Sub-Task Force

Chair: Pete Anslow

Cu Sub-Task Force

Chair: Chris Diminco

Logic Sub-Task Force

Chair: Mark Gustlin

Extended Reach Parallel MMF

Ad HocCo-Chairs: A. Barbieri

John Petrilla

802.3 RulesOwner:

Wael Diab(Vice-Chair)

Layer 2 Ad-Hoc

Chair: Wael Diab

Wake ShrinkageAd-Hoc

Chair: David Law

Layer 2Ad-HocChair:

Wael Diab

Transformerand Channel

Ad-HocChair:

Fred Schindler

16

EEE Objectives• Define a mechanism to reduce power consumption during

periods of low link utilization for the following PHYs– 100BASE-TX (Full Duplex)– 1000BASE-T (Full Duplex)– 10GBASE-T– 10GBASE-KR– 10GBASE-KX4– 1000BASE-KX

• Define a protocol to coordinate transitions to or from a lower level of power consumption

• The link status should not change as a result of the transition• No frames in transit shall be dropped or corrupted during the

transition to and from the lower level of power consumption• The transition time to and from the lower level of power

consumption should be transparent to upper layer protocols and applications

17

EEE Objectives

• Define a 10 megabit PHY with a reduced transmit amplitude requirement such that it shall be fully interoperable with legacy 10BASE-T PHYs over 100 m of Class D (Category 5) or better cabling to enable reduced power implementations

• Any new twisted-pair and/or backplane PHY for EEE shall include legacy compatible auto negotiation

18

Low Power Idle Overview

Low-PowerActive Active

Deassert LPI

Hold

TrTs

Quiet Quiet Quiet

Tq

Wait a minimum of Tw_Sys before sending data (Tw_sys >= Tw_PHY)

Sleep Data / IdleData / Idle Alert Wake IdleRefreshRefresh

Assert LPI

• Low Power Idle (LPI) – PHY powers down during idle periods• During power-down, maintain coefficients and synchronization

to allow rapid return to Active state• Wake times for the respective twisted-pair PHYs

100BASE-TX: Tw_PHY <= 30 usec1000BASE-T: Tw_PHY <= 16.5 usec10GBASE-T: Tw_PHY < ~8 usec (2 modes)

Tw_PHYTw_sys

19

IEEE P802.3az Progress and Timeline

Draft Progress Successful initial Working Group Ballot & Recirc (D2.0 & D2.1)

All comments against D2.0 and D2.1 consideredD2.1 published October 19th, Recirc closed November 4th

D2.2 published December 24th

Recirc closes January 13th

LegendIEEE 802 PlenaryIEEE 802 Interim IEEE-SA Standards Board

Task Force

Sep2007

Jul2007

Nov2007

May2008

Mar2008

Jul2008

Nov2008

May2009

Nov2009

Mar2010

Sep2009

Sep2008

Mar2009

Jan2008

Jan2009

Jul2009

Jan2010

May2010

Jul2010

Sep2010

PAR

TF Review WG Ballot

You arehere

SponsorBallot

Std!

D3.0D2.0D1.0

Last newproposal

Lastfeature

Lasttechnicalchange

Baseline

20

Agenda






21

Framework• Physical layer

– Rate established at link up– Rate constrains operation of physical layer and upper layers– IEEE P802.3az standard, EEE, allows for suspending the energy on the wire by

turning IDLES to Low Power Idles• Allows energy savings on the PHY layer• Enables further savings up the stack

• MAC Si and Above– Offers a large opportunity for additional power savings– Includes additional interfaces and subsystems– Implementations vary by network position and node architecture– Additional savings enabled by

• Broadcom architectures that leverage PHY EEE• Optimized control policies• 802.3az defines a layer 2 mechanism that allows for additional power savings

22

ControlPolicy

Broadcom’s EEN: End-to-End SavingsEND POINT SWITCH

PHY

MAC

Controller

Controller SW

OS

PHY

MAC

NWSW

NWSW SW

Application

OS

802.3azEEE

ControlPolicy

Coordinatedand Optimized

End-to-End Control Policy

Well Integrated Control Policy and EEE

Within a Box

EEN

23

Broadcom’s EEN: End-to-End SavingsEND POINT SWITCH

PHY

MAC

Controller

Controller SW

OS

PHY

MAC

NWSW

NWSW SW

Application

OSBroadcom Customizable and Optimized Control Policy

• Maximize energy efficiency by maximizing operation in saving states• Minimize performance, latency impact by avoiding unnecessary transitions• Customizable via Software

Broadcom Enabled

Broadcom Enabled

24

Agenda






25

Layer 1 Support Layer 2 Support

SwitchController

EEE Enhanced Layer 2 Operations

CPU

1000BASE-T MDI

Layer 2Control Policy

• Opportunity to save additional power within a box (link partner)– Additional circuits beyond the PHY can be turned off

• Additional RX wakeup time negotiated using 802.3az’s Layer 2 — Standards based

PCI-e

Server with Controller NIC Switch

26

Review: LLDP Overview

• Operates over a point to point link• Completely enclosed protocol

– We define data, it gets transported• We don't get to make changes to the protocol

• Data in ‘Local MIB’ transported to ‘Remote MIB’– Transported by TLVs (type, length, value)

Local MIB

Remote MIB

Local MIB

Remote MIBA change in a ‘Local MIB’ attribute appears in the corresponding ‘Remote MIB’ attribute

System 1 System 2

Source: diab_02_0109.pdf

27

• Officially called “Data Link Layer” – Effort Chaired by Broadcom in standards• Several Components: (a) Transport (b) State machine (c) MIB (d) Features• In EEE allows for the dynamic negotiation of wake up time for RX

– Allows for cascaded interfaces– Deeper sleep modes– Fallback– Efficient buffering

• In PoEP allows for the dynamic negotiation of power allocation for PD – Power budgeted when it is needed, supplies operate at near optimal efficiency point, power backup

systems and batteries optimized for load (e.g. UPS) – Smart allocation: Features scalability with power budget

• Nearly identical SMs for PoEP and EEE

802.3at’s and 802.3az’s Layer 2

dot3at localsystem MIB

dot3at remotesystem MIB

aRemAbc

aLocDef

State 1

State 2

abc = True

def <= True

def <= False

Chassis IDTLV

Port IDTLV

Time To Live TLV

End Of LLDPDUTLV

dot3atTLV

OptionalTLV

..LLDPDU Format

OptionalTLV

Source: joint_diab_1_0708.pdf

28

Energy Efficient Ethernet TLV

TLV Type = 127

TLV informationString length = 14

802.3 OUI00-12-0F

802.3 Subtype =

TBD

TX Tw_sys

RX Tw_sys

FallbackTw_sys

Echo TX

Tw_sys

Echo RX

Tw_sys

TLV header TLV information string

TLV information string (continued)

7 bits 9 bits 3 octets 1 octet

2 octets 2 octets 2 octets 2 octets 2 octet

...

...

29

Fallback State: Overview

• 2nd “fall back” preference of RX system when requesting Tw_Sys from TX partner

• RX systems have discrete states that TX partner is unaware of.

– TX may have dynamic buffering scheme

• TX can take this information into account when making an allocation decision

• Efficiencies / benefits can be realised • Similar concept exists in P802.3at

30

Agenda






31

Leading Industry with Next Gen Power Initiative

100%

50%

Without EEE With BRCM EEEPHY

Server Switch

PCStorage

Lowers Powerby

50%

www.broadcom.com

Thank you

Energy-efficient Ethernet Networks - Calit2 · Energy Efficient Ethernet Networks UCI – Broadcom...

Documents

Transcript of Energy-efficient Ethernet Networks - Calit2 · Energy Efficient Ethernet Networks UCI – Broadcom...