NETWORKING SOLUTIONS FOR A SERVER VIRTUALIZATION ENVIRONMENT

APRICOT 2011

Russell Cooper

russ@juniper.net

2

WHAT YOU WILL GET FROM THIS SESSION

1. Talk: about challenges Server Virtualization technologies brings for the data center networks.

2. Demonstrate: standards based approach, where available, to improve the experience and economics in a virtualized environment.

3

AGENDA

1. Market Drivers

2. Limitations of legacy network

3. Solutions Simplification Infrastructure Enhanced services

4. Summary

4

THE EVOLUTION OF SERVER VIRTUALIZATION

PHASE 1 PAST

Server Consolidation

Guiding Principle: Improve utilization of physical resources

Driver: Power and space Improvements in server utilization Savings

Network had no role

PHASE 2 FUTURE

Business Agility

Guiding Principle: : Improve utilization of a pool of resources

Driver: Adapt quickly to new demands Heightened compliance & security Better disaster management Cloud Based Computing Models

Network has a huge role

5

LEGACY NETWORKS RESTRICT AGILITY

VM2 VM3

SERVER 1

NIC

VM2 VM3VM1

SERVER 2

NIC

VM1

COMPLEX:Too Many Devices

to ManageAdditional virtual

switches

INFRASTRUCTURE: LACK OF ADDITIONAL SERVICES:

POOR PERFORMANCEMultiple layersAcross North-South path

PROPRIETARY:Pre-standard protocols

MOBILITY:North-south pathScale & scope of L2 adjacenciesAcross sites

SECURITY:Silo’ed , unavailable across domains Intra-VM traffic

MANAGEABILITY:Orchestration between the physical and virtual network

6

NETWORK SIMPLIFICATION FOR SUPPORTING SERVER VIRTUALIZATION

VM2 VM3

SERVER 1

NIC

VM2 VM3VM1

SERVER 2

NIC

VM1

INFRASTRUCTURE: LACK OF ADDITIONAL SERVICES:

POOR PERFORMANCEMultiple layersAcross North-South path

PROPRIETARY:Pre-standard protocolsInteroperability Lock-in

MOBILITY:North-south pathScale & scope of L2 adjacenciesAcross sites

SECURITY:Silo’ed , unavailable across domains Intra-VM traffic

MANAGEABILITY:Orchestration between the physical and virtual network

HIGH PERFORMANCE

INFRASTRUCTURE THAT IS:

OPEN, STANDARDS

BASED

MOBILITY

MANAGEABILITY

SECURITY

ENHANCED SERVICES NEEDED

COMPLEX:Too Many Devices

to ManageAdditional virtual

switchesSIMPLIFICATION

7

BEFORE AFTER

Fewer devices to manage: 44 -> 4

SIMPLIFICATION

NETWORK DEVICE CLUSTERING

8

TECHNOLOGY APPROACHES

Facts Simplify operations Behaves as a single node

both at L2 & L3 layers so it inherits all benefits found in L2 Table Synch approach

Control Plane Unification

Facts Distributed link

aggregation (LAG) plus some L2/L3 protocols enhancements to minimize interchassis link load

L2 Table Synch

Multiple Devices – One Control PlaneMultiple Devices – Enhanced

Protocols

9

INFRASTRUCTURE THAT IS:

OPEN STANDARDS BASED

SIMPLIFICATION

HIGH PERFORMANCE MOBILITY

MANAGEABILITY

SECURITY

ENHANCED SERVICES NEEDED

OPEN, STANDARDS

BASED

10

VM2VM1

NIC

VM3VM2VM1

NIC

VM3VM2VM1

NIC

COMMUNICATION BETWEEN THE VIRTUAL MACHINES

1. In the hypervisor vendor’s switch(e.g.VM Ware vSwitch)

2. In the NIC 3. In the existing external physical switch (VEPA)

VM3

11

COMPARING VEPA AND VEB

VM2VM1

NIC

VM3 VM2VM1

NIC

VM3

Virtual Ethernet Port Aggregator (VEPA)

North – South optimizedFull functioned hardware

switch

Virtual Ethernet Bridge (VEB)

East – West optimizedLimited function software

switch

Hypervisor/softwareswitch

Physical switch

Network servicesin hardware

Network servicesin software

12

COMPARISON OF OPTIONS

1 2 3

Switching done in Software Hardware Hardware

Customer’s Time to adopt solution

Low – comes in- built with hypervisor

UnknownLow - simple

software upgrade

Latency for switching Very LowVery Low

Low

vSwitch NIC VEPA

Industry support (standards based)

NA Unknown Yes

Virtual switching managed by

Server admin UnknownNetwork Admin

Customers’ Cost to adopt

Low – comes with hypervisor

UnknownFree - software

upgrade

Compatibility with any existing network

Yes Unknown Yes

Feature Richness Very Low Low High

13

VEPA

Virtual Ethernet Port Aggregator Uses external physical network for intra-

server VM to VM communication It’s an evolving open standard IEEE

802.1Qbg / 802.1Qbh Supported by almost all the major IT

vendors For more information

http://www.ieee802.org/1/files/public/docs2009/new-bg-thaler-par-1109.pdf http://www.ieee802.org/1/pages/802.1bg.html

VEPA brings the evolved Ethernet functionality to virtual networking

VM2VM1

NIC

VM3

14

TOP 3 BENEFITS OF VEPA

Features & Scale

Switching where it belongs – on the switches

Elegant

VEPA is a non-disruptive and cost-effective

Open

Server and hypervisor agnostic, maximum flexibility.

15

INFRASTRUCTURE THAT IS:

HIGH PERFORMANCE

SIMPLIFICATION

OPEN, STANDARDS

BASED

MOBILITY

MANAGEABILITY

SECURITY

ENHANCED SERVICES NEEDED

HIGH PERFORMANCE

16

LATENCY WITH LEGACY NETWORK

Every hop adds additional latency

Increases load on uplinks

Requires VLANs to span multiple access switches to support VM migration

BA

17

VIRTUALIZATION WITH CHASSIS CLUSTERING

Clustered Access

Switches

10x latency improvement by eliminating trip to upper layers

Single-point lookup model

Works with any Hypervisor

BA

18

INFRASTRUCTURE THAT IS:

MOBILITY

SIMPLIFICATION

OPEN, STANDARDS

BASED

MANAGEABILITY

SECURITY

ENHANCED SERVICES NEEDED

HIGH PERFORMANCE MOBILITY

19

NETWORK REQUIREMENTS FOR VM MOBILITY

IP network with 622 Mbps is required.

The maximum latency between the two servers < 5 milliseconds (ms).

Access to the IP subnet & data storage location

Access from vCenter Server and vSphere Client. 

Same IP subnet & broadcast domain Layer 2 adjacency VLAN stretch

20

VM MIGRATION SCENARIOS

Within Same Data Center

Rack A

Layer 2 domain across racks

Scenario #1

Clustered Access Switches

Rack A

Data Centers in the same City - two different locations

Layer 2 domain across fiber connected data centers

Scenario #2

Clustered Access Switches

Data Center Data Center

Layer 2 domain across virtual private LAN

Scenario #3

Clustered Access Switches

Data Center Data Center

VPLS

Data Centers in different Cities

Remember the vMotion Requirements!Bandwidth/Latency/IP Subnet/VLAN

21

Top-of-Rack / End-of-Row Clustered

Switches

RACK TO RACK

RACK 1 RACK 2

Managed as a single device

Automatic VLAN update propagation.

Sub 10us latency

VM2 VM5VM3

NIC NIC

VM4VM1

22

VM2VM1 VM5VM4VM3

NIC NIC

VM2VM1 VM5VM4VM3

NIC NIC

POD TO POD

CoreClustered Chassis

Extends L2 domain across multiple Rows/Pods in a DC

Extends L2 adjacency to over 10,000 1GbE servers

Eliminates STP

Core managed as a single device

VM2 VM5

NIC NIC

POD NPOD 1

Clustered Access Switches

VM3 VM4VM1

23

ACROSS DC/CLOUDS

Extends L2 domain across DC /clouds

Allows VM Motion across locations.

VPLS can be provisioned or orchestrated using vendor tools and scripts

VLAN to VPLS mapping

DB/Storage mirroringVM2VM1 VM5VM4VM3

NIC NIC

VM2VM1 VM5VM4VM3

NIC NIC

VM2 VM5VM4

NIC NIC

VM2VM1 VM5VM4VM3

NIC NIC

VM2VM1 VM5VM4VM3

NIC NIC

VM2VM1 VM5VM3

NIC NIC

VM6

VPLS Over MPLS Cloud

Routers with VPLS

Core Switches

AccessSwitches

RoutersWith VPLS

VM3 VM4

CoreSwitches

AccessSwitches

VM1

24

INFRASTRUCTURE THAT IS:

MANAGEABILITY

SIMPLIFICATION

OPEN, STANDARDS

BASEDSECURITY

ENHANCED SERVICES NEEDED

HIGH PERFORMANCE MOBILITY

MANAGEABILITY

25

Network Admin

Server Admin

DC MANAGEABILITY CHALLENGES WITH SERVER VIRTUALIZATION

1. Blurred roles between the server and network admin.

2. No automation/orchestration to sync-up the 2 networks.

3. VM Migration can fail.

4. Proprietary products & protocols

B

AVirtual n/w

Physical n/w

PP

VM1 VM2 VM3 VM1 VM2

A

26

ONE STEP ORCHESTRATION

1. Clear roles and responsibilities

2. Automated orchestration between physical and virtual networks

3. Scalable solution – allows VMs to move freely

4. Open Architecture

Network Admin

Server Admin

VM1 VM2

Orchestration Tools

A

AA

A

Virtual n/w

Physical n/w

PPA A

VM2 VM3VM1

27

INFRASTRUCTURE THAT IS:

SECURITY

SIMPLIFICATION

OPEN, STANDARDS

BASED

ENHANCED SERVICES NEEDED

HIGH PERFORMANCE MOBILITY

MANAGEABILITY

SECURITY

28

VIRTUAL NETWORK

SECURITY IMPLICATIONS OF VIRTUAL SERVERS

PHYSICAL NETWORK

ES

X H

os

t

Physical Security is “Blind” toTraffic Between Virtual Machines

Firewall/IPS InspectsAll Traffic Between Servers

HYPERVISOR

VM1 VM2 VM3

29

APPROACHES TO SECURING VIRTUAL SERVERS:THREE METHODS

2. Agent-based

Each VM has a software firewall

Drawback: Significant performance implications; Huge management overhead of maintaining software and signature on 1000s of VMs

ES

X H

ost

VM1 VM2 VM3

FW Agents

HYPERVISOR

3. Kernel-based Firewall

VMs can securely share VLANs

Inter-VM traffic always protected

High-performance from implementing firewall in the kernel

Micro-segmenting capabilities

ES

X H

ost

FW as Kernel Module

VM1 VM2 VM3

HYPERVISOR

1. VLAN Segmentation

ES

X H

ost

Each VM in separate VLAN

Inter-VM communications must route through the firewall

Drawback: Possibly complex VLAN networking

HYPERVISOR

VM1 VM2 VM3

30

Hypervisor Kernel Stateful Firewall

Purpose-built virtual firewall Secure Live-Migration (VMotion) Security for each VM by VM ID Fully stateful firewall

Tight Integration with Virtual Platform Management, e.g. VMware vCenter

Fault-Tolerant Architecture

ES

X H

ostKERNEL VF

INTRODUCING THE IDEA OF A STATEFUL KERNEL FIREWALL

SecurityPolicy

Management

Data CenterFirewall

AccessSwitch

NetworkSecurity

InformationAnd Event

Management

VM1 VM2 VM3

31

ES

X H

ost

FOLLOW-ME POLICIES

Data Centre Firewall

Access Switch

ES

X H

ost

Access Switch

When a VM migrates, the network policies of the VM are migrated to the new server port.

Traffic between VMs still gets re-directed to the same appliance in the Services cluster

No migration of services state is required

Policy

VM2 VM3 VM3VM2

KERNEL VF KERNEL VF

Policy

VM1

32

SIMPLIFCATION: Few DevicesFewer Devices to

Manage

SUMMARY OF SOLUTIONS FOR SERVER VIRTUALIZATION

INFRASTRUCTURE: ADDITIONAL SERVICES

HIGH PERFORMANCEFew layersClustered Switches

OPEN:VEPAStandards Based

MOBILITY:VPLSClustered Switch domains

SECURITY:Kernel Stateful FirewallsIntegration with DC FWs for follow me policies

MANAGEABILITY:VEPAOrchestration Tools

Routers

Core Switch

Clusters

Data Center Firewalls

Access Switch Clusters

VM2VM3

SERVER 1

NIC

VM2 VM3VM1

SERVER 2

NIC

VM1