Tech Update – ACI - Cisco€¦ · Tech Update – ACI ... • All traffic within the ACI Fabric...
Transcript of Tech Update – ACI - Cisco€¦ · Tech Update – ACI ... • All traffic within the ACI Fabric...
Michael Petersen, CCIE #39836 Systems Engineer
September 2015
Hypervisor integration, lessons learned Tech Update – ACI
• VMM/Hypervisor integration – VMware, Microsoft, OpenStack
• Lessons learned – Customer Design & Deployments
• Lessons learned – Building the fabric, preparing for transition
• Tour of Demo lab – New tenant creation, new features, VMM integration & troubleshooting
Agenda:
September 2015
Cisco Data Center Strategy & Vision Defined by Applications. Driven by Policy. Delivered as a Service / Solution
BUSINESS OUTCOMES
Business Agility New Business Models Lower TCO
BUSINESS REQUIREMENTS
Compute Cloud Network
Policy
Policy
Policy
4 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMM/Hypervisor integration - Why do we need it? How does I work?
5 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI – Policy Defined Networking Logical network provisioning of stateless hardware
5
Outside (Tenant VRF)
App DB Web
QoS
Filter
QoS
Service
QoS
Filter
ACI Fabric
Application Policy Infrastructure Controller
Non-Blocking Penalty Free Overlay
APIC
HYPERVISOR HYPERVISOR HYPERVISOR
6 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Hypervisor Interaction with ACI Two modes of Operation
• ACI Fabric as an IP-Ethernet Transport
• Encapsulations manually allocated • Separate Policy domains for Physical
and Virtual
VLAN 10 VLAN 10 VXLAN 10000
Non-Integrated Mode
• ACI Fabric as a Policy Authority • Encapsulations Normalized and
dynamically provisioned • Integrated Policy domains across
Physical and Virtual
APP WEB DB
Integrated Mode
DB
6
7 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
vCenter DVS SCVMM
§ Relationship is formed between APIC and Virtual Machine Manager (VMM)
§ Multiple VMMs likely on a single ACI Fabric
§ Each VMM and associated Virtual hosts are grouped within APIC
§ Called VMM Domain
§ There is 1:1 relationship between a Virtual Switch and VMM Domain VMM Domain 1
Hypervisor Integration with ACI Control Channel - VMM Domains
vCenter AVS
VMM Domain 2 VMM Domain 3
7
8 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VXLAN VNID = 5789
VXLAN VNID = 11348
NVGRE VSID = 7456
Any to Any
802.1Q VLAN 50
Normalized Encapsulation
Localized Encapsulation
IP Fabric Using VXLAN Tagging
Payload IP VXLAN VTEP
• All traffic within the ACI Fabric is encapsulated with an extended VXLAN header • External VLAN, VXLAN, NVGRE tags are mapped at ingress to an internal VXLAN
tag • Forwarding is not limited to, nor constrained within, the encapsulation type or
encapsulation ‘overlay’ network • External identifies are localized to the Leaf or Leaf port, allowing re-use and/or
translation if required
Payload
Payload
Payload
Payload
Payload
Eth IP VXLAN Outer
IP
IP NVGRE Outer IP
IP 802.1Q
Eth IP
Eth MAC
Normalization of Ingress Encapsulation
ACI Fabric – Integrated Overlay Data Path - Encapsulation Normalization
8
9 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Hypervisor Integration with ACI VMM Domains & VLAN Encapsulation
EP
EP
EP EP
VMM Domain 1 4K EPGs
VMM Domain 2 4K EPGs
VLAN 5 VLAN 16
16M Virtual Networks
VNID 6032
§ VLAN ID only gives 4K EPGs (12 bits)
§ Scale by creating pockets of 4K EPGs
§ Map EPGs to VMM Domain based on scope of live migration
§ Place VM anywhere
§ Live migrate within VMM domain
9
10 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Hypervisor Integration with ACI Endpoint Discovery
DVS Host
APIC
VMM
Control (vCenter API)
Data Path
§ Virtual Endpoints are discovered for reachability & policy purposes via 2 methods:
§ Control Plane Learning:
- Out-of-Band Handshake: vCenter APIs
- Inband Handshake: OpFlex-enabled Host (AVS, Hyper-V, etc.)
§ Data Path Learning: Distributed switch learning
§ LLDP used to resolve Virtual host ID to attached port on leaf node (non-OpFlex Hosts)
OpFlex Host
Control (OpFlex)
Data Path
10
11 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
APIC
OpFlex: AN OPEN, extensible policy protocol OPFLEX WAS DESIGNED TO OFFER:
Policies: • Who can talk to whom
• What about
• Ops requirements
Abstract policies rather than device-specific configuration 1. Flexible, extensible definition of using XML / JSON 2.
Support for any device including virtual switches, physical switches, network services with strong interoperability across vendors
3.
Open, standardized API with an open source reference implementation 4.
OPFLEX PROXY
OPFLEX AGENT
OPFLEX AGENT
OPFLEX AGENT
HYPERVISOR SWITCH ADC FIREWALL
https://wiki.opendaylight.org/view/OpFlex:Opflex_Architecture 11
12 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
L/B
EPGAPP
EPG DB F/W
EPG WEB
Application Network Profile
VM VM VM
WEB PORT GROUP
APP PORT GROUP
DB PORT GROUP
Hypervisor Integration with ACI APIC
§ ACI Fabric implements policy on Virtual Networks by mapping Endpoints to EPGs
§ Endpoints in a Virtualized environment are represented as the vNICs
§ VMM applies network configuration by placement of vNICs into Port Groups or VM Networks
§ EPGs are exposed to the VMM as a 1:1 mapping to Port Groups or VM Networks
12
13 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
One L2 hop between Leaf and Hypervisor
vCenter 1. Blade Switch sends LLDP* to
Leaf and ESX
2. ESX & Leaf send parsed LLDP information to vCenter & APIC resp.
3. APIC receives LLDP information from vCenter
4. APIC identifies the leaf and ports where ESXi hosts are attached for the given DVS. APIC download policy to leaf and provision on
*Can use CDP or mix of CDP/LLDP
1 1
1 1
1 2
1 2 1 3 1 4
14 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
EPGs are Port-Groups – What does it look like?
14
15 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMM/Hypervisor integration - VMware vCenter integration (DVS)
16 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMWare Integration Three Different Options
+
Distributed Virtual Switch (DVS) vCenter + vShield Application Virtual Switch
(AVS)
• Encapsulations: VLAN • Installation: Native • VM discovery: LLDP • Software/Licenses:
vCenter with Enterprise+ License
• Encapsulations: VLAN, VXLAN
• Installation: Native • VM discovery: LLDP • Software/Licenses:
vCenter with Enterprise+ License, vShield Manager with vShield License
• Encapsulations: VLAN, VXLAN
• Installation: VIB through VUM or Console
• VM discovery: OpFlex • Software/Licenses:
vCenter with Enterprise+ License
16
17 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Hypervisor Integration – VMware DVS
ACI VMware vCenter
Tenant N/A EPG Port Group
Subnet N/A VMM Controller vCenter Datacenter VMM Domain Virtual Distributed Switch
Web
Policy
App
Policy
DB
Port Group – Web VLAN 100
Port Group – App VLAN 101
Port Group – DB VLAN 102
Virtual Distributed Switch
VMware vCenter Datacenter VMM
Domain
18 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
APIC Admin
VI/Server Admin Instantiate VMs, Assign to Port Groups
L/B
EPG APP
EPG DB
F/W
EPG WEB
Application Network Profile
Create Application Policy
Web Web Web App
HYPERVISOR HYPERVISOR
VIRTUAL DISTRIBUTED SWITCH
WEB PORT GROUP
APP PORT GROUP
DB PORT GROUP
vCenter Server / vShield
8
5
1
9 ACI Fabric
Automatically Map EPG To Port Groups
Push Policy
Create VDS 2
Cisco APIC and VMware vCenter Initial
Handshake
6
DB DB
7 Create Port Groups
ACI Hypervisor Integration – VMware DVS/vShield
APIC
3
Attach Hypervisor to VDS
4 Learn location of ESX Host through LLDP
18
19 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Hypervisor Integration – VMware DVS
Name of VMM Domain Type of vSwitch (DVS or AVS) Associated Attachable Entity Profile (AEP) VLAN Pool
vCenter Administrator Credentials
vCenter server information
19
20 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMM/Hypervisor integration - VMware vCenter integration (AVS)
21 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Southbound OpFlex API
VM VM VM VM
N1KV VEM
vSphere
Hypervisor Manager
§ OpFlex Control protocol - Control channel - VM attach/detach, link state
notifications § VEM extension to the fabric § vSphere 5.0 and above § BPDU Filter/BPDU Guard § SPAN/ERSPAN § Port level stats collection § Remote Virtual Leaf Support
(future)
Application Virtual Switch (AVS) Integration Overview
21
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public TECACI--2009
Extending ACI to Existing Virtual & Physical Networks VLAN & VXLAN Extension
Layer 2
AVS
Layer 2
AVS AVS
AVS AVS
AVS
§ AVS supports OpFlex to integrate with APIC
§ Supports a Full multi-hop Layer 2 Network between Nexus 9k and AVS: Investment Protection
§ Layer 2 network is required to support OpFlex bootstrapping in this phase
OpFlex
OpFlex
22
23 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Application Virtual Switch (AVS)
Hypervisor
VM VM EPG App
No Local Switching Mode
VM VM EPG Web
Punt to Leaf for all traffic
Hypervisor
VM VM EPG App
Local Switching Mode
VM VM EPG Web
Punt to Leaf for Inter-EPG traffic
No Local Switching Mode • Policy enforcement in the iLeaf • VXLAN encap only • aka “FEX Enable Mode”
Local Switching Mode (recommended) • Intra-EPG local switching • Both VLAN and VXLAN encap • aka “FEX Disable Mode”
23
24 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
APIC Admin
VI/Server Admin Instantiate VMs, Assign to Port Groups
L/B
EPG APP
EPG DB F/W
EPG WEB
Application Network Profile
Create Application Policy
Web Web Web App
HYPERVISOR HYPERVISOR
Application Virtual Switch (AVS)
WEB PORT GROUP
APP PORT GROUP
DB PORT GROUP
vCenter Server
8
5
1
9 ACI Fabric
Automatically Map EPG To Port Groups
Push Policy
Create AVS VDS 2
Cisco APIC and VMware vCenter Initial
Handshake
6
DB DB
7 Create Port Groups
ACI Hypervisor Integration – AVS
APIC
3
Attach Hypervisor to VDS
4 Learn location of ESX Host through OpFlex
OpFlex Agent OpFlex Agent
24
25 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Hypervisor Integration – VMware AVS
Name of VMM Domain Type of vSwitch (DVS or AVS)
Associated Attachable Entity Profile (AEP) VXLAN Pool
vCenter Administrator Credentials
vCenter server information
Switching mode (FEX or Normal)
Multicast Pool
25
26 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
What is the ACI vCenter Plugin?
§ A VMware vCenter Web Client plugin (vSphere 5.5) for ACI
§ Empowers virtualization admins to define network connectivity independently of the networking team while sharing the same infrastructure Virtualization admin is able to configure network connectivity (subnets, port-groups) with tenant isolation
• Focused on simplicity User does not need to understand the ACI Policy Model Follows the vCenter Web Client GUI standards No configuration of “in-depth” networking stuff – this is done through APIC by the networking expert
26
27 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI-vCenter Interactions Current Model
Network Team
Virtualiza1on Team
vCenter
ACI Port Group
ESXi Hypervisors
APIC Cluster
27
28 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI-vCenter Interactions vCenter Plugin Model
Network Team
Virtualiza1on Team
vCenter
ACI Plugin
ACI Port Group
ESXi Hypervisors
APIC Cluster
1
2
3
Virt. Admin
28
29 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMware vCenter Plugin View
29
30 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMware vCenter Plugin View
30
31 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMM/Hypervisor integration - MS SCVMM & Azure Pack integration
32 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Microsoft Interaction with ACI Two modes of Operation
• Policy Management: Through APIC • Software / License: Windows Server with
HyperV, SCVMM • VM Discovery: OpFlex • Encapsulations: VLAN, NVGRE (Future) • Plugin Installation: Manual
Integration with SCVMM
APIC
Integration with Azure Pack
APIC
• Superset of SCVMM • Policy Management: Through APIC or
through Azure Pack • Software / License: Windows Server with
HyperV, SCVMM, Azure Pack (free) • VM Discovery: OpFlex • Encapsulations: VLAN, NVGRE (Future) • Plugin Installation: Integrated
+
32
33 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
APIC Admin
SCVMM Admin Instantiate VMs, Assign to VM Networks
L/B
EPG APP
EPG DB F/W
EPG WEB
Application Network Profile
Create Application Policy
MSFT SCVMM
8
5
1
9 ACI Fabric
Automatically Map EPG To VM Networks
Push Policy
Create Virtual Switch
2
Cisco APIC and MSFT SCVMM Initial
Handshake
6
ACI Hypervisor Integration – MSFT SCVMM
APIC
3 Attach Hypervisor to Virtual Switch
4 Learn location of HyperV Host through OpFlex
HYPERVISOR HYPERVISOR
OpFlex Agent
HYPERV VIRTUAL SWITCH
7 Create VM Networks
OpFlex Agent
WEB VM NETWORK
APP VM NETWORK
DB VM NETWORK
33
Web Web App App DB
34 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential ACI FABRIC
Microsoft System Center | R2 with Service Provider Foundation
Azure Pack Portal
Cisco ACI with Azure Pack Microsoft System Center/Azure Pack
Websites, Apps, Database, VMs, ACI Provider Portal Consumer
Self-Service Portal
Websites VMs SQL Service Bus
Policy Management: APIC / Azure Pack
VM Discovery: OpFlex
Encapsulation: VLAN,
Zero touch network provisioning
Service Insertion (Physical/Virtual)
ACI PROVIDER SERVICE
OpFlex Driver
35 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco ACI and Microsoft AzurePack Workflow
ACI Fabric
WAP User Network Admin
Instantiate VMs 1
2 Create/Attach to VM Networks
Push Network Profiles to the Cisco® APIC
Fabric Bring Up
2
1
Automatically Pushes Policy on Leaf Where VM Attaches
Fabric Tracks VM Start, Attach/Detach
Get VLAN Pools Allocated For Each EPG 3
WEB APP WEB APP DB
Server 1 Server 2
Windows Azure Pack Portal
Virtual Switch
3 Access Physical & Virtual Services
4 Publish Shared Services
Full Infrastructure Visibility, Telemetry 4
Policy Enforcement
36 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Microsoft Azure Pack Integration Admin Experience
Add & Configure service providers for this deployment (APIC IP Address, Login Credentials, etc.)
Usage & Billing statistics per user and other admin functions
36
37 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Microsoft Azure Pack Integration Tenant Experience
Services this account has access to Resources of ACI service currently created and
consumed by this tenant
Application Network Profiles are created through Azure Pack, and pushed to APIC using REST
APIs
37
38 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
APIC Admin (Basic Infrastructure)
Azure Pack Tenant
3
6
ACI Fabric
Push Network Profiles to APIC
Pull Policy on leaf where EP attaches
Indicate EP Attach to attached leaf when VM starts
1
2
HYPERVISOR HYPERVISOR HYPERVISOR
ACI Azure Pack Integration
APIC
Get VLANs allocated for each EPG
Create Application Policy
7
Azure Pack \ SPF
SCVMM Plugin APIC Plugin OpFlex Agent OpFlex Agent OpFlex Agent
Instantiate VMs
5
1
4Create VM Networks
4
38
Web Web Web Web App App DB DB
39 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMM/Hypervisor integration - OpenStack integration
40 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
OpenStack definition!?
40
41 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Initial Focus on Networking (Neutron)
OpenStack Components
41
42 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI and OpenStack Integration
APIC Plugin for OpenStack
Group-Based Policy (optional)
Hypervisor: KVM with Open vSwitch
Zero Touch Provisioning
Service Insertion (Physical/ Virtual)
ACI FABRIC
OpenStack Controller Group-Based Policy
(optional) APIC ML2
Heat Horizon
HYPERVISOR OPEN
VSWITCH
OPFLEX AGENT OPEN
VSWITCH
OPFLEX AGENT
CLI
43 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Tenant
Network Security Group
Security Group Rule
Network: external Router
Port Subnet
Core API L3 + External Net Extension
Sec Grp Extension
OpenStack Neutron Networking Model
43
44 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Tenant
Bridge Domain Context (VRF)
Subject
App Profile Outside Network
Subnet
Endpoint Group
Contract
Cisco ACI Model
44
45 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco OpenStack ACI Model Neutron API Mapping
OpenStack ACI Tenant Tenant
No Equivalent Application Profile
Network EPG + Bridge Domain
Subnet Subnet
Security Group Handled by Host
Security Group Rule Handled by Host
Router L3 Context
Network:External L3 Outside
45
46 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
APIC Admin (Performs Steps 3)
OpenStack Tenant (Performs Steps 1,4) Instantiate VMs
Create Application Policy
Web Web Web Web App App 4
3
5 ACI Fabric
Automatically Push Network Profiles to APIC
Push Policy
Create Network, Subnet, Security Groups, Policy
NETWORK
ROUTING
SECURITY
1
2
DB DB
HYPERVISOR HYPERVISOR HYPERVISOR
NOVA
NEUTRON
OPEN VIRTUAL SWITCH OPEN VIRTUAL SWITCH OPEN VIRTUAL SWITCH
ACI OpenStack Integration – Phase 1
APIC
46
47 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Group-based Policy in OpenStack Approved for Juno Release
• Messy mapping ACI to current OpenStack components – Endpoint Groups (Ports + Security Groups) – Contracts (Security Groups + Security Group Rules)
• Goal : Introduce ACI model into OpenStack
• Starting with Groups and Group based Policies
https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction
47
48 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI OpenStack Integration – Phase 2
2
ACI Admin (manages physical
network, monitors tenant state)
L/B
EPG APP
EPG DB F/W
L/B
EPG WEB
Application Network Profile
Create Application Policy
3
5 ACI Fabric
Push Policy
OpenStack Tenant (Performs step 1,4) Instantiate VMs
Web Web Web Web App App 4
Create Application Network Profile
1
DB DB
HYPERVISOR HYPERVISOR HYPERVISOR
NOVA
NEUTRON
Automatically Push Network Profiles to APIC
L/B
EPG APP
EPG DB F/W
L/B
EPG WEB
Application Network Profile
APIC
49 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Coming Soon: OpFlex Integration for OVS
§ Local policy enforcement on each hypervisor
§ Floating IP / NAT support
§ APIC GUI integration / VMM Domain for OpenStack
§ Per host statistics
§ Service redirection
OpFlex Offers:
Hypervisor
vm4
Project 1 Project 2 Project 3
vm5 vm3
vm5 vm6
OpFlex Agent
OpFlex Proxy
V(X)LAN
OpenStack Controller Group-Based
Policy (optional) APIC ML2
50 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Lessons learned Customer Designs & Deployments
51 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Typical Stretched ACI fabric (1:1 L2-stretch DC)
DC-1 DC-2
WAN/Edge
ACI Fabric (3 controllers)
Compute Compute Edge/L4-7 Services Edge/L4-7 Services
Spines; 9336(Fixed), 9504 (1 SUP), Leafs 9372PX, Dedicated Leafs, POD´s
52 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Lessons learned Building the fabric, preparing for transition
VLAN 10 VLAN 20 New Server Group APIC Cluster
Policy Automation w/ ACI
2-Stage Migration Policy automation for existing devices
54 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Contracts are required for outside L2 connectivity
Bridge Domain: outside_vlan600
Tenant: ESXi VRF: VRF01
Node-101/eth1/1 Node-102/eth1/1
EPG: L2-OUT-EXT-BD
Contract = Allow Communication ANP: ESXi-hosts
vPC_to_UCS_a vlan-10
vPC_to_UCS_b vlan-10
EPG: ESXi-HOST-EPG 192.168.10.11 192.168.10.10
Outside
55 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
• Functionally we are expanding the VLAN’s into ACI.
Connect Fabric to existing Network
VLAN 600 / Subnet 10.XX.XX.0/24
P P VM VM VM
ACI Fabric
EPG-ESXi-HOST = VLAN 600
Trunk (.1Q)
56 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Tour – Demo Lab New tenant/App, EPG creation with VMM, Troubleshooting
57 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Thank you!