or distribution Networking with NSX-T for publication Kubernetes Container · 2018-09-05 ·...
Transcript of or distribution Networking with NSX-T for publication Kubernetes Container · 2018-09-05 ·...
#vmworld
Kubernetes Container Networking with NSX-TData Center Deep Dive
Yasen Simeonov, VMware, Inc.
NET1677BU
#NET1677BU
VMworld 2018 Content: Not for publication or distribution
Disclaimer
2©2018 VMware, Inc.
This presentation may contain product features orfunctionality that are currently under development.
This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.
VMworld 2018 Content: Not for publication or distribution
Agenda
3©2018 VMware, Inc.
NSX-T IntroQuick level set on NSX-T
Kubernetes OverviewTechnical overview of Kubernetes, nomenclature & networking details
NSX-T & KubernetesDetails of the NSX-T integration with Kubernetes
DemoSeeing is believing
VMworld 2018 Content: Not for publication or distribution
4©2018 VMware, Inc.
NSX-T Data Center IntroQuick level set on NSX-T Data Center
VMworld 2018 Content: Not for publication or distribution
5©2018 VMware, Inc.
BRANCH
BRANCH
BRANCH
BRANCH
BRANCH
BRANCH
BRANCH
BRANCH
TELCO/NFV
TELCO/NFV
EDGE/IOT
TELCO/NFV
BRANCH
BRANCH
EDGE/IOT
EDGE/IOT
The Virtual Cloud NetworkConnect and protect your business
VMworld 2018 Content: Not for publication or distribution
6©2018 VMware, Inc.
Identity
Apps and Data
Policy ScalabilityAnalytics and Insights
Secure Connectivity Availability
Users
Private Data Centers
VMs, Containers, Microservices
Branch Offices
Public Clouds
Telco Networks
Things
Virtual Cloud NetworkingConnect & Protectany workload across any environment
Built-in
Automated
Programmable
Application Centric
VMworld 2018 Content: Not for publication or distribution
7©2018 VMware, Inc.
NETWORKING AND SECURITY MANAGEMENT AND AUTOMATION
vRealize AutomationEnd-to-end workload automation
Network InsightNetwork discovery and insights
Cloud-Based Management Workflow Automation Blueprints / Templates Insights / Discovery Visibility
NETWORK AND SECURITY VIRTUALIZATION
AppDefenseModern application
security
NSX SD-WANby VeloCloud
WAN connectivity services
NSX Hybrid ConnectData center and cloud
workload migration
NSX Data CenterNetworking and
security for data centerworkloads
NSX CloudNetworking and
security for Public Cloud workloads
Security Integration Extensibility Automation Elasticity
VMware NSX PortfolioThe foundation of the Virtual Cloud Network
VMworld 2018 Content: Not for publication or distribution
8©2018 VMware, Inc.
Central Control ClusterCCP
Local Control PlaneLCP
NSX-T Data Center Architecture and Components
Cloud Consumption
Data PlaneESXi
(+ kernel modules)
Control Plane
Management Plane
NSX Edge VM or
Bare Metal
Layer 2 Bridge
KVM(+ kernel modules)
Highly available and scalableBuilt for consumption by
developers
Support for endpoint heterogeneity
Improved performance and resiliency
OpenStack, k8s or Custom
VMworld 2018 Content: Not for publication or distribution
9©2018 VMware, Inc.
Data Plane
Improved performance and resiliency
Admin
Tenants/CMP
Designed for multi-tenancy and scale
New distributed edge architecture with increased
performance with DPDK
p1 p2
HV TN1vSwitch1
TEP
Overlay Transport Zone
TEP: Overlay Tunnel End Point
(with its own IP address)
GENEVE Tunnel
p1 p2
HV TN1 vSwitch2
TEP
Next gen overlay maintaining
performance with increased flexibility
EdgeNode
Edge Cluster
EdgeNode
EdgeNode
EdgeNode
VMworld 2018 Content: Not for publication or distribution
10©2018 VMware, Inc.
Kubernetes OverviewTechnical overview of Kubernetes, nomenclature & networking details
VMworld 2018 Content: Not for publication or distribution
11©2018 VMware, Inc.
Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure.
What is Kubernetes?
VMworld 2018 Content: Not for publication or distribution
12©2018 VMware, Inc.
Kubernetes Components
K8s Cluster Consists of Master(s) and Nodes
K8s Master Components• API Server• Scheduler• Controller Manager• Dashboard
K8s Node Components• Kubelet• Kube-Proxy• Containers Runtime
(Docker or Rocket)
K8s masterK8s master
K8s Master
Controller Manager
K8s APIServer
Key-Value Store
dashboard
Scheduler
K8s nodeK8s nodeK8s nodeK8s node
K8s Nodes
kubelet c runtime
Kube-proxy
> _ Kubectl
CLI
K8s Master(s)
VMworld 2018 Content: Not for publication or distribution
13©2018 VMware, Inc.
Kubernetes Pod
A Pod is a group of one or more containers that shares an IP address and a Data Volume Pod
pause container(‘owns’ the IP stack)
10.24.0.0/16
10.24.0.2
nginxtcp/80
mgmttcp/22
loggingudp/514
IPC
External IP Traffic
VMworld 2018 Content: Not for publication or distribution
14©2018 VMware, Inc.
Kubernetes Namespace
Namespaces are a way to divide cluster resources amongst users and groups
They can be thought of as Tenants
They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Name uniqueness
Namespace: fooBase URI: /api/v1/namespaces/foo
‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master
‚redis‘ service:/api/v1/namespaces/foo/services/redis-master
Namespace: barBase URI: /api/v1/namespaces/bar
‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master
‚redis‘ service:/api/v1/namespaces/bar/services/redis-master
VMworld 2018 Content: Not for publication or distribution
15©2018 VMware, Inc.
Kubernetes Service
A Kubernetes Service defines a logical set of Pods, selected with matching labels
Serves multiple functions:• Service Discovery / DNS• East/West load balancing
in the Cluster (Type: ClusterIP)
• External load balancing for L4 TCP/UDP (Type: LoadBalancer)
• External access to the service through the nodes IPs (Type: NodePort)
Redis Slave Pods
redis-slave svc
10.24.0.5
ClusterIP172.30.0.24
Web Front-EndPods
10.24.2.7
▶ kubectl describe svc redis-slaveName: redis-slaveNamespace: defaultLabels: name=redis-slaveSelector: name=redis-slaveType: LoadBalancerIP: 172.30.0.24LoadBalancer Ingress: 134.247.200.20Port: <unnamed> 6379/TCPEndpoints: 10.24.0.5:6379,
10.24.2.7:6379
DNS:
redis-slave.<ns>.cluster.local 172.30.0.24
ExternalIP134.247.200.20
DNS:
redis-slave.external.com 134.247.200.20
VMworld 2018 Content: Not for publication or distribution
16©2018 VMware, Inc.
Kubernetes Ingress
A Kubernetes Ingress Object is a L7 LoadBalancing rule that binds a hostname and url to a Service
The LoadBalancer Datapath can be implemented as an external Load Balancer or as a K8s Pod
Web Front-EndPods (shop svc)
http://www.bikeshop.com/shop
Web Front-EndPods (special-offers svc)
http://www.bikeshop.com/special-offers
LoadBalancer Datapath
(External or K8s Pods)
▶ kubectl describe ingress bikeshop-ingress-shopName: bikeshop-shopNamespace: bikeshopAddress: 100.64.240.9,134.247.200.1Default backend: default-http-backend:80 (<none>)
Rules:Host Path Backends---- ---- --------www.bikeshop.com /shop
web-svc-1:80 (<none>)
External IP: 134.247.200.1
DNS: *.bikeshop.com 134.247.200.1
VMworld 2018 Content: Not for publication or distribution
17©2018 VMware, Inc.
Kubernetes Networking Topologies
Every Node is an IP Router and responsible for its Pod Subnet
Subnets are associated with Nodes, not Tenants
Physical Network Configuration is required
Non-multitenant routed topology
Nodeint eth0
10.240.0.4
int cbr0
10.24.2.1/24
10.24.2.2 10.24.2.3 10.24.2.4
ip route 10.24.1.0/24 10.240.0.3ip route 10.24.2.0/24 10.240.0.4
Nodeint eth0
10.240.0.3
int cbr0
10.24.1.1/24
10.24.1.2 10.24.1.3 10.24.1.4
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
VMworld 2018 Content: Not for publication or distribution
18©2018 VMware, Inc.
Kubernetes Networking Topologies
Overlays are typically used to avoid Physical Network Configuration
Subnets are still associated with Nodes, not Tenants
External outbound connectivity needs SNAT using the Nodes IP
External inbound connectivity needs Node Port or Ingress in Host Network Mode
Node-to-Node overlay topology
Nodeint eth0
10.240.0.4
int cbr0
10.24.2.1/24
10.24.2.2 10.24.2.3 10.24.2.4
Nodeint eth0
10.240.0.3
int cbr0
10.24.1.1/24
10.24.1.2 10.24.1.3 10.24.1.4
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
Overlay
Key-Value Store
VMworld 2018 Content: Not for publication or distribution
19©2018 VMware, Inc.
NSX-T & KubernetesDetails of the NSX-T integration with Kubernetes
VMworld 2018 Content: Not for publication or distribution
20©2018 VMware, Inc.
Key Design Goals of the NSX-T Data Center Kubernetes Integration
Don't stand in the way of the developer!
Provide solutions to map the Kubernetes
constructs to enterprise networking
constructs
Secure Containers, VMs and any other
endpoints with overarching Firewall
Policies
Provide visibility & troubleshooting tools to ease the
container adoption in the enterprise
VMworld 2018 Content: Not for publication or distribution
21©2018 VMware, Inc.
Kubernetes NSX Topology
Dynamically network topology per K8s namespace
K8s Nodes are not doing IP routing
Every Pod has its own logical port on a NSX logical switch, and is supporting all features a VM interface supports
Every Pod has Dynamic Firewall rules applied on its logical Interface
Dynamic per Namespace Topology
Namespace: foo Namespace: bar
NSX/ K8s topology
10.4.0.0/26 10.4.0.64/26 34.1.2.33/26
VMworld 2018 Content: Not for publication or distribution
22©2018 VMware, Inc.
K8s / NSX Components
NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod.
NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems at some point
NSX Container Plugin (NCP)
NCM Infra
K8s / OSAdapter
CloudFoundryAdapter
NSX Container Plugin
More…
NSX Manager
API Client
NSX Manager
NS: foo NS: barNSX/ K8s topology
K8s master
etcd
API-Server
Scheduler
VMworld 2018 Content: Not for publication or distribution
23©2018 VMware, Inc.
With most networking technologies in K8s like Flannel, OpenShift OVS Networking, Calico, etc. the source IP of the traffic can't be mapped to the tenancy. This is the biggest hurdle today to get K8s integrated in enterprise IT environments
Tenancy / Topology MappingThe open source way
Node VM
IPTables(NAT)
vnic
mgmt IP
Pods
10.255.0.10/2410.255.0.9/24
172.16.1.11/24
Node VM
IPTables(NAT)
vnic
mgmt IP
Pods
10.255.1.3/2410.255.1.5/24
172.16.1.12/24
Physical or virtual Router
172.16.1.1/24
Tenant: fooTenant: barTenant: foo
Database (VM based or Physical)
Physical DC FirewallSNAT to Node IP
Did the traffic come from 'foo’ or 'bar'?
SNAT to Node IP
VMworld 2018 Content: Not for publication or distribution
24©2018 VMware, Inc.
With NSX-T each Tenant (Kubernetes Namespace) either gets its own SNAT IP (NAT Mode), or is directly identifiable by its source subnet (No NAT Mode)
Tenancy / Topology MappingPersistent IPs for K8s Namespaces
Node VM
OpenvSwitch
10.12.5.5/2410.12.1.8/24
172.16.1.11/24
mgmt IP
vnic
Namesp. FooT1 router
PAS VMsT1 router
VLAN Trunk
NSX-T Logical Switch
Namesp. BarT1 router
172.16.1.1/24 10.12.1.1/24 10.12.5.1/24
Pods
Database (VM basedor Physical)
Physical DC Firewall
A new SNAT IP is allocated on the T0 router for each Tenant for NAT Mode
In NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source SNAT IP that is allocated to a specific Tenant.
Tenant: fooTenant: bar
In No-NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source IP Subnet that is allocated to a specific Tenant.
VMworld 2018 Content: Not for publication or distribution
25©2018 VMware, Inc.
Infrastructure Teams can pre-create Firewall rules in existing DC physical Firewalls to allow traffic from specific workloads in K8s
The K8s user / DevOps can deploy applications that are easily identifiable in the physical network
With this feature a set of Kubernetes Workloads (Pods) can be assigned to use a specific IP or group of SNAT IPs to source their traffic from
Before this feature we only assigned a SNAT IP to a Kubernetes Namespace
Feature
Benefits
Persistent SNAT IP per K8s ServiceSpecifying the source IP Kubernetes Workloads using the K8s service
Tier0 LR
Corporate network
DB
allow – from: 134.247.100.10 (App) to: 134.247.200.9 (DB)
Tier1 LR
Kubernetes Namespace: Foo
Web-FrontendPods
App Logic Pods
K8s Svc for AppK8s Svc for Web
Namespace LS(s)
SNAT App Svc Pods to: 134.247.100.10For all other Pods
use namespace SNAT IP
VMworld 2018 Content: Not for publication or distribution
26©2018 VMware, Inc.
Central Visibility
With most other networking technologies in K8s and PCF like Flannel, OpenShift OVS Networking, PCF Silk, Calico, etc. there is no centralized control plane. So, there’s no counters, troubleshooting tools, 'span ports', Firewall Rules Overview, etc.
VMworld 2018 Content: Not for publication or distribution
27©2018 VMware, Inc.
Central Visibility
With NSX-T you are gain deep visibility into the container networks, and you can use the same troubleshooting tools we created for VM based workloads
VMworld 2018 Content: Not for publication or distribution
28©2018 VMware, Inc.
Kubernetes Metadata / NSX Logical Port Mapping
▶ kubectl get pod nsx-demo-rc-c7x65 -o yaml
apiVersion: v1kind: Podmetadata:creationTimestamp: 2018-07-25T12:05:56ZgenerateName: nsx-demo-rc-labels:
app: nsx-demoname: nsx-demo-rc-c7x65namespace: nsx-ujo
Metadata within Kubernetes like Namespace, Pod names, Labels all get copied to the NSX Logical Port as Port Tags
VMworld 2018 Content: Not for publication or distribution
29©2018 VMware, Inc.
Pre-Created Security Groups / Firewall rules (admin rules)
NSX can be configured to collect ports and switches in dynamic security groups based on Tags (Kubernetes Metadata) and apply Firewall rules on them
Match on Port Tags
Matching Pods are part of the Group
Groups are used in Firewall sections as src and dst
VMworld 2018 Content: Not for publication or distribution
30©2018 VMware, Inc.
Unified Policy for K8s, PCF & VMs
Both K8s and PCF have 'built-in' micro segmentation policy languages (network policy), and there's a broad set of products and open source projects implementing micro segmentation inside of K8s or PCF. However there is no technology other than NSX-T today that allows you to define policies across K8s, PCF and VM based workloads using Metadata from each system
PCF Org FooT1 routerDB VMs
T1 router
Kubenetes Namespace: BarT1 router
NSX-T Logical Switch NSX-T LS NSX-T Logical Switch
K8s PodsPCF AIs
vSphere VMs
allow: tcp/443
allow: tcp/3306 (mysql)
VMworld 2018 Content: Not for publication or distribution
31©2018 VMware, Inc.
Support of Kubernetes Network Policy
Besides supporting admin pre-defined rules, NCP is also translating Kubernetes NetworkPolicy Objects to NSX security groups and Firewall rules
Admin pre-defined rules can be used concurrently in NSX, admin rules are put in sections before or after K8s network policy rules
apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:name: default-deny
spec:podSelector: {}policyTypes:- Ingress
apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:
name: nsx-demo-policyspec:
podSelector:matchLabels:
app: nsx-demopolicyTypes:- Ingressingress:- from:
- ipBlock:cidr: 100.64.160.11/32
ports:- port: 80
protocol: TCP
VMworld 2018 Content: Not for publication or distribution
32©2018 VMware, Inc.
Built-in Load Balancing
NCM Infra
K8s / OSAdapter
CloudFoundry Adapter
Libnetwork Adapter
NSX Container Plugin
More…
NSX Manager
API Client
NSX Manager
K8s master
etcd
API-Server
Scheduler
Virtual Server10.114.209.209HTTP and/or
HTTPS traffic
Server Pool 1
Server Pool 2Rule 2/bar/
Rule 1/foo/
LB Service
NCM Infra
K8s / OSAdapter
CloudFoundry Adapter
Libnetwork Adapter
NSX Container Plugin
More…
NSX Manager
API Client
NSX Manager
K8s master
etcd
API-Server
Scheduler
Virtual Server10.114.209.212TCP and/or
UDP traffic
Server Pool
LB Service
We have built-in support for Ingress (L7 HTTP/HTTPS) and Svc Type LB (L4 TCP/UDP) in the NSX-K8s integration. Most other K8s networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX from Ingress (L7).
VMworld 2018 Content: Not for publication or distribution
33©2018 VMware, Inc.
K8s / NSX Workflows
1. NCP watches for Svc events in Kubernetes
2. User creates a new Svc of Type LoadBalancer
3. The Kubernetes API server notifies NCP of the new Svc
4. NCP creates a new Virtual Server with a unique IP and a Server Pool with the Pods as targets
Svc Type LB
NCM Infra
K8s / OSAdapter
CloudFoundryAdapter
NSX Container Plugin
More…
NSX Manager
API Client
NSX Manager
K8s master
etcd
API-Server
Scheduler
1)2)
3)
4)
Virtual Server10.114.209.212TCP
and/orUDP traffic
Server Pool
LB Service
VMworld 2018 Content: Not for publication or distribution
34©2018 VMware, Inc.
K8s / NSX Workflows
1. NCP watches for Ingress events in Kubernetes
2. User creates a new Ingress rule
3. The Kubernetes API server notifies NCP of the new Ingress rule
4. NCP creates a new forwarding rule sending a specific HTTP/S hostname and path to a specific Server Pool
Ingress
NCM Infra
K8s / OSAdapter
CloudFoundryAdapter
NSX Container Plugin
More…
NSX Manager
API Client
NSX Manager
K8s master
etcd
API-Server
Scheduler
1)2)
3)
4)
LB Service
Virtual Server 10.114.209.209HTTP and/or
HTTPS traffic
Server Pool 1
Server Pool 2Rule 2/bar/
Rule 1/foo/
VMworld 2018 Content: Not for publication or distribution
35©2018 VMware, Inc.
NSX-T Data Center TimelineKubernetes, OpenShift and PKS
2017 2018
September October November December January February March
NSX-T 2.1Support for PKS 0.8 and PKS 1.0
Support for K8s Ingress and Svc Type LB with Platform LB
Core value add:
• One of the only SDN solution in the market that includes LB with Ingress and Svc Type LB for K8s
• PKS / OPS MGR Integration
• Gives PKS support for Network Policy
NSX-T 2.0:Support for 'Do It Yourself' K8s & OpenShift
Core value add:
• Mapping of K8s Namespaces to Network Topology & source IP Addresses
• NAT & No-NAT modes per Namespace
• Network Policy (Firewall) across K8s and VM workloads
• Support for K8s Network Policy
• Logical Network Port per K8s workload (Pod) for visibility and troubleshootingVMworld 2018 Content: Not for publication or distribution
36©2018 VMware, Inc.
NSX-T TimelinePCF 2.0
2018
January February March April May June July
NSX-T 2.1Support for PCF 2.0 -> PAS
Core value add:
• Allows mapping of CF tenancy (Orgs) to Network Topology & source IP Addresses
• Network Policy (Firewall) support across PKS, PCF and VM workloads
• Only solution that allows for direct, no_NATcommunication from CF Apps to backend services
• Logical Network Port per CF workload (AI) for visibility and troubleshooting
NSX-T 2.2Operational Enhancement &Additional LB features
Core value add:
• Persistent SNAT IP for Kubernetes Services and CF Apps
• TLS/SSL Offload support for Kubernetes Ingress
• OpenShift 'router' support for HTPP and HTTPS (feature parity with K8s Ingress)
• URL rewrite support for K8s Ingress
• Various install & operational improvementsVMworld 2018 Content: Not for publication or distribution
©2018 VMware, Inc. 37
NSX-T & Kubernetes –Demo
VMworld 2018 Content: Not for publication or distribution
38©2018 VMware, Inc.
NSX-T Data Center Values for Containers
Enterprise-class Networking
Advanced Security
Enhanced Operations
Full Network Visibility
Enterprise Support
Unified VM-to-Container Networking
Micro-Segmentation
N S X - T V a l u e s f o r C o n t a i n e r s
F e a t u r e sVMworld 2018 Content: Not for publication or distribution
39©2018 VMware, Inc.
Join the NSX VMUG Communityvmug.com/nsxConnect with your Peerscommunities.vmware.com
Embrace the NSX Mindsetnsxmindset.comFind NSX Resourcesvmware.com/go/networkingRead the Network Virtualization Blogblogs.vmware.com/networkvirtualization
Where to Get Started
Attend the Networking and Security SessionsShowcases, breakouts, quick talks & group discussions
Visit the VMware BoothProduct overviews, use-case demos
Visit Technical Partner BoothsIntegration demos – Infrastructure, security, operations, visibility, and more
Meet the ExpertsJoin our experts in an intimate roundtable discussion
Free Hands-on Labslabs.hol.vmware.com
Virtual Cloud Network Guided Demovcndemo.com
VMware Education – Training and Certificationvmware.com/go/nsxtraining
Free NSX Training on Courseravmware.com/go/coursera
Engage and Learn Experience
Try Take
VMworld 2018 Content: Not for publication or distribution
40©2018 VMware, Inc.
VMworld 2018 Content: Not for publication or distribution
PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.
#vmworld #NET1677BU
VMworld 2018 Content: Not for publication or distribution
THANK YOU!
#vmworld #NET1677BU
VMworld 2018 Content: Not for publication or distribution