Download - DevOps Guide to Container Networking

DevOps Guide toContainer Networking

Dirk Wallerstorfer DevOpsSummit New York, June 8th

2

Technology Lead SDN, OpenStack

[email protected]@wall_dirkblog.ruxit.com

3Dirk Wallerstorfer, @wall_dirk


SDN

http://systematicrelativestrength.com/2013/11/12/your-plan-vs-reality/


web:$ docker run -itd wordpress



user:wordpress$ ping 8.8.8.8

iptables –t nat –A POSTROUTING –o eth0 –j MASQUERADE


web:$ docker run –itd –p 8080:80 wordpress


web:$ docker run –itd –p 8080:80 wordpress

iptables –t nat –A PREROUTING ... –j DOCKERiptables –t nat –A DOCKER --dport 8080 --redirect-to 172.18.0.2:80


SDN

Dirk Wallerstorfer, @wall_dirk 19

Three reasons for SDN• Permanent connectivity• Virtualization of everything• Paradigm shift in software development


Three reasons for SDN

Networking had to keep up somehow!

Continuous delivery

Virtualize everything

Permanent connectivity


SDN• Classic SDN

• SD WAN

• Network Overlay


Multi-host Container NetworkingNo SDNdb:$ docker run -itd –p 3306:3306 mysql

web:$ docker run -itd –p 8080:80 –e WORDPRESS_DB_HOST=172.16.198.248:3306 wordpress


Multi-host Container NetworkingPrerequisites• Underlying network

• Distributed K/V store

• Accessible ports


Multi-host Container Networking

Overlay No overlay

http://s568.photobucket.com/user/LMG_09/media/CrowdSurfftw.jpg.html Ocean’s Eleven, Warner Bros, 2001


Multi-host Container NetworkingOverlay Protocols

• VXLAN

OuterEthernet

OuterIP

OuterUDP VXLAN Ethernet IP TCP Payload



• VXLAN

OuterEthernet

OuterIP


Ethernet IP UDP VXLAN Ethernet IP TCP Payload



• VXLAN

OuterEthernet

OuterIP



Flags Reserved VXLAN Network Identifier (VNI) Reserved



• VXLAN

OuterEthernet

OuterIP



14 bytes 20 bytes 8 bytes 8 bytes

+ 50 bytes



• VXLAN• Ethernet in UDP, defacto standard, won the overlay war

• NVGRE• Ethernet in IP, Microsoft’s answer to a question nobody asked

• STT• Ethernet in fake TCP, to utilize TSO of NIC

• Geneve• Ethernet in UDP, best of breed approach• A+ for extensibility• https://packetpushers.net/podcast/podcasts/pq-show-68-geneve-data-center-overlay-update/

https://packetpushers.net/podcast/podcasts/pq-show-68-geneve-data-center-overlay-update/

https://packetpushers.net/podcast/podcasts/pq-show-68-geneve-data-center-overlay-update/


Multi-host Container NetworkingOverlay

• Docker Libnetwork• WeaveNet• Flannel


Docker libnetwork

https://blog.docker.com/2015/04/docker-networking-takes-a-step-in-the-right-direction-2/


Docker libnetwork


Why UDP?


Departmentof

RedundancyDepartment


Multi-host Container NetworkingNo overlay

• Project Calico • Flannel host-gw• Romana• Contiv• MACVLAN/IPVLAN


Project Calico

https://www.projectcalico.org/docker-libnetwork-is-almost-here-and-calico-is-ready/


© http://de.slideshare.net/grkvlt/metaswitch-project-calico

Host Host

Containers Containers


Project Calico• Host is a router for the workloads• BGP to distribute routes• etcd backed• Pure Layer 3, no encapsulation


Project Calico


Location of services

k8s pods, marathon application groups, swarm constraints, fleet units


Connectivity Problemsnf_conntrack: table full, dropping packet.

dirk@fueldev:~$ sudo sysctl –a | grep conntrack...net.netfilter.nf_conntrack_buckets = 8192net.netfilter.nf_conntrack_count = 0net.netfilter.nf_conntrack_max = 31760...

• Large number of iptables rules


Connectivity Problems• The notorious MTU• https://www.youtube.com/watch?v=H2lBkj5zbYs

dirk@fueldev:~$ ip addr show enp0s32: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:f3:4e:5d brd ff:ff:ff:ff:ff:ff inet 172.16.99.14 brd 172.16.11.255 scope global enp0s3 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:fef4:4e56/64 scope link valid_lft forever preferred_lft forever


TCP/IP over VXLAN Overhead


14 bytes 20 bytes 8 bytes 8 bytes

+ 50 bytes

Send 1MB of data

1,000,000 bytes = 710 packets á 1410 bytes710 x 50 bytes = 35,500 bytes overhead

1,035,500 bytes are transmitted

3.55 %


Send 1MB of data

1,000,000 bytes = 736 packets á 1330 bytes736 x 100 bytes = 73,600 bytes overhead

1,073,600 bytes are transmitted

TCP/IP over VXLAN over VXLAN Overhead

Ethernet IP UDP VXLAN Ethernet IP UDP VXLAN Ethernet IP TCP Payload

14 bytes

20 bytes

8bytes

8bytes

14 bytes

20 bytes

8bytes

8bytes

+ 100 bytes

7.36 %


YOU WERE SO PREOCCUPIED WITH WHETHER OR NOT YOU COULD

YOU DIDN’T STOP TO THINK IF YOU SHOULD


1460

1410136013101260

12101160

MTU overhead

25,9%20,7%

15,9%11,5%

7,4%3,6%

0%

70Dirk Wallerstorfer, @wall_dirkFebruary 20, 2016http://machinezone.github.io/research/networking-solutions-for-kubernetes/

Performance Comparison of Networking

Solutions for Kubernetes




with --net=host

aws-vpc

vxlan

host-gw

IPvlan

February 20, 2016http://machinezone.github.io/research/networking-solutions-for-kubernetes/




with --net=host

aws-vpc

vxlan

host-gw

IPvlan

libnetwork



https://github.com/machinezone/tcpkali

serving 350 byte responsemaking 250,000 requests per second



Different network options - latency?


250,000 requests per second, 350 bytes response



> 3 sec

46 %response time

will leave the page


+0.5 s

-11 %response time

in revenue

keep it manageable

keep it simple

keep it fast


http://i.coastingfish.com/image/3M

Volume-oriented network metrics

Quality-oriented network metrics


http://i.coastingfish.com/image/3M

83

Technology Lead SDN, OpenStack

[email protected]@wall_dirkblog.ruxit.com

Image sources:pixabay.com (3, 4, 5, 7, 9, 10, 23, 41, 57, 59, 60, 61)