Leverage LXC/LXD with Kubernetes Jason McGee, IBM Fellow and VP, IBM Lin Sun, Senior Software Engineer, IBM

Agenda• Background • Why are we looking at this? • Experiments • Demo • Summary

Beta available March 20th.

Combining Docker and Kubernetes to deliver powerful tools, an intuitive user experience, and built-in security and isolation to enable rapid delivery of applications - all while leveraging IBM Cloud Services including cognitive capabilities from Watson.

www.ibm.com/cloud-computing/bluemix/containers

IBM Bluemix Container Service

Intelligent Scheduling Automated rollouts and rollbacks Container Security & IsolationDesign Your Own Cluster

Self-healing Horizontal scaling Leverages IBM Cloud & Watson Integrated Operational Tools

S M L

Service discovery & load balancing Secret & configuration management Simplified Cluster Management Native Kubernetes Experience

IBM Bluemix Container Service

IBM Bluemix | IBM Confidential | ©2017 IBM Corporation

Architecture

• Free tier worker is deployed in our account

• One free tier worker per account

• Paid tier workers are deployed in customer’s account

• Carrier-Cruiser model • Hub-Spoke model

Free tier of IBM Bluemix Container Service

Requirements for free tier

• Each tenant has only 1 kubernetes worker (2 CPU, 4 GB memory) • Isolation between each tenant • Fast launch and destroy clusters • Minimum cost yet providing a lightweight native kubernetes

experience • Easy migration to paid tier

Why are we looking at this?

• Increase density for free tier • Reduce cost for free tier • Fast deployment for free tier worker • Quick tear down for free tier worker • Many free tier clusters are idle

Experiments we explored• Run kubernetes worker in docker containers • Run kubernetes in LXC container • Run kubernetes worker in LXC Container

Run Kubernetes worker in Docker containers

• We started with running kubernetes worker in Docker • It works but requires Docker container in privileged mode

Introduction of LXD• LXD is a container hypervisor and a new user experience for LXC • Not a rewrite of LXC, led by Canonical, Ltd • 2 Key components

• A system-wide deamon (lxd) • A command line client (lxc)

• Docker vs LXD • Docker specializes in deploying applications • LXD specializes in deploying (Linux) Virtual Machines

Run kubernetes in LXC containers• Kubernetes (master + worker) in non privileged LXC container

docker profile • Can’t run Docker privileged container

• Kubernetes processes directly run in LXC • A few kubernetes containers require privileged access

Run kubernetes worker in LXC containers• Kubernetes worker in

non privileged LXC container docker profile

• Kubernetes worker processes directly run in LXC

• Easy migration to paid tier

Run kubernetes worker in LXC containers

Run kubernetes worker in LXC containers

Run kubernetes worker in LXC containers• Demo!

Density with LXC & Kubernetes• Current Free tier: 2 Core, 4 GB memory • With our LXC Experiment

• 8 Core, 8GB memory LXD host • Each LXC with idle k8s worker running: 140MB peak, 100MB average • Each LXC with k8s worker and guestbook example: 1.5GB peak, 800MB average • LXC supports hard memory limit by default but allows for soft limit • Can run 10+ LXC Kubernetes workers, assume 20% workers are highly used while

rest are idle

List of Issues we opened• Privileged Docker containers in LXD: https://github.com/lxc/lxd/

issues/2825 • Skip OOM score adjust in unprivileged containers

• https://github.com/kubernetes/kubernetes/pull/43079 • https://github.com/opencontainers/runc/pull/1386

Summary of the experiment• LXC/LXD provides fast deployment, much higher density thus lower cost • Easy migration to paid tier • Wish lists:

• Explore cpu/mem limits options • Explore copy/snapshot features • Explore DNS • Explore Kubernetes keys and certs

Thank you!