An Introduction to Container Organization with Docker Swarm, Kubernetes, Mesos, and Neo4j
-
Upload
neo4j-the-fastest-and-most-scalable-native-graph-database -
Category
Software
-
view
374 -
download
3
Transcript of An Introduction to Container Organization with Docker Swarm, Kubernetes, Mesos, and Neo4j
A Gentle (Hands-on) Introductionto Container Orchestration withDocker Swarm, Kubernetes, Mesos
Dippy AggarwalPh.D. Candidate, University of Cincinnati, Ohio
Dippy Aggarwal Dissertation focus: Graph databases (Neo4j), data warehouses, schema evolution and provenance
Summer Intern 2016 at Cincinnati Children’s Hospital, Biomedical Informatics, High Performance Computing Team
High Performance Computing Team, Biomedical Informatics,
Cincinnati Children’s Hospital
Carmen De VitoPrakash Velayuthum
Roberto Perea
Kevin Sandy
Mark Cunningham Jason Curtis
OutlineMotivation: To present a conceptual and hands-on introduction for deploying Neo4j containers across a cluster using the three popular container orchestration tools –
Docker Swarm, Kubernetes and Mesos.
Orchestration 101
Overview of three orchestration tools: Kubernetes, Docker Swarm and Mesos
Demos – Automated cluster deployment of neo4j with the three orchestration approaches
What is Docker/Containerization all about?
Ships goods Ships software
Stackable, portable, isolated
What purpose does containers solve in general?
[1] http://www.computerweekly.com/feature/Demystifying-Kubernetes-the-tool-to-manage-Google-scale-workloads-in-the-cloud[2] The Docker Book: Containerization is the new virtualization, James Turnbull, https://www.docker.com/what-docker
• Avoid “Runs on my machine” issues• Package your application as a standardized unit
Docker Progression (contd.)
Docker Adoption Is Up 30% in One Year 2/3 of Companies That Try Docker Adopt It
Adopters 5x Their container Count within 9 Months Docker Now Runs on 10% of the Hosts We Monitor
Source: https://www.datadoghq.com/docker-adoption/
https://github.com/neo4j/docker-neo4j/blob/master/2.3.5/Dockerfile
Environment variables
Download and install Neo4j
Exposing ports
Running neo4j on start
Neo4j dockerfile
docker build –t neo4j:2.3 .docker run –publish 7474:7474 –publish 7473:7473 neo4j:2.3
The Challenge• Containers by themselves are difficult to scale, achieve fault-tolerance• How to handle replication?• How to make multiple containers communicate?
How to deploy and manage multiple containers across a cluster of machines?
Orchestration is this idea of going from launching a container on one machine to multi-containers spread across a fleet of machines.
Orchestration Tools Survey
Kubernetes, Docker Swarm and Mesos – among the
leading products
http://thenewstack.io/tns-research-present-state-container-orchestration/
Last year@GraphConnect SF 2015
GraphConnect 2015 David Makogon, Microsoft and Patrick Chenzon, Docker - Containerized Neo4j: Automating Deployments with Docker
https://neo4j.com/blog/neo4j-containers-docker-azure/
Docker Swarm
Single Docker Engine Docker Swarm
- Native tool by Docker- Serves the standard Docker API
Swarm-manager• Add additional nodes to the cluster seamlessly• Support single pool of resources• Maintains state of all the containers running on different docker engines• Make scheduling decisions
Image credits: http://www.slideshare.net/rajdeep/docker-swarm-introduction
Scheduling in Docker Swarm
Spread strategy: Swarm optimizes for the node with the least number of containers. Binpack strategy: Swarm optimize for the node which is most packed.
Swarm scheduler strategies
Running two containers on the same hostRunning containers on nodes meeting certain constraints: health checks, storage etc.docker tcp://<manager_ip:manager_port> run -d --name logger -e affinity:container==frontend logger
Swarm filters
Docker swarm cluster-setup
Commands to set up a Swarm cluster
Create discovery tokensdocker-machine create –d virtualbox localdocker run swarm createexport TOKEN=<token obtained from the last command>
Launching master and two agent nodes forming a cluster
Master: docker-machine create –d virtualbox --swarm --swarm-strategy=binpack --swarm-master --swarm-discovery token://${TOKEN} swarm-master Agent: docker-machine create –d virtualbox –swarm --swarm-discovery token://${TOKEN} swarm-agent1
Optional. Default strategy is spread
Demo: Deploy Neo4j Docker Container locally (VirtualBox) using Docker Swarm
Docker SwarmPros and Cons
+ Simplicity+ With Docker 1.12, several advanced features such as filters, auto-scaling made simpler
- Limited by Docker API functionality
docker service create –name frontend –replicas 5 -p 80:80/tcp nginx:latestScaling: docker service scale frontend=100
What is Kubernetes• Container Orchestration tool developed by Google but
many participants• Container cluster manager
Image credit: http://www.webopedia.com/TERM/G/google-container-engine.html
Kubernetes- core concepts• Pod• Service• Replication Controller• Deployment
• Etcd• API server• Scheduler• Kubelet daemon• Kube-proxy
Service
Replication Controller
Understanding Kubernetes Primitives
Image credits: https://keithtenzer.com/2015/06/01/application-containers-kubernetes-and-docker-from-scratch/
Hands-On: Deploy Neo4j Docker Container on Google Cluster using Kubernetes
Kubernetes - Pros and Cons+ Driven by Google+ Provides more concepts than Swarm+ Docker 1.12 leveraging Kubernetes idea of abstraction using pods and services
Cluster Manager
How Mesos help
Image Credits: http://www.slideshare.net/charmalloc/introductionapachemesosjstein20140714?next_slideshow=2
How does Mesos help?
No static cluster partitioning required
Mesos offers a level of abstraction
Interleaved workloads
Another stack variation for Mesos
Use Kubernetes as container management tool
Can even have heterogeneous cluster : private and cloud
Mesos – core components• Master: Mediator between the underlying resources and the different
frameworks.-- Makes offers to frameworks about available resources and launches tasks on slaves for accepted offers. • Slaves: actual workhorses of the cluster.-- Execute tasks submitted by frameworks. • Frameworks: applications that run on Mesos and solve a specific use-case.-- Two components: Scheduler and Executor.
Hands-On: Deploy Neo4j Docker Container using Mesos/Marathon
Which one to choose?• Use Docker Swarm if:
You want to use the familiar Docker API to build Docker containers• Use Kubernetes if: - You want to launch pods, which are groups of containers co-scheduled and co-located together, sharing resources.
- You are a google fan!• Use Marathon if: You want to launch Docker or non-Docker long-running apps/services.
Choose your own adventure!
References• http://talkincloud.com/cloud-computing-and-open-source/081914/mesosphere-launches-clusters-
google-compute-engine• http://events.linuxfoundation.org/sites/events/files/slides/CloudOpen14MesosKubGCE%20(1).pdf• "Mesos: The Operating System for your Cluster“, David Greenberg, https://www.youtube.com/watch?
v=gVGZHzRjvo0, Published: 09-21-2014• Building Web Scale Apps with Docker and Mesos, Alex Rukletsov, https://www.youtube.com/watch?
v=UdHG170jBxs, Published: 01-07-2015• http://stackoverflow.com/questions/29198840/marathon-vs-kubernetes-vs-docker-swarm-on-
mesosphere-with-docker-containers• https://open.mesosphere.com/advanced-course/installing-software/• Comparison Guide: https://apprenda.com/thank-you/gaw-container-orchestration-comparison-guide/
Thank you!
Questions/Feedback?