Clocker - How to Train your Docker Cloud
-
Upload
andrew-kennedy -
Category
Technology
-
view
178 -
download
8
Transcript of Clocker - How to Train your Docker Cloud
@grkvlt
Introduc/on • Andrew Kennedy – Clocker Project Founder and Lead Engineer – Open Source and Distributed Systems – github.com/grkvlt
• CloudsoG Corpora/on – Open Source Applica/on Management Specialists – We’re Hiring…
@grkvlt
Agenda • Clocker Introduc/on • What is a Docker Cloud? • Training a Docker Cloud • Demonstra/on • Roadmap
@grkvlt
What does it do?
1. Spins up and Manages Docker Clouds
2. Serves up Containers on Demand 3. Manages Composite Applica/on
Deployments on Docker
@grkvlt
What does it provide? • Mul/ Host and Mul/ Container Applica/ons • Seamless Networking – Communica/on Between Services
• Orchestra/on and Clustering – Control of Containers – Container Management
@grkvlt
Who is using it? • Tes/ng and Proof of Concept Stage – Financial Services – Insurance
• Produc/on – Mul/-‐tenant Applica/on Trial – Container per service – Via IBM BlueMix
@grkvlt
Where can I find it? • Open Source • Apache 2.0 Licensed • h[p://clocker.io
• Status • 0.7.0 Beta Release • 0.8.0-‐SNAPSHOT Development
@grkvlt
Clocker and Brooklyn • What is it? – Brooklyn Applica/on – Brooklyn Loca/on
• What does it provide? – First Class Docker Support in Brooklyn – Op/mized Brooklyn Blueprints for Docker
Apache Brooklyn • Apache Brooklyn – Open Source – Java – Donated to the Apache SoGware Founda/on – ASF Incubator Status
@grkvlt
Apache Brooklyn • Applica/on Management Pla_orm • Deploy, Manage and Monitor Blueprints • Provisioning, Installa/on and Customiza/on • Management – AutoScaling, Resilience, Performance, Security
Apache jclouds • Java Cloud Library • API Agnos/c – CloudStack, OpenStack, AWS EC2, GCE…
• Create Virtual Machines – Return SSH Endpoint – Manage Proper/es
@grkvlt
Apache jclouds • Drivers for REST APIs • Docker Driver – Wri[en by @turlinux
• Virtual Container – Using SSH Daemon – Same Endpoint Type as VM – Composi/on on any Image or Dockerfile
@grkvlt
Docker • Popular – Large Ecosystem – Growing – Complex
• Containers – Isola/on – Performance – Composable
@grkvlt
Brooklyn Blueprints • Describe Applica/ons • OASIS CAMP Standard • List of Services • Tree Structure • Sensors, Effectors and Policies
@grkvlt
Blueprint Example name: appserver-‐w-‐policy services: -‐ type: brooklyn.entity.webapp.ControlledDynamicWebAppCluster initialSize: 1 memberSpec: $brooklyn:entitySpec: type: brooklyn.entity.webapp.jboss.JBoss7Server brooklyn.config: wars.root: http://search.maven.org/remotecontent?filepath=io/brooklyn/example/brooklyn-‐example-‐hello-‐world-‐sql-‐webapp/0.6.0/brooklyn-‐example-‐hello-‐world-‐sql-‐webapp-‐0.6.0.war http.port: 8080+ java.sysprops: brooklyn.example.db.url: $brooklyn:formatString("jdbc:%s%s?user=%s\\&password=%s", component("db").attributeWhenReady("datastore.url"), "visitors", "brooklyn", "br00k11n") brooklyn.policies: -‐ policyType: brooklyn.policy.autoscaling.AutoScalerPolicy brooklyn.config: metric: $brooklyn:sensor("brooklyn.entity.webapp.DynamicWebAppCluster", "webapp.reqs.perSec.windowed.perNode") metricLowerBound: 10 metricUpperBound: 100 minPoolSize: 1 maxPoolSize: 5 -‐ type: brooklyn.entity.database.mysql.MySqlNode id: db name: DB HelloWorld Visitors brooklyn.config: datastore.creation.script.url: https://github.com/apache/incubator-‐brooklyn/raw/master/usage/launcher/src/test/resources/visitors-‐creation-‐script.sql
@grkvlt
Applica/on Management • Policies • Sensor Data Driven
• Auto Scaling • Cluster Management
• Resilience and Failover • Follow the Sun
@grkvlt
Docker Cloud • Brooklyn Blueprint for Docker Cluster • Docker Engine on Cloud VM – Configura/on for TLS Cer/ficates – Setup Volumes (Logging?) – Install Agents (Security? SDN?)
@grkvlt
Clocker Features • Applica/on Deployment – Oasis CAMP YAML Blueprint – TOSCA in Development – Same as Core Brooklyn
• Mixed Des/na/ons – Some Virtual Machines – Some Bare Metal – Some Containers
@grkvlt
Clocker Features • Docker Extensions to Brooklyn – Dockerfile or Image Specifica/on for Installa/on – Placement Strategies for Containers – Create Docker Images and Networks
• Manages Docker Engine – Deployment and Management – Installa/on and Configura/on – SoGware-‐Defined Networking
@grkvlt
Clocker Orchestra/on
Docker Engine
Virtual Machine
Container Clocker
Network Segment SDN
Cloud
@grkvlt
Training…? • Container Management • Placement and Provisioning • Autonomics – Policies and Scaling
• Headroom • Networks
@grkvlt
Container Management • Sources – Docker Image Defini/on – Docker Hub – Dockerfile – Brooklyn En/ty Defini/on
• Create Image Automa/cally – Commit or Push for Reuse
@grkvlt
Container Management • Installa/on of Services – Defined by Brooklyn or Dockerfile – Common to all En/ty Instances
• Commit Image – Available for next En/ty
• Push Image – Available for all Hosts
@grkvlt
Container Management id: dockerfile-‐mysql name: "Docker Hub MySQL Database" origin: "https://registry.hub.docker.com/_/mysql/" locations: -‐ my-‐docker-‐cloud services: -‐ type: brooklyn.entity.container.docker.application.DockerfileApplication id: mysql name: "MySQL" brooklyn.config: docker.dockerfile.url: "https://s3-‐eu-‐west-‐1.amazonaws.com/brooklyn-‐clocker/mysql-‐5.6.tgz" docker.container.environment: MYSQL_ROOT_PASSWORD: "s3cr3t”
@grkvlt
Container Management id: dockerfile-‐mysql name: "Docker Hub LAMP Stack" locations: -‐ my-‐docker-‐cloud services: -‐ type: docker:mysql:5.7.5 id: mysql env: MYSQL_ROOT_PASSWORD: "s3cr3t" -‐ type: docker:grkvlt/myapp:latest id: application env: MYSQL_HOST: component(”mysql").attributeWhenReady("host.hostname")
@grkvlt
Container Placement • Demand
– Adding an Applica/on – Scaling exis/ng Applica/on
• Requirements – Host Loca/on – Service Resources – CPU, Memory
@grkvlt
Container Placement • Supply – Choose a Host from available – Create new Host
• Start Container – Set CPU and Memory – A[ach to Network
@grkvlt
Container Placement • Placement Strategies
– Filter Hosts – Sort Hosts
• Informa/on from – Docker – Underlying Machine – Cloud Environment – En/ty or Service
@grkvlt
Container Placement • Placement Strategies – Random, Depth or Breadth First – CPU or Memory Usage – Memory, CPU or Container Limits – Geographic Constraints
• User Defined – Java Predicate
@grkvlt
Placement Strategy • Determinis/c • Simple – Predicate and Comparator
docker.container.strategies: -‐ $brooklyn:object: type: "brooklyn.location.docker.strategy.MaxContainersPlacementStrategy” brooklyn.config: maxContainers: 16 -‐ $brooklyn:object: type: "brooklyn.location.docker.strategy.CpuUsagePlacementStrategy” brooklyn.config: maxCpu: 0.75
@grkvlt
Container Provisioning • Provisioning Strategy – New Docker Host Loca/on
• Constraints – Docker Infrastructure Constraints – En/ty or Applica/on Constraints
• User Defined Strategies • Intelligent Container Orchestra/on
@grkvlt
Autonomics • Brooklyn Policies • A[ached to En//es in Applica/on – Nothing Docker Specific
• Elas/c Scaling – Cluster Resizing – Sensor Driven
@grkvlt
Applica/on Resilience • Service Resilience and Replacement – Restart Service and Container – Applica/on Level, Not Infrastructure – Same as Cloud
• Could be Integrated Further – Snapshot Running Container for Restart
@grkvlt
Headroom • Ensure resources available • Based on MaxContainers strategy limit – Or Percentage U/lisa/on – Or CPU and RAM alloca/on
• Scale Docker Host Cluster Automa/cally – Add new Docker hosts – Remove empty Docker hosts
@grkvlt
SoGware-‐Defined Networking • Needed for Seamless Provisioning • Host to Host Communica/on – Same LAN Segment – No Port Forwarding – Natural Applica/on Configura/on
• Ini/al Driver was EPMD Applica/ons
@grkvlt
Networking Providers • Standardized Interfaces • Pluggable Providers – Weave – Metaswitch Calico – IBM OpenDOVE – Others?
@grkvlt
Networking Providers • Implementa/on Agnos/c – L2 over L3 etc. – Similar to Hypervisor in Clouds
• Generic Interfaces – Host Component – Service Component (or Endpoint)
@grkvlt
Networking Capabili/es • Provide Mul/ple Networks – Single Applica/on or Shared – Private Addresses – Segmented by CIDR
• Docker Port Forwarding Access – Debug Mechanism
@grkvlt
Networking Capabili/es • IP Pool Controlled by Clocker • Clocker Controls Subnet Alloca/on – Applica/ons Segmented by CIDR
• Delegate to SDN or Cloud – Bring your own IP
• Docker Port Forwarding Access
@grkvlt
Future Capabili/es • Adding More SDN Providers • Name Resolu/on – BIND and DNSmasq or Weave DNS – Needed for JMX et al – Enables Many More En//es
• Adding More Supported Blueprints
@grkvlt
Virtual Network En/ty • New Clocker En/ty • Provisions a Virtual Network • Looks for NetworkProvisioningExtension – Registered in DockerLoca/on by SDN Provider – Could also add to JcloudsLoca/on for Clouds
• Part of Applica/on Blueprint
@grkvlt
Network Provisioning • Create Named Virtual Networks • Uses Available Provider
– Currently Weave, Calico and IBM Supported -‐ type: brooklyn.networking.VirtualNetwork networkId: my-‐application cidr: 192.168.12.0/24 gateway: 192.168.12.1 dnsServers: -‐ 8.8.8.8 brooklyn.config: sdn.example.securityGroup: "my-‐security-‐group"
@grkvlt
Network Provisioning • Allow Minimal (Zero!) Configura/on • Use Sensible Defaults
– Also SDN Specific Configura/on
• Allocate Address Space on Demand
-‐ type: brooklyn.networking.VirtualNetwork networkId: database-‐net cidr: 192.168.34.0/24 -‐ type: brooklyn.networking.VirtualNetwork networkId: couchbase-‐net
@grkvlt
Network Provisioning • A[ach Containers to Networks brooklyn.config: network.list: -‐ couchbase-‐net -‐ management-‐net
• Create Networks as Required • Also A[ach to VMs and Metal – Where Supported
@grkvlt
Features • Orchestrated Docker 1.5.0 deployment with Weave SDN integra/on
• Automated a[achment of containers to mul/ple dynamic networks
• Brooklyn applica/on blueprints with network topology
@grkvlt
Roadmap • Latest Docker 1.5.0 Feature Support
– Stats command, Read-‐only containers, Named Dockerfiles and more
• Swarm – Add Clocker as an extension to Swarm itself? – Or add features to exis/ng jclouds driver?
• Compose – Support as alterna/ve blueprin/ng language
@grkvlt
Roadmap • Networking – DNS and DNSmasq Integra/on – Rancher Integra/on – Open vSwitch Integra/on
• Gemng Started – Clocker Dockerfile
@grkvlt
Roadmap • Repositories – Docker Enterprise Hub – Bintray, Ar/factory, Quay.io – Private Repositories
• Applica/on Defini/on – TOSCA – Docker Compose
@grkvlt
Roadmap • Plugin API will be helpful • Also projects like Powerstrip
• Provide services to Docker this way • Orchestra/on • Policies
• Consume wider range of services • Networking and Storage
@grkvlt
Summary • Clocker – Brooklyn + jclouds + Docker + Weave + … – @clockercentral
• Solves – Composite Applica/on Management – Docker Cloud Networking – Container Placement and Provisioning