State of Union - Containerz
---------------------Shiva (narshiva@) --------------------
\ ^__^ \ (oo)\_______
(__)\ )\/\||----w | || ||
Containerized Microservices
Dom 0
Instance Instance Instance
OS OS OS
Container Runtime Container Runtime Container Runtime
App App
Service Service
App App
Service Service
App App
Service Service
Container Orchestration
Dom 0
Instance Instance Instance
OS OS OS
Container Runtime Container Runtime Container Runtime
App Service App App Service Service
Container Orchestration
Container Orchestration
Dom 0
Instance/OS Instance/OS Instance/OS
App Service App App Service Service
Service Management
Scheduling
Resource ManagementOrc
hest
ration
Service Management§Labels§Groups/Namespaces§Dependencies§Load Balancing§Health Check§Service Discovery
Container Orchestration
Dom 0
Instance/OS Instance/OS Instance/OS
App Service App App Service Service
Service Management
Scheduling
Resource ManagementOrc
hest
ration
Scheduling§Placement§Replication/Scaling§Resurrection§Rescheduling§Rolling deploys§Upgrades§Downgrades§Colocation
Container Orchestration
Dom 0
Instance/OS Instance/OS Instance/OS
App Service App App Service Service
Service Management
Scheduling
Resource ManagementOrc
hest
ration
Resource Management§Memory§CPU§GPU§Volumes§Ports§IPs
Non Functional Capabilities
ScalabilityPerformance, Responsiveness, Efficiency
AvailabilityFault Tolerance, Reliability, DR
FlexibilityExtensibility, Portability, Interoperability
UsabilityFamiliarity, Debuggability, Maintainability
PortabilityContainer Runtime, Host OS, Cloud Provider, On-prem
SecurityIsolation, Encryption, Secrets Management, Auditability
Container Operations
Development LifecycleSource repo, CI-CD, Artefact repo
Container OrchestrationScheduling, Resource Management, Service Management
BAU OperationsMonitoring and Metrics, Maintenance, Debugging
Did you hear that?
In no particular order…
[ ] Schedulers and Orchestration[ ] Networking[ ] Security[ ] Operating Systems[ ] PaaS[ ] Storage[ ] Monitoring[ ] Container Integration and Container Deployment[ ] Miscellaneous
In no particular order…
[ ] Schedulers and Orchestration[ ] General Blurb[ ] ECS[ ] Kubernetes[ ] Mesos[ ] Docker Swarm[ ] Orchestration Wars
Schedulers – General Blurb
Cluster Machines
Cluster StateInformation
Monolothic Two-Level Shared State
No Concurrency Pessimistic Concurrency(offers)
OptimisticConcurrency
(transactions)
Scheduling Logic
DockerTask
Container Instance
Amazon ECS
Container
ECS Agent
ELB
Internet
ELB
User / Scheduler
API
Cluster Management Engine
TaskContainer
DockerTask
Container Instance
Container
ECS Agent
TaskContainer
DockerTask
Container Instance
Container
ECS Agent
TaskContainer
AZ 1 AZ 2
Key/Value Store
Agent Communication Service
ECS
MesosMaster
Marathon
ZooKeeper
Mesos + Marathon
Mesos Slaves
Long Running Tasks
Jobs
Coordination & Configuration
Kubernetes
Replication ControllerAPI Server
Kubernetes Master
Kubelet KubeProxy Docker
Container Container
Pod Pod
Kubelet KubeProxy Docker
Container Container
Pod Pod
Kubernetes Cluster
etcd
In no particular order…
[X] Schedulers and Orchestration[ ] Networking[ ] Security[ ] Operating Systems[ ] PaaS[ ] Storage[ ] Monitoring[ ] Container Integration and Container Deployment[ ] Miscellaneous
Container Networking
Dom 0
Instance Instance Instance
OS OS OS
Container Runtime Container Runtime Container Runtime
Container Container Container Container Container Container
Mode
Swarm Mode Manager
Swarm Mode Node
TLS CA
Load Balancing
Service Discovery
Distributed Store
Docker Engine
Libnetwork
Volumes
Plugins
Container Runtime
In no particular order…
[X] Schedulers and Orchestration[X] Networking[ ] Security[ ] Operating Systems[ ] PaaS[ ] Storage[ ] Monitoring[ ] Container Integration and Container Deployment[ ] Miscellaneous
Host Security
• Lock it down!• Namespaces and cgroups are your friends• Select few belong to docker UNIX group• SELinux is also your friend• Docker daemon runs as root!
Docker daemon security
• Do not run in privileged mode• Lock down inter container comms –icc=false• Secure APIs with TLS certificates
Whale-say
“If you run Docker on a server, it is recommended to run exclusively Docker in the server, and move all other services within containers controlled by Docker”
Container Image Security
• Use a small selection of trusted images• Scan your images
• CoreOS’s Clair scans Quay.io,• Docker Security Scanning works with Docker Trusted Registry
• Red Hat has built a new scanner in Project Atomic for its Atomic Registry.
• Other scanners are such as Aqua Peekr, Anchore, and Twistlock Trust work independently of specific registries
Lot more prescriptive advice here…
https://benchmarks.cisecurity.org/tools2/docker/CIS_Docker_1.6_Benchmark_v1.0.0.pdf
In no particular order…
[X] Schedulers and Orchestration[X] Networking[X] Security[ ] Operating Systems[ ] PaaS[ ] Storage[ ] Monitoring[ ] Container Integration and Container Deployment[ ] Miscellaneous
Micro OS
• CoreOS• RancherOS• Ubuntu Snappy• RedHat Atom• VmWare Photon• ECS Optimized Amazon Linux
RedHatAtomic
VmWarePhoton
UbuntuSnappy CoreOS RancherOS
395 MB
317 MB
215 MB
20 MB
150 MB
In no particular order…
[X] Schedulers and Orchestration[X] Networking[X] Security[X] Operating Systems[ ] PaaS[ ] Storage[ ] Monitoring[ ] Container Integration and Container Deployment[ ] Miscellaneous
Convox
$ convox apps create go-app$ convox deploy$ convox apps info go-app$ convox build --app go-app –d "Hello Build”$ convox releases promote RLYSUALSGCT$ convox ps$ convox scale main --count=2
Docker Data Center
Universal Control Plane (UCP)
Security Content Trust
Docker Trusted Registry
Orchestration Swarm
Container Runtime Engine
Operating System
In no particular order…
[X] Schedulers and Orchestration[X] Networking[X] Security[X] Operating Systems[X] PaaS[ ] Storage[ ] Monitoring[ ] Container Integration and Container Deployment[ ] Miscellaneous
In no particular order…
[X] Schedulers and Orchestration[X] Networking[X] Security[X] Operating Systems[X] PaaS[-] Storage[-] Monitoring[-] Container Integration and Container Deployment[-] Miscellaneous
Demoz
• Marathon scheduler on ECS (Credit : Ryosuke-san)• Convox• Docker Swarm• Weave Net and Weave Scope• ECS (ALB, Task AutoScaling, Task IAM Role)