2016 10-26 docker meetup - kubernetes on open stack
-
Upload
amrita-prasad -
Category
Technology
-
view
281 -
download
2
Transcript of 2016 10-26 docker meetup - kubernetes on open stack
![Page 1: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/1.jpg)
Mario Siegenthaler
11th Docker Switzerland User Group, 26.10.2016
a field report
Ferdinand Hübner
Kubernetes on OpenStack
![Page 2: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/2.jpg)
About Us
450
ops & dev
...
Solutions for the
Government
(Cantons)
![Page 3: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/3.jpg)
450
ops & dev
About Us
450
...
Ferdinand Hübner
Mario Siegenthaler
CTO development
Solutions for the
Government
(Cantons)
![Page 4: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/4.jpg)
Project Goals
Testing Automated
Exploration
(Production)
Continuous Delivery
User Acceptance
![Page 5: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/5.jpg)
on
premise
production
quality
Requirements
easy setup
+ persistent data
+ “useful” stability
![Page 6: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/6.jpg)
Technology Evaluation
Quelle: classroomclipart.com
kubernetes
Infrastructure
Plattform
Application
packaging
![Page 7: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/7.jpg)
Classification of Applications
![Page 8: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/8.jpg)
Analysis - Dimensions
storage
communication scalability
![Page 9: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/9.jpg)
Analysis - Storage
storage cheap
fast
durable volatile
large small application-level
replicated
storage-level
replicated
![Page 10: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/10.jpg)
Analysis - Communication
communication
load
balancing
security
external
access
![Page 11: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/11.jpg)
Analysis - Scalability
scalability auto
scale
multi
AZ
vertical
only
heterogenous
(Master/Slave)
stateless
stateful
![Page 12: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/12.jpg)
Excluded Functions
storage
communication scalability huge scale
auto scale multi AZ
redundancy
backup
fast storage dynamic scaling of
persistent replicas
![Page 13: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/13.jpg)
IMPLEMENTATION
![Page 14: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/14.jpg)
Implementation goals
Figure out what makes kubernetes tick
Consider AuthN and AuthZ
Cluster creation: fast, easy, automated
![Page 15: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/15.jpg)
AuthN
● We chose certificates over tokens
● Password files were never an option
● Create simple and easy to use PKI tools
![Page 16: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/16.jpg)
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
![Page 17: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/17.jpg)
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
● If you’re authenticated, you can do everything or nothing
![Page 18: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/18.jpg)
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
● Attribute Based Access Control
● Cumbersome - requires API server restart
![Page 19: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/19.jpg)
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
● Roles Based Access Control
● In Alpha
● Kubernetes API Objects
![Page 20: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/20.jpg)
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
● Delegate to a webservice
● Implement a bridge to your corporate directory
![Page 21: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/21.jpg)
AuthZ: Our choice
Always
● Freedom and responsibility for developers
● Separate teams with multiple clusters
![Page 22: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/22.jpg)
Cluster overview
![Page 23: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/23.jpg)
Ingress: Layer 4 and Layer 7
● Expose services easily with DNS and HTTPS
● Existing implementations are written in go
● LBaaS missing on our OpenStack
● We can’t use Let’s Encrypt
● Ingress spec doesn’t cover Layer 4
![Page 24: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/24.jpg)
Ingress: Layer 4 and Layer 7
● Implementation: Java, Akka, HAProxy
● Assign services to ingress nodes
● Register and refresh DNS in SkyDNS cluster
● Wildcard SSL certificate
● HAProxy targets service endpoints
![Page 25: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/25.jpg)
Ingress: Layer 4 and Layer 7
Custom with drawbacks, but good enough for now
kind: Service metadata: annotations: bedag.ch/ingress-bedag: mysvc.xy.bedag.ch bedag.ch/ingress-mode: https
![Page 26: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/26.jpg)
Ingress: Next steps
Implement:
● Service Load-Balancer
● Ingress
Or:
● Use existing implementations
● Write addons
![Page 27: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/27.jpg)
Persistent Volumes
● Plugin-Based architecture
● Started with NFS on K8S
● Moved to Ceph
● Reference Volumes by ID
kubernetes
Ceph
cinder
NFS
![Page 28: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/28.jpg)
Ceph Cluster
● Traditional disks on OpenStack nodes
● No special tuning
● Sequential Read/Write is OK
● Write-Latency is awful
![Page 29: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/29.jpg)
etcd WAL fsync duration
local SSDs
![Page 30: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/30.jpg)
etcd WAL fsync duration
local SSDs
our Ceph
![Page 31: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/31.jpg)
CREATING CLUSTERS
![Page 32: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/32.jpg)
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
![Page 33: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/33.jpg)
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
● (Our own) documentation gets outdated and fuzzy
![Page 34: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/34.jpg)
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
● (Our own) documentation gets outdated and fuzzy
Infrastructure as code
![Page 35: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/35.jpg)
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
● (Our own) documentation gets outdated and fuzzy
Infrastructure as code
● We don’t want snowflakes in our datacenter
![Page 36: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/36.jpg)
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
● (Our own) documentation gets outdated and fuzzy
Infrastructure as code
● We don’t want snowflakes in our datacenter
Immutable servers
![Page 37: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/37.jpg)
OS choice
“CoreOS Linux is the leading container operating system,
designed to be managed and run at massive scale, with
minimal operational overhead.“
-- CoreOS, Inc
![Page 38: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/38.jpg)
(Our pick at) Infrastructure as code
define a server entirely
through its cloud-config.yaml
by writing code that
templates it
![Page 39: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/39.jpg)
Immutable servers
Throw away servers that
● need a configuration change
● need updating
● misbehave
![Page 40: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/40.jpg)
Creating clusters and servers
cluster name and server role
associate ips, create certificates, create storage
template cloud-config.yaml
boot server, attach storage
![Page 41: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/41.jpg)
![Page 42: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/42.jpg)
Logging, Monitoring
● fluentd daemon set
● ships to graylog
● not using docker’s native
GELF output
● prometheus
● grafana
![Page 43: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/43.jpg)
Kubernetes configuration files
Configuration files in git repositories
● Changes can be traced
● Easy to deploy to other clusters
● Deployment can be automated
![Page 44: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/44.jpg)
Conclusion
great, powerful and stable!
perfect for prototyping and testing
storage can be a challenge
![Page 45: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/45.jpg)
QUESTIONS
![Page 46: 2016 10-26 docker meetup - kubernetes on open stack](https://reader033.fdocuments.net/reader033/viewer/2022042723/5876e6321a28ab046d8b6061/html5/thumbnails/46.jpg)
Contact Us
linkyard.ch
Mario Siegenthaler
Consulting Software Architecture & DevOps
Partner
Ferdinand Hübner
CTO Software Development
www.bedag.ch