a log message A day in the life of - Linux Foundation Events · A day in the life of a log message...
Transcript of a log message A day in the life of - Linux Foundation Events · A day in the life of a log message...
![Page 1: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/1.jpg)
A day in the life of a log messageKyle Liberti, Josef Karasek
@Pepe_CZ
![Page 2: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/2.jpg)
● Order is vital for scale
● Abstractions make systems manageable
![Page 3: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/3.jpg)
Problems of Distributed Systems
● Reliability
● Data throughput
● Latency
![Page 4: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/4.jpg)
Abstracted Tools
● Allow us to leverage complex systems with little work
● When things get too complicated, we add another layer of abstraction
● ….and we repeat this process
![Page 5: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/5.jpg)
OKD
● The Origin Community Distribution of Kubernetes that powers Red Hat OpenShift● Built around Kubernetes container cluster management, carries all k8s features, e.g.
○ Services, Pods, Controllers○ Readiness & Iiveness probes○ Persistent Volumes and Persistent Volume Claims○ Release versions correspond to k8s releases
● Complete open source container application platform○ Security and multi-tenancy
■ Access to namespaces per users/groups○ Container image registry and source-to-image builds○ CI/CD and devops workflows
![Page 6: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/6.jpg)
Kafka
● A publish/subscribe messaging system
● Scalable and reliable data delivery
● Integrates well with other systems
![Page 7: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/7.jpg)
Use CaseIntegrate distributed Logging with Kafka
Log sources Collector Data
Warehouse(Cluster)
Visualization
...Host
Logging SystemMessaging
System
Central data hub for
scalable log delivery
“Big Data” AnalysisArchival“Tailing”
Monitoring
Host
Log sources Collector
![Page 8: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/8.jpg)
Origin Aggregated Logging
● Part of OKD● Based on Elasticsearch, Fluentd and Kibana● Collecting distributed logs● Common data model● Security model - multi-tenancy● All open source
![Page 9: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/9.jpg)
Logging Architecture
Data Warehouse
(Cluster)Visualization
Host
...Host
Load Balancer
Logging System
Monitoring
Log files
Journal Collector
Appcontainers
/Infrastructure
logs Syslog
Log files
Journal Collector
Appcontainers
/Infrastructure
logs Syslog
![Page 10: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/10.jpg)
Container Logging
Container
Kubernetesmaster
stdoutcri-o Collector
(Fluentd)Files on
diskstderrsend off the node
kubeletkubectl logs <pod>
/var/log/containers/*.logSingle-linelogs
Node
![Page 11: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/11.jpg)
Loggy
● Loggy is a log message
● Needs to get to work safely and reliably
![Page 12: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/12.jpg)
![Page 13: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/13.jpg)
Log Collector
● Collects all container logs from node’s filesystem
● Ability to tag, filter and enrich logs for export
○ Add Kubernetes metadata to every log line
![Page 14: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/14.jpg)
![Page 15: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/15.jpg)
Kafka Connect Source
● Imports data from external systems into Kafka brokers
● Pluggable Connectors
● Rest Interface, Tasks and Workers
![Page 16: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/16.jpg)
![Page 17: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/17.jpg)
![Page 18: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/18.jpg)
Kafka Broker
● Like a post box
● Topics, partitions and the distribution of load
● Smart clients / dumb brokers
![Page 19: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/19.jpg)
![Page 20: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/20.jpg)
Broker Reliability
● Replication
● Partition Leadership
● Performance/Reliability trade offs
![Page 21: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/21.jpg)
![Page 22: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/22.jpg)
![Page 23: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/23.jpg)
Kafka Connect Sink
● Exports Kafka broker data to external systems
● Distributed mode
● Convenience of Connect Framework
![Page 24: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/24.jpg)
![Page 25: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/25.jpg)
![Page 26: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/26.jpg)
Strimzi Operators
● OKD/Kafka Integration
● Automates and manages Kafka deployment
● Operator Pattern
![Page 27: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/27.jpg)
Cluster Operator
● Manages clusters:
○ Kafka
○ Kafka Connect
○ Zookeeper
○ Mirror Maker
● Advanced integration features
![Page 28: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/28.jpg)
Entity Operator
● Manages different Kafka objects
● Currently consists of two Operators:
○ Topic Operator
○ User Operator
![Page 29: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/29.jpg)
Custom Resources
● Blueprint to describe Kafka cluster components
● Operators monitor these blueprints, matching cluster state with what is described
![Page 30: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/30.jpg)
apiVersion: kafka.strimzi.io/v1alpha1kind: Kafkametadata: name: my-clusterspec: kafka: replicas: 3 listeners: plain: {} tls: {} config: offsets.topic.replication.factor: 3 transaction.state.log.replication.factor: 3 transaction.state.log.min.isr: 2 storage: type: ephemeral zookeeper:
replicas: 3storage:
type: ephemeral entityOperator:
topicOperator: {}userOperator: {}
![Page 31: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/31.jpg)
![Page 32: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/32.jpg)
DEMO
![Page 33: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/33.jpg)
Image Credits
In order of appearance:
1. Maxmillion. (2013, January 4). NYC vs Boston [Digital image]. Retrieved October 15, 2018, from https://i.imgur.com/uffaY.jpg
![Page 34: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/34.jpg)
Content Credit
● Narkhede, N., Shapira, G., & Palino, T. (2017). Kafka - the definitive guide: Real-time data and stream processing at scale. Sebastopol, Kalifornien: O’Reilly.
● Strimzi authors. (2018, October 15). Strimzi Documentation (Master). Retrieved October 15, 2018, from http://strimzi.io/docs/master/
● OKD authors. (2018, October 15). OKD Latest Documentation. Retrieved October 15, 2018, from https://docs.okd.io/latest/welcome/index.html
![Page 35: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/35.jpg)
THANK YOU
![Page 36: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/36.jpg)
EXTRAS
![Page 37: a log message A day in the life of - Linux Foundation Events · A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ ... Kafka A publish/subscribe messaging system](https://reader030.fdocuments.net/reader030/viewer/2022040413/5f09ea107e708231d4291c15/html5/thumbnails/37.jpg)
Cluster-level logging multi-tenancy
Elasticsearch(Cluster)
ES service/route
Logging System - OpenShift PlatformFluentd
OpenShift ES plugin
SearchGuard plugin
Kibana container
oauth proxy container
OpenShiftOAuth
OpenShiftAPI
User project
and roles
Browser
Add token and userid
headers
Token and
userid headers
Kibana Pod