1. Logitoring :log-driven monitoringthe Rocket scienceEugene IstominIT Architecte.istomin@edss.eeCone Center,Tallinnand

2. Topic goal:talking about a common way ofdelivering, storing and analyzingmonitoring/log/trace data flows.In brief:Does log-driven monitoringfits all needs? 3. 4TOCMetrics in monitoring and loggingBy log-driven monitoring todataflow -driven services.Rsyslog event transportLet's try. Live test case 4. Monitoring ~ protocol/agent,desired data descr, centralizedstorage, notifications = Reactive5Metrics in monitoring and loggingIT Monitoring - sum of methods used tocollect defined metrics using checks.What is the classicalmonitoring metric?Numeric! (int/bool/etc) 5. 6Metrics in monitoring and loggingClassical monitoring interfaces Zabbix Nagios 6. 7Metrics in monitoring and loggingMonitoring is usually used for: Servers status dashboard creation IT-administrators notification Numeric info visualization IT-inventoryMonitoring In brief: Schema-based Use common networkprotocols or agents Stored data not reusable Needs by IT technics/admins 7. Logs ~ syslog, pid/severity/program, transport, centralizedstorage. = Proactive8Metrics in monitoring and loggingIT Logging - sum of methods used tocollect pass-through flows information.What is the classicallogs metric?String! 8. 9Metrics in monitoring and loggingClassical logging interfaces Loganalyzer Greylog 9. 10Metrics in monitoring and loggingLogging is usually used for: Problem resolving Debugging & developmentLogging In brief: Security access violationevents storage Schema-less Use syslog or API/REST Stored data are reusable Needs by developers 10. 11Metrics in monitoring and loggingDo the data in monitoring is the sameas data in logging? Yes, if: the data is well tokenized (known keys)A=2, TO=me@z.com... the data have a common syntax(JSON/CSV){ A:2, TO:[me@z.com,...] } 11. 12Metrics in monitoring and loggingDo the data in monitoring is the sameas data in logging? Yes, if: the data have known type mapping(field TO = string, field ID = int){ A:int, TO:array } storage layer use efficienttoken/key/hash-based algorithmsme@z.com=>me,@,z,.,com 12. 13Metrics in monitoring and loggingDo the data in monitoring is the sameas data in logging? Yes, if: user UI/API can ask for mixedfields content using complexexpressions(regexp/ranges/sorts)"query_string" : {"fields" : ["TO.*"],"query" : "a AND com OR z"} 13. 14Metrics in monitoring and loggingDo the data in monitoring have the samenature as data in logging? Yes, Monitoring and logging aresubsets of events Monitoring is mainly reactive Logging is mainly proactiveEvents is a set includes allpossible types of messages(monitoring, logging, JSON dataexchange by HTTP or TCP, etc) 14. 15TOCMetrics in monitoring and loggingBy log-driven monitoring to dataflow -drivenservices.Rsyslog event transportLet's try. Live test case 15. 16Log-driven monitoringLog-driven monitoring -a sum of techniques that providesaccess to logs events fields fromhigh-level decision-makerapplication using complexexpressions.1. { haproxy_status=503},... { jmx_app1_status=500 }2. "query_string" : {"fields" : ["haproxy_status",jmx_*_status],"query" : ">=500"}3. action notify admins 16. 17Log-driven monitoringLLD is superb 17. 18Log-driven monitoringLLD is superbItems are created byJSON POST usingexernal CMDBdatabaseAll items aretrappers 18. 19Log-driven monitoringFast-as-light Erlang Zabbix senderSystemd-basedservice 19. 20Log-driven monitoringBut whats about datastore? 20. 21Log-driven monitoringmails count per second in normal & percent grade 21. 22Log-driven monitoringderivative of mails count per second coerced tospamscore SUM & MAX 22. 23TOCMetrics in monitoring and loggingBy log-driven monitoring todataflow -driven services.Rsyslog event transportLet's try. Live test case 23. 24Rsyslog event transportRsyslog templates for CEE logging CEE (LumberJack) JSON messages timestamp field in ES format (us) 24. 25Rsyslog event transportRulesets & actionsAll logs must been tokenized: Before Rsyslog (in application) Using Rsyslog (mmnormalize) 25. 26Rsyslog event transportRulesets & actions Every ruleset have own queue Queues are disk-backed (watermark-based) Shutdowns and restarts are safe 26. 27Talk is cheap. Show me the code. 27. 28TOCMetrics in monitoring and loggingBy log-driven monitoring todataflow -driven services.Rsyslog event transportLet's try. Live test case 28. Show accepted mails, 2.2Km, time to retrieve 92 ms29Let's try. Live test caseApplication MTA + spamfilter, scope-12hShow all msg, ~1.7Mm, time to retrieve 400 ms 29. 30Let's try. Live test caseApplication HTTP(S) gate, scope-12h (08:00-18:00)Show all HTTPS traffic per hour- ~1.9Mm,- time to retrieve 210 ms 30. 31Let's try. Live test caseApplication All app-specific logs (no per-field tokenizing)- ~4Mm,- time to retrieve- 8 s 31. Mail your CV:info@cone.eeTalk is cheap. Show me the code. L.TorvaldsEugene IstominIT Architecte.istomin@edss.eeCone Center,TallinnThanks!