How we use Fluentd in Treasure Data
-
Upload
sadayuki-furuhashi -
Category
Technology
-
view
761 -
download
1
description
Transcript of How we use Fluentd in Treasure Data
Sadayuki FuruhashiTreasuare Data, Inc.Founder & Software Architect
How we use Fluentdin Treasure Datafor system metrics
A creator of Fluentd
Self-introduction
> Sadayuki Furuhashi> twitter/github: frsyuki
> Treasure Data, Inc.Founder & Software Architect
> Open source projectsMessagePack - efficient serializer (original author)
Fluentd - event collector (original author)
Metrics collection from Rails apps
fluentdRails app
fluentd
fluentdRails app
MetricSense for Ruby
aggregate logs usingout_forward plugin
local fluentd collects logsand buffer them in local disk
class ApplicationController before_filter :setup_metrics after_filter :post_metrics
def setup_metrics MetricSense.clear! MetricSense.tag "#{params[:controller]}_#{params[:action]}" if current_user.present? MetricSense.segment( :account=>current_user.account_id, :agent=>request.env[‘HTTP_AGENT’]) end end
def post_metrics MetricSense.measure! end
# in application code: # MetricSense.value :size=>10
Metrics collection from Hadoop
fluentd
HadoopCluster
HadoopClusterhadoop metrics plugin
(original plugin)
✓ number of failed tasks✓ number of used slots✓ blacklisted nodes✓ memory usage of JobTracker✓ memory usage of NameNode/DataNode✓ HDFS put/get latencyetc...
Metrics collection from DB
fluentd
script in_exec pluginruns a script every1 hour
runs SQL statements
✓ users’ activity and stats✓ job queue length (PerfectQueue stats)etc...
Metrics collection from Hadoop
fluentdRails app
fluentdRails app
fluentd
HadoopJobTracker
HadoopJobTracker
script
Metrics utilization
fluentd
Librato Metricsfor realtime analysis
Treasure Datafor historical analysis
Pager Dutyfor alerts
out_tdlogout_metricsense
Next version of Fluentd
> <filter>> <match> in <source>> <label> for error records> New process model & Live restart
<source> type tail path /var/log/httpd.log format apache tag not_filtered.apache</source>
<match not_filetered.**> type rewrite remove_prefix not_filtered <rule> key status pattern ^500$ ignore true </rule></match>
<match **> type forward host log.server</match>
Before
Mysterious tag
tag operations
<source> type tail path /var/log/httpd.log format apache tag apache</source>
<filter **> type rewrite <rule> key status pattern ^500$ ignore true </rule></filter>
<match **> type forward host log.server</match>
After (v11)
Filter plugins!
<source> type tail path /var/log/httpd.log format apache tag apache
<filter **> type rewrite <rule> key status pattern ^500$ ignore true </rule> </filter></source>
<match **> type forward host log.server</match>
After (v11)
<filter>/<match> in <source>
<source> type tail path /var/log/httpd.log tag apache</source>
<match **> type forward host log.server</match>
Before
I want to add flowcounter here...
<source> type tail path /var/log/httpd.log tag apache</source>
<match flow.traffic> type forward host traffic.server</match>
<match **> type copy <store> type flowcounter tag flow.traffic </store>
<store> type forward host log.server </store></match>
Before
Nested!
<source> type tail path /var/log/httpd.log tag apache</source>
<filter **> type copy <match> type flowcounter tag flow.traffic <match> type forward host traffic.server </match> </match></filter>
<match **> type forward host log.server</match>
After (v11)
Filtering pipeline
<source> type forward</source>
<filter **> type copy <match> type file path /mnt/local_archive </match></filter>
<label alert> <match **> ... </match></label>
<label analysis> ...</label>
# copy & label & forward<filter **> type copy <match> type forward label alert host alerting.server </match></filter>
# copy & label & forward<filter **> type copy <match> type forward label analysis host analysis.server </match></filter>
After (v11)