Fluentd - Flexible, Stable, Scalable
-
Upload
shu-ting-tseng -
Category
Data & Analytics
-
view
158 -
download
0
Transcript of Fluentd - Flexible, Stable, Scalable
What is Fluentd?
• Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
• Treasure Data: td-agent
What is Fluentd?
• Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
• Treasure Data: td-agent
input: tail$ cat /etc/td-‐agent/conf.d
<source>
type tail
path /var/log/nginx/access.log
pos_file /var/log/td-‐agent/httpd-‐access.log.pos
tag nginx.access
</source>
<match nginx.access>
blah blah
</match>
input: forward$ cat /etc/td-‐agent/conf.d
<source>
type forward
port 24224
</source>
<match flask.index>
blah blah
</match>
input: forward$ cat ~/example.py
from fluent import sender
from fluent import event
sender.setup('flask', host='localhost', port=24224)
event.Event("index", {
"user": "foo",
"token": "bar",
"action": "POST"
})
Output plugins
• forward
• copy
• Elasticsearch / MongoDB
• statsd / influxDB / graphite
• S3 / GCS / BigQuery
output: elasticsearch$ cat /etc/td-‐agent/conf.d
<source>
foo bar
tag nginx.access
</source>
<match nginx.access>
type elasticsearch
hosts es-‐host1,es-‐host2
index_name nginx
type_name access
flush_interval 60s
</match>
output: splunk$ cat /etc/td-‐agent/conf.d
<source>
foo bar
tag nginx.access
</source>
<match nginx.access>
type splunk
hosts splunk-‐host1
</match>
Buffer example$ cat /etc/td-‐agent/conf.d
<source>
foo bar
tag nginx.access
</source>
<match nginx.access>
type splunk
hosts splunk-‐host1
buffer_chunk_limit 10m
buffer_queue_limit 1000
flush_interval 5m
</match>
App + Fluentd
Fluentd
Elastic search
Elastic search
Elastic search
Elastic search
App + Fluentd
App + Fluentd
Fluentd
Elastic search
Elastic search
Elastic search
Elastic search
Fluentd
App + Fluentd
App + Fluentd
App + Fluentd
Fluentd
Elastic search
Elastic search
Elastic search
Elastic search
Fluentd
FluentdLoad
balance
App + Fluentd
App + Fluentd
App + Fluentd
Auto scaling group
Message Delivery
• At-most-once: data may be lost
• At-least-once: data may be duplicated
• Exactly-once: perfect
Idempotent
• HTTP PUT
• Maintain a unique id in application level or
• Concatenate (instance-id, time, ….) as id