Sadayuki Furuhashi

Fluent

@frsyuki

e Event Collector Service

Treasure Data, Inc.

Structured logging

Pluggable architecture

Reliable forwarding

What’s Fluent?

It's like syslogd, but uses JSON for log messages

What’s Fluent?

“2011-04-01 host1 myapp: cmessage size=12MB user=me”

2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, “user”: “me”}

What’s Fluent?

2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, “user”: “me”}

time tag

record

Fluent vs Scribe

• Deals with structured logs• Easy to install> “gem install fluentd”> also apt-get and yum http://packages.treasure-data.com/

• Easy to customize> add/modify plugins without compile> “gem search -rd fluent-plugin”

Fluent vs Flume

• Easy to setup> “sudo fluentd --setup && fluentd”

• Compact> small engine (2,000 lines) + plugins> JVM-free

• Easy to configure

# receive events via HTTP<source> type http port 8888</source>

# read logs from file<source> type tail path /var/log/httpd.log format apache tag apache.access</source>

# save alerts to file<match alert.**> type file path /var/log/fluent/alerts</match>

# save access logs to MongoDB# and forward to other server<match apache.access> type copy <store> type mongo host 127.0.0.1 </store> <store> type forward <server> host 192.168.0.11 weight 20 </server> <server> host 192.168.0.12 weight 60 </server> </store></match>

Architecture

Input Buffer Output

HTTP+JSONFile tailSyslog...

MemoryFile

FileAmazon S3Fluent...

Pluggable Pluggable Pluggable

Architecture

Input

HTTP+JSONFile tailSyslog...

Pluggable

✓ Receive logs✓ Or pull logs from data sources✓ Non-blocking

Input plugins:

Architecture

Pluggable

✓ Improve performance✓ Provide reliability✓ Provide thread-safety

Buffer plugins:

Buffer

MemoryFile

Architecture

Pluggable

✓ Improve performance✓ Provide reliability✓ Provide thread-safety

Buffer plugins:

chunk

chunk

chunk output

Input

Architecture

Pluggable

✓ Write or send event logs

Output plugins:

Output

FileAmazon S3Fluent...

Log collection

FluentApps

Client library

Apache log

HTTP+JSON(in_http)

TCP+MessagePack(in_tcp)

follow growth of files (in_tail)

Client library

Client library

Ruby (by @frsyuki)Python (by @kzk_mover)PHP (by @chobi_e)Java (by @muga_nishizawa)C++ (by @nobu_k)...

Fluent.open(“myapp”, “localhost”)Fluent.event(“login”, {“user”=>38})#=> 2011-02-03 04:56:01 myapp.login {“user”:38}

Forwarding

Fluent

FluentApps

FluentApps

FluentApps

HA Forwarding

Fluent

Fluent

FluentApps

FluentApps

FluentApps

Active

Active

✓ load balancing✓ continuous heartbeat✓φ accrual failure detector

External filter program

FluentFluent

Externalprogram

・stdin / stdout・TSV

External output program

FluentFluent

Externalprogram

Externalprogram

・TSV file

External input program

Fluent

Externalprogram

Externalprogram

Externalprogram

receiveor pull

・stdout・TSV

Incremental Aggregationby @doryokujin #fluent_meetup2

FluentApps MongoDB

/ Redis

count[“2011-01-01 01h”.”/index”] += 1count[“2011-01-01 01h”.”/index”] += 1count[“2011-01-01 01h”.”/landing”] += 1count[“2011-01-01 02h”.”/index”] += 1...

count[“2011-01-01 01h”.”/index”] == 42count[“2011-01-01 01h”.”/landing”] == 38count[“2011-01-01 02h”.”/index”] == 55count[“2011-01-01 02h”.”/landing”] == 55

FluentApps

MySQL replicator (plan)

FluentMySQL

read binlog continuouslyusing MySQL binlog API

mysql://127.0.0.1

http://www.slideshare.net/mkindahl/binary-log-api-presentation-oscon-2011

3rd Party Plugins

• Scribe input/output (by @kzk_mover)> receive logs sent from scribed

• Amazon S3 output (by @frsyuki)> upload log files to Amazon S3

• MongoDB output (by @repeatedly)> write structured logs to MongoDB

• Redis output (by @yuki24)> write structured logs to Redis

3rd Party Plugins

• Cassandra output (by @railute)> write structured logs to Cassandra

• Growl output (by @goth_writst_cut)> notify messages using Growl(Growl: notification service for Mac OS X)

• Aggregation (by @doryokujin)> incremental aggregation using NoSQL DBs

• ...

http://fluentd.org/plugin/> gem search -dr fluent-plugin

Plugin Developer APIs

• Unit test framework (like “MRUnit”)> Fluent::Test::InputTestDriver> Fluent::Test::OutputTestDriver> Fluent::Test::BufferedOutputTestDriver

• Fluent::BasicTail (base class of “tail” plugin)> text parser is customizable def parse_line(line)

What’s Fluent?

• Event collector service> deals with structured event logs

• Compact> small engine (2,000 lines) + plugins

• Plugin archtecture> install plugins using RubyGems

http://github.com/!uent

Demo