Fluentdでログ収集「だけ」やる話What to do is to collect raw logs

#study2study 2012/03/29@tagomoris

12年3月30日金曜日

@tagomoris

NHN Japan Corp.Web Service Business DivisionDevelopment Department 2

Development Platforms

12年3月30日金曜日

Fluentd

• 'That' we all love.

12年3月30日金曜日

Basic method• Fluentd in_tail + out/in_forward + out_file

Web server

Web server

Web server

fluentd

fluentd

fluentd

Collect server

fluentd

12年3月30日金曜日

in_tail + out_file: JSON

20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031", ...}20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031", ...}20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031", ...}

• JSON is good for new systems

• But traditional (historically important) systems requires raw logs

• How do we get raw logs with Fluentd ?

12年3月30日金曜日

we neeeeeed raw log• fluent-agent-lite

• Fluentd in_forward + out_file_alternative

Web server

Web server

Web server

fluent-agent-liteCollect server

fluentd

fluent-agent-lite

fluent-agent-lite

12年3月30日金曜日

fluent-agent-lite• https://github.com/tagomoris/fluent-agent-lite

• easy to intall, run

• run with perl 5.8 (only!)

• bin/install.sh or rpm (.spec bundled)

• start-stop script bundled

• lightweight, raw log intensive

• read a line, send it as one msg with one field

• with high throughput and low load12年3月30日金曜日

fluent-agent-lite

• many features

• fluentd cluster support (select one randomly)

• failover support (primary / secondary servers)

• configurations for high performance

12年3月30日金曜日

fluent-plugin-file-alternative

• 100% compatible to standard out_file

• with same configuration, get same output

• simple configuration

• path w/ time placeholder w/o time_slice_format

• /path/to/log/access.%Y-%m-%d.log

12年3月30日金曜日

fluent-plugin-file-alternative• custom output line formats

• time of fluentd message OR NOT

• tag of fluentd message OR NOT

• custom output data formats

• whole record as json

• selected single field

• selected multi fields seperated by TAB/SPACE/COMMA

12年3月30日金曜日

fluent-plugin-file-alternative

• Misc

• add newline at the end of data OR NOT

• appending data to single file with flush_interval (without compression)

12年3月30日金曜日

simple configuration

<match fluentd.status.gc.**> type file_alternative path /var/log/fluentd.status/gc.%Y-%m-%d.%H.log</match>

<match fluentd.status.gc.**> type file path /var/log/fluentd.status/gc time_slice_format %Y-%m-%d.%H</match>

12年3月30日金曜日

for raw logs

<match apache.log.**> type file_alternative path /var/log/archive/access.%Y-%m-%d.%H.log output_include_time false output_include_tag false output_data_type attr:message add_newline true</match>

12年3月30日金曜日

Thanks!

12年3月30日金曜日