KOWAZA for mackerel

36
KOWAZA for Mackerel id:karupanerura Mackerel Meetup #7 LT

Transcript of KOWAZA for mackerel

Page 1: KOWAZA for mackerel

KOWAZA for Mackerelid:karupanerura Mackerel Meetup #7 LT

Page 2: KOWAZA for mackerel

KOWAZA means small trick in Japanese

Page 3: KOWAZA for mackerel

about me

• id:karupanerura (twitter/github)

• Perl/Go/Swift/Kotlin/Java/JS/etc..

• Senior Engineer at Mobile Factoy, Inc.

• Join to Infrastructure Team since 2016/02

• Director at Japan Perl Association (JPA)

Page 4: KOWAZA for mackerel

mackerel with me

• Mackerel contributer

• mackerel-agent-plugins (gearman)

• mackerel-check—plugins (masterha, ssh)

• Mackerel UG Organizer ← NEW

Page 5: KOWAZA for mackerel

monitor processes

Page 6: KOWAZA for mackerel

check-procs -p mysqld

Page 7: KOWAZA for mackerel

Wait…

Page 8: KOWAZA for mackerel

(In my environment) MySQL is stopped

to backup at 5am-6am.

Page 9: KOWAZA for mackerel

🤔

Page 10: KOWAZA for mackerel

true && echo “true” || echo “false”

Page 11: KOWAZA for mackerel

😙

Page 12: KOWAZA for mackerel

[ 5 -le “$hour” -and “$hour” -le 6 ] && echo “OK check-procs: In shutdown” || check-procs -p mysqld

Page 13: KOWAZA for mackerel

😎

Page 14: KOWAZA for mackerel

monitor logs

Page 15: KOWAZA for mackerel

grep foo /path/to/log | grep bar | grep -v baz | wc -l

Page 16: KOWAZA for mackerel

check-log plugin

Page 17: KOWAZA for mackerel

check-log -f /path/to/log -p foo -p bar -E baz -c 1

Page 18: KOWAZA for mackerel

Wait…

Page 19: KOWAZA for mackerel

check-log -f /path/to/log -p foo -p bar ←Cannot do it :( -E baz -c 1

Page 20: KOWAZA for mackerel

🤔

Page 21: KOWAZA for mackerel

check-log -f <(grep foo /path/to/log) -p bar -E baz -c 1

Page 22: KOWAZA for mackerel

😙

Page 23: KOWAZA for mackerel

grep foo /path/to/log | check-log -f /dev/stdin -p bar -E baz -c 1

Page 24: KOWAZA for mackerel

😎

Page 25: KOWAZA for mackerel

monitor mouts

Page 26: KOWAZA for mackerel

Should mount a HDD as readable & writable

Page 27: KOWAZA for mackerel

(If mount with errors=remout-ro) When the HDD is breaking, re-mount the HDD readonly

Page 28: KOWAZA for mackerel

monitor with Mackerel

Page 29: KOWAZA for mackerel

# XXX: exit code 2 is CRITICAL touch /tmp/write-test.watch || exit 2

Page 30: KOWAZA for mackerel

But, hackish…

Page 31: KOWAZA for mackerel

no kind message outputs…

Page 32: KOWAZA for mackerel

🤔

Page 33: KOWAZA for mackerel

check-log -f /proc/mounts -p ' ro(,|$)’ --no-state

Page 34: KOWAZA for mackerel

😙

Page 35: KOWAZA for mackerel

Let’s hack Mackerel !!

Page 36: KOWAZA for mackerel

😎