Ninefold's Logging System Technology Stack
27
ELASTICSEARCH, LOGGING, AND NINEFOLD
-
Upload
ryan-clark -
Category
Technology
-
view
209 -
download
0
description
These are the slides from a talk I gave at the San Francisco Rails meetup at Crowdflower on June 19, 2014. I talked about Elasticsearch(a little), logstash, load balancing, high availability,processing millions of logs for multiple large-scale productoin environments, and lessons we learned implementing this system.
Transcript of Ninefold's Logging System Technology Stack
ELASTICSEARCH,LOGGING,ANDNINEFOLD
Hi,I'mRyan
LeadRailsSupportEngineeratNinefold
WHATWE'LLCOVER
Ninefold'sstack(ELK,sorta)ArchitectingthesystemGettingthelogsoutLessonslearned
LOGGING:
UNTILIT'SNOT!
WHATISELASTICSEARCH?
HighlyscalableAuto-clusteringAuto-shardingAuto-indexingAuto-magical!
ApacheLuceneunderthecovers
WHYELASTICSEARCH?
SeepreviousslideEasytosetupEasytoscale
ReadilyavailableresourcesonthewebSupersimpletounderstandwhat'sgoingon
Lotsoftools/integrations/plugins
INSTALLINGANDRUNNING
$curl-L-Ohttp://download.elasticsearch.org/PATH/TO/LATEST/$VERSION.zip$unzipelasticsearch-$VERSION.zip
$cdelasticsearch-$VERSION
$bin/elasticsearch#-dtodaemonize$curl'http://localhost:9200/?pretty'
OURSUPPORTINGROLES:
Beaver(https://github.com/josegonzalez/beaver)LogstashRabbitMQ
ALITTLEBITABOUTBEAVER:
WritteninPythonRunsasadaemonontheclient
Canlistentoandsendasmany/fewlogsasyouwantCandosomeoralllogformatting
FormatsandtransmitslogsonelineatatimetothequeueinJSON
Attachesmetadataforlater
MOREONLOGSTASH:
PullslogsfromthequeueFormatslogsagainandpushesthemtoElasticsearch
Groktoeasilyformatmessages
LOGSTASHGROK:
...{#findcheckpoint,turnitintoafieldandadditasatag:grok=>{:tags=>['cheflog'],:pattern=>"#{node[:ninefold_logger][:checkpoint_tag]}:%{WORD:checkpoint_type}",:tag_on_failure=>false,:add_tag=>"#{node[:ninefold_logger][:checkpoint_tag]}%{checkpoint_type}",:singles=>true},...
ABOUTRABBITMQ:
FastAMQPMessagingHighAvailability
Abilitytoaddnodestoexistingcluster
THEIDEA:
THEIMPLEMENTATION:
Uhh..
OK...
GIMMEMYLOGS!
LogEntryModel:
Findandsortthem:
DEMOTIME!!
WHATWE'VELEARNEDMulti-linelogsareachallengeMultipleinstancesofLogstashcancomplicatethingsLocalstorage>networkstorageDon'tuseglobs-bespecific.Elasticsearchisawesome!
