Administration Guide - support-public.cfm.quest.com · Introduction to syslog-ng 30 What syslog-ng...

syslog-ngOpenSourceEdition3.16

AdministrationGuide

Copyright 2018 One Identity LLC.

ALL RIGHTS RESERVED.

Thisguidecontainsproprietaryinformationprotectedbycopyright.Thesoftwaredescribedinthisguideisfurnishedunderasoftwarelicenseornondisclosureagreement.Thissoftwaremaybeusedorcopiedonlyinaccordancewiththetermsoftheapplicableagreement.Nopartofthisguidemaybereproducedortransmittedinanyformorbyanymeans,electronicormechanical,includingphotocopyingandrecordingforanypurposeotherthanthepurchaserspersonalusewithoutthewrittenpermissionofOneIdentityLLC.TheinformationinthisdocumentisprovidedinconnectionwithOneIdentityproducts.Nolicense,expressorimplied,byestoppelorotherwise,toanyintellectualpropertyrightisgrantedbythisdocumentorinconnectionwiththesaleofOneIdentityLLCproducts.EXCEPTASSETFORTHINTHETERMSANDCONDITIONSASSPECIFIEDINTHELICENSEAGREEMENTFORTHISPRODUCT,ONEIDENTITYASSUMESNOLIABILITYWHATSOEVERANDDISCLAIMSANYEXPRESS,IMPLIEDORSTATUTORYWARRANTYRELATINGTOITSPRODUCTSINCLUDING,BUTNOTLIMITEDTO,THEIMPLIEDWARRANTYOFMERCHANTABILITY,FITNESSFORAPARTICULARPURPOSE,ORNON-INFRINGEMENT.INNOEVENTSHALLONEIDENTITYBELIABLEFORANYDIRECT,INDIRECT,CONSEQUENTIAL,PUNITIVE,SPECIALORINCIDENTALDAMAGES(INCLUDING,WITHOUTLIMITATION,DAMAGESFORLOSSOFPROFITS,BUSINESSINTERRUPTIONORLOSSOFINFORMATION)ARISINGOUTOFTHEUSEORINABILITYTOUSETHISDOCUMENT,EVENIFONEIDENTITYHASBEENADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGES.OneIdentitymakesnorepresentationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisdocumentandreservestherighttomakechangestospecificationsandproductdescriptionsatanytimewithoutnotice.OneIdentitydoesnotmakeanycommitmenttoupdatetheinformationcontainedinthisdocument.Ifyouhaveanyquestionsregardingyourpotentialuseofthismaterial,contact:

OneIdentityLLC.Attn:LEGALDept4PolarisWayAlisoViejo,CA92656

RefertoourWebsite(http://www.OneIdentity.com)forregionalandinternationalofficeinformation.

Patents

OneIdentityisproudofouradvancedtechnology.Patentsandpendingpatentsmayapplytothisproduct.Forthemostcurrentinformationaboutapplicablepatentsforthisproduct,pleasevisitourwebsiteathttp://www.OneIdentity.com/legal/patents.aspx.

Trademarks

OneIdentityandtheOneIdentitylogoaretrademarksandregisteredtrademarksofOneIdentityLLC.intheU.S.A.andothercountries.ForacompletelistofOneIdentitytrademarks,pleasevisitourwebsiteatwww.OneIdentity.com/legal.Allothertrademarksarethepropertyoftheirrespectiveowners.

Legend

WARNING: A WARNING icon indicates a potential for property damage, personal injury, or death.

CAUTION: A CAUTION icon indicates potential damage to hardware or loss of data if instructions are not followed.

IMPORTANT,NOTE,TIP,MOBILE,orVIDEO:Aninformationiconindicatessupportinginformation.

syslog-ngOSEAdministrationGuideUpdated-July2018Version-3.16

http://www.oneidentity.com/http://www.oneidentity.com/legal/patents.aspxhttp://www.oneidentity.com/legal

Contents

Preface 18

Summaryofcontents 18

Targetaudienceandprerequisites 19

Productscoveredinthisguide 19

Summaryofchanges 20

Version3.15-3.16 20

Version3.14-3.15 20

Version3.13-3.14 21

Version3.12-3.13 21

Version3.11-3.12 22

Version3.10-3.11 23

Version3.9-3.10 24

Version3.8-3.9 25

Version3.7-3.8 25

Version3.6-3.7 26

Version3.5-3.6 28

Feedback 29

Acknowledgments 29

Introduction to syslog-ng 30

Whatsyslog-ngis 30

Secureandreliablelogtransfer 30

Flexibledataextractionandprocessing 31

Bigdataclusters 31

Messagequeuesupport 31

SQL,NoSQL,andmonitoring 32

Wideprotocolandplatformsupport 32

Whatsyslog-ngisnot 32

Whyissyslog-ngneeded? 32

Whatisnewinsyslog-ngOpenSourceEdition3.16? 33

Whousessyslog-ng? 34

Supportedplatforms 34

syslog-ng OSE 3.16 Administration Guide 3

The concepts of syslog-ng 35

Thephilosophyofsyslog-ng 35

Loggingwithsyslog-ng 35

Therouteofalogmessageinsyslog-ng 36

Modesofoperation 37

Clientmode 37

Relaymode 38

Servermode 38

Globalobjects 39

Timezonesanddaylightsaving 40

Howsyslog-ngOSEassignstimezonetothemessage 41

Anoteontimezonesandtimestamps 42

Productlicensing 42

Highavailabilitysupport 42

Thestructureofalogmessage 42

BSD-syslogorlegacy-syslogmessages 43

ThePRImessagepart 43

TheHEADERmessagepart 45

TheMSGmessagepart 45

IETF-syslogmessages 45

ThePRImessagepart 46

TheHEADERmessagepart 47

TheSTRUCTURED-DATAmessagepart 48

TheMSGmessagepart 48

Enterprise-widemessagemodel(EWMM) 48

Messagerepresentationinsyslog-ngOSE 49

Structuringmacros,metadata,andothervalue-pairs 51

Specifyingdatatypesinvalue-pairs 52

value-pairs() 53

Thingstoconsiderwhenforwardingmessagesbetweensyslog-ngOSEhosts 57

Commercialversionofsyslog-ng 59

Installing syslog-ng 62

Compilingsyslog-ngfromsource 62

Compilingoptionsofsyslog-ngOSE 64

Uninstallingsyslog-ngOSE 67


ConfiguringMicrosoftSQLServertoacceptlogsfromsyslog-ng 67

The syslog-ng OSE quick-start guide 74

Configuringsyslog-ngonclienthosts 74

Configuringsyslog-ngonserverhosts 77

Configuringsyslog-ngrelays 79

Configuringsyslog-ngonrelayhosts 79

Howrelayinglogmessagesworks 81

The syslog-ng OSE configuration file 83

Locationofthesyslog-ngconfigurationfile 83

Theconfigurationsyntaxindetail 83

Notesabouttheconfigurationsyntax 86

Definingconfigurationobjectsinline 87

Usingchannelsinconfigurationobjects 88

Globalandenvironmentalvariables 90

Modulesinsyslog-ngOSE 91

Loadingmodules 91

Managingcomplexsyslog-ngconfigurations 92

Includingconfigurationfiles 92

Reusingconfigurationblocks 93

Passingargumentstoconfigurationblocks 95

Generatingconfigurationblocksfromascript 96

source: Read, receive, and collect log messages 99

Howsourceswork 99

default-network-drivers:Receiveandparsecommonsyslogmessages 103

default-network-drivers()sourceoptions 105

internal:Collectinginternalmessages 108

internal()sourceoptions 108

file:Collectingmessagesfromtextfiles 110

Notesonreadingkernelmessages 111

file()sourceoptions 111

wildcard-file:Collectingmessagesfrommultipletextfiles 122

wildcard-file()sourceoptions 123

network:CollectingmessagesusingtheRFC3164protocol(network()driver) 136

network()sourceoptions 138


nodejs:ReceivingJSONmessagesfromnodejsapplications 149

nodejs()sourceoptions 150

mbox:Convertinglocale-mailmessagestologmessages 152

mbox()sourceoptions 153

osquery:Collectandparseosqueryresultlogs 154

osquery()sourceoptions 157

pipe:Collectingmessagesfromnamedpipes 160

pipe()sourceoptions 160

pacct:CollectingprocessaccountinglogsonLinux 171

pacct()options 171

program:Receivingmessagesfromexternalapplications 173

program()sourceoptions 174

snmptrap:ReadNet-SNMPtraps 181

snmptrap()sourceoptions 184

sun-streams:CollectingmessagesonSunSolaris 187

sun-streams()sourceoptions 187

syslog:CollectingmessagesusingtheIETFsyslogprotocol(syslog()driver) 194

syslog()sourceoptions 195

system:Collectingthesystem-specificlogmessagesofaplatform 207

system()sourceoptions 209

systemd-journal:Collectingmessagesfromthesystemd-journalsystemlogstorage 211

systemd-journal()sourceoptions 213

systemd-syslog:Collectingsystemdmessagesusingasocket 218

systemd-syslog()sourceoptions 218

tcp,tcp6,udp,udp6:CollectingmessagesfromremotehostsusingtheBSDsyslogprotocolOBSOLETE 220

tcp(),tcp6(),udp()andudp6()sourceoptions:OBSOLETE 220

Changeanoldsourcedrivertothenetwork()driver 221

unix-stream,unix-dgram:CollectingmessagesfromUNIXdomainsockets 222

UNIXcredentialsandothermetadata 222

unix-stream()andunix-dgram()sourceoptions 223

stdin:Collectingmessagesfromthestandardinputstream 232

stdin()sourceoptions 233

destination: Forward, send, and store log messages 244

amqp:PublishingmessagesusingAMQP 246


amqp()destinationoptions 247

elasticsearch:SendingmessagesdirectlytoElasticsearchversion1.x 257

Prerequisites 259

Howsyslog-ngOSEinteractswithElasticsearch 260

Clientmodes 261

Elasticsearchdestinationoptions 261

elasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigher 273

Prerequisites 275

Howsyslog-ngOSEinteractswithElasticsearch 276

Clientmodes 277

SearchGuardandsyslog-ngOSE 278

Elasticsearch2destinationoptions 279

ExampleusecasesofsendinglogstoElasticsearchusingsyslog-ng 299

file:Storingmessagesinplain-textfiles 299

file()destinationoptions 301

graphite:SendingmetricstoGraphite 312

graphite()destinationoptions 313

SendinglogstoGraylog 315

graylog2()destinationoptions 317

hdfs:StoringmessagesontheHadoopDistributedFileSystem(HDFS) 318

Prerequisites 320

Howsyslog-ngOSEinteractswithHDFS 321

StoringmessageswithMapR-FS 322

Kerberosauthenticationwithsyslog-nghdfs()destination 323

HDFSdestinationoptions 324

PostingmessagesoverHTTP 334

HTTPdestinationoptions 335

http:PostingmessagesoverHTTPwithoutJava 339

HTTPdestinationoptions 340

kafka:PublishingmessagestoApacheKafka 354

Prerequisites 355

Howsyslog-ngOSEinteractswithApacheKafka 356

Kafkadestinationoptions 357

loggly:UsingLoggly 363

loggly()destinationoptions 364


logmatic:UsingLogmatic.io 366

logmatic()destinationoptions 367

mongodb:StoringmessagesinaMongoDBdatabase 369

Howsyslog-ngOSEconnectstheMongoDBserver 370

mongodb()destinationoptions 371

network:SendingmessagestoaremotelogserverusingtheRFC3164protocol(network()driver) 381

network()destinationoptions 382

osquery:Sendinglogmessagestoosquery'ssyslogtable 395

osquery()destinationoptions 396

pipe:Sendingmessagestonamedpipes 399

pipe()destinationoptions 399

program:Sendingmessagestoexternalapplications 406

program()destinationoptions 407

pseudofile() 416

pseudofile()destinationoptions 417

redis:Storingname-valuepairsinRedis 419

redis()destinationoptions 420

riemann:MonitoringyourdatawithRiemann 426

riemann()destinationoptions 427

smtp:GeneratingSMTPmessages(e-mail)fromlogs 438

smtp()destinationoptions 440

Splunk:SendinglogmessagestoSplunk 448

sql:StoringmessagesinanSQLdatabase 448

Usingthesql()driverwithanOracledatabase 450

Usingthesql()driverwithaMicrosoftSQLdatabase 451

Thewaysyslog-nginteractswiththedatabase 453

MySQL-specificinteractionmethods 454

MsSQL-specificinteractionmethods 454

sql()destinationoptions 454

stomp:PublishingmessagesusingSTOMP 466

stomp()destinationoptions 467

syslog:SendingmessagestoaremotelogserverusingtheIETF-syslogprotocol 473

syslog()destinationoptions 474

syslog-ng()destinationoptions 488


tcp,tcp6,udp,udp6:SendingmessagestoaremotelogserverusingthelegacyBSD-syslogprotocol(tcp(),udp()drivers) 500

tcp(),tcp6(),udp(),andudp6()destinationoptions 500

Changeanolddestinationdrivertothenetwork()driver 501

Telegram:SendingmessagestoTelegram 502

telegram()destinationoptions 502

unix-stream,unix-dgram:SendingmessagestoUNIXdomainsockets 505

unix-stream()andunix-dgram()destinationoptions 506

usertty:Sendingmessagestoauserterminal:usertty()destination 515

WriteyourowncustomdestinationinJavaorPython 516

log: Filter and route log messages using log paths, flags, and filters 517

Logpaths 517

Embeddedlogstatements 518

Usingembeddedlogstatements 520

if-else-elif:Conditionalexpressions 522

Junctionsandchannels 522

Logpathflags 525

Managingincomingandoutgoingmessageswithflow-control 528

Flow-controlandmultipledestinations 532

Configuringflow-control 532

Usingdisk-basedandmemorybuffering 534

Enablingreliabledisk-basedbuffering 536

Enablingnormaldisk-basedbuffering 537

Enablingmemorybuffering 537

Aboutdiskqueuefiles 538

Filters 539

Usingfilters 539

Combiningfilterswithbooleanoperators 540

Comparingmacrovaluesinfilters 541

Usingwildcards,specialcharacters,andregularexpressionsinfilters 542

Taggingmessages 543

Filterfunctions 544

Droppingmessages 549

Global options of syslog-ng OSE 551

Configuringglobalsyslog-ngoptions 551


Globaloptions 551

TLS-encrypted message transfer 569

SecureloggingusingTLS 569

EncryptinglogmessageswithTLS 570

ConfiguringTLSonthesyslog-ngclients 571

ConfiguringTLSonthesyslog-ngserver 572

MutualauthenticationusingTLS 574

ConfiguringTLSonthesyslog-ngclients 575

ConfiguringTLSonthesyslog-ngserver 576

Password-protectedkeys 578

TLSoptions 579

template and rewrite: Format, modify, and manipulate log messages 586

Customizemessageformatusingmacrosandtemplates 586

Formattingmessages,filenames,directories,andtablenames 587

Templatesandmacros 587

Date-relatedmacros 589

Hardvs.softmacros 590

Macrosofsyslog-ngOSE 591

Usingtemplatefunctions 600

Templatefunctionsofsyslog-ngOSE 601

Modifyingtheon-the-wiremessageformat 623

Modifyingmessagesusingrewriterules 623

Replacingmessageparts 624

Settingmessagefieldstospecificvalues 625

Unsettingmessagefields 628

CreatingcustomSDATAfields 629

Settingmultiplemessagefieldstospecificvalues 630

map-value-pairs:Renamevalue-pairstonormalizelogs 631

Conditionalrewrites 632

Howconditionalrewritingworks 632

Addinganddeletingtags 633

Anonymizingcreditcardnumbers 634

Regularexpressions 634

Typesandoptionsofregularexpressions 635


Optimizingregularexpressions 637

parser: Parse and segment structured messages 639

Parsingsyslogmessages 640

Optionsofsyslog-parserparsers 642

Parsingmessageswithcomma-separatedandsimilarvalues 644

OptionsofCSVparsers 647

Parsingkey=valuepairs 651

Optionsofkey=valueparsers 654

The JSON parser 655

TheJSONparserTheJSONparser 655

OptionsofJSONparsers 658

TheXMLparser 660

OptionsofXMLparsers 663

Parsingdatesandtimestamps 666

Optionsofdate-parser()parsers 667

TheApacheAccessLogParser 669

Optionsofapache-accesslog-parser()parsers 670

TheCiscoParser 671

TheLinuxAuditParser 673

Optionsoflinux-audit-parser()parsers 675

ThePythonParser 676

Parsingenterprise-widemessagemodel(EWMM)messages 681

Thesudoparser 681

Theiptablesparser 682

db-parser: Process message content with a pattern database (patterndb) 684

Classifyinglogmessages 684

Thestructureofthepatterndatabase 685

Howpatternmatchingworks 686

Artificialignorance 687

Usingpatterndatabases 688

Usingparserresultsinfiltersandtemplates 689

Downloadingsamplepatterndatabases 691

Correlatinglogmessagesusingpatterndatabases 692

Referencingearliermessagesofthecontext 694


Triggeringactionsforidentifiedmessages 695

Conditionalactions 697

Externalactions 698

Actionsandmessagecorrelation 699

Creatingpatterndatabases 702

Usingpatternparsers 702

Patternparsersofsyslog-ngOSE 704

What'snewinthesyslog-ngpatterndatabaseformatV5 707

Thesyslog-ngpatterndatabaseformat 707

Element:patterndb 709

Element:ruleset 709

Element:patterns 710

Element:rules 711

Element:rule 712

Element:patterns 714

Element:urls 715

Element:values 716

Element:examples 716

Element:example 717

Element:actions 718

Element:action 720

Element:create-context 722

Element:tags 725

Correlating log messages 726

Correlatingmessagesusingthegrouping-by()parser 726

Referencingearliermessagesofthecontext 730

Optionsofgrouping-byparsers 731

Enriching log messages with external data 735

Addingmetadatafromanexternalfile 735

Usingfiltersasselector 737

Optionsadd-contextual-data() 738

LookingupGeoIPdatafromIPaddresses(DEPRECATED) 740

Optionsofgeoipparsers 742

LookingupGeoIP2datafromIPaddresses 743


Referringtopartsofthemessageasamacro 744

UsingtheGeoIP2parser 744

TransferringyourlogstoElasticsearchusingGeoIP2 745

Optionsofgeoip2parsers 746

Statistics of syslog-ng 748

Metricsandcountersofsyslog-ngOSE 748

Logstatisticsfromtheinternal()source 751

Multithreading and scaling in syslog-ng OSE 753

Multithreadingconceptsofsyslog-ngOSE 753

Configuringmultithreading 755

Optimizingmultithreadedperformance 755

Troubleshooting syslog-ng 757

Possiblecausesoflosinglogmessages 757

Creatingsyslog-ngcorefiles 759

Collectingdebugginginformationwithstrace,truss,ortusc 759

Runningafailurescript 760

Stoppingsyslog-ng 761

Reportingbugsandfindinghelp 761

Recoverdatafromorphaneddiskbufferfiles 762

Nolocallogsafterspecifyinganunusualstoragedirectory 762

Nologsafterspecifyinganunusualportnumber 762

Errormessages 763

Best practices and examples 765

Generalrecommendations 765

Handlinglargemessageload 765

Usingnameresolutioninsyslog-ng 766

Resolvinghostnameslocally 767

Collectinglogsfromchroot 767

Configuringlogrotation 768

The syslog-ng manual pages 770

Thedqtooltoolmanualpage 770

Name 770

Synopsis 770


Description 770

Thecatcommand 771

Files 772

Seealso 772

Author 772

Copyright 772

Theloggenmanualpage 772

Name 773

Synopsis 773

Description 773

Options 773

Examples 776

Files 776

Seealso 776

Author 777

Copyright 777

Thepdbtoolmanualpage 777

Name 777

Synopsis 777

Description 778

Thedictionarycommand 778

Thedumpcommand 778

Thematchcommand 779

Themergecommand 781

Thepatternizecommand 782

Thetestcommand 783

Files 783

Seealso 783

Author 784

Copyright 784

Thesyslog-ngcontroltoolmanualpage 784

Name 784

Synopsis 784

Description 785

Enablingtroubleshootingmessages 785


syslog-ng-ctlquery 786

Thestatscommand 788

Handlingpassword-protectedprivatekeys 789

Reloadingtheconfiguration 790

Files 790

Seealso 791

Author 791

Copyright 791

Thesyslog-ng-debunmanualpage 791

Name 791

Synopsis 792

Description 792

GeneralOptions 792

Debugmodeoptions 792

Systemcalltracing 793

Packetcaptureoptions 793

Examples 793

Files 795

Seealso 795

Author 795

Copyright 795

Thesyslog-ngmanualpage 795

Name 795

Synopsis 796

Description 796

Options 796

Files 799

Seealso 799

Author 799

Copyright 799

Thesyslog-ng.confmanualpage 799

Name 800

Synopsis 800

Description 800

Basicconceptsofsyslog-ngOSE 800


Configuringsyslog-ng 801

Files 805

Seealso 805

Author 806

Copyright 806

Third-party contributions 807

GNUGeneralPublicLicense 807

Preamble 807

TERMSANDCONDITIONSFORCOPYING,DISTRIBUTIONANDMODIFICATION 808

Section0 808

Section1 809

Section2 809

Section3 810

Section4 810

Section5 810

Section6 811

Section7 811

Section8 811

Section9 812

Section10 812

NOWARRANTYSection11 812

Section12 812

HowtoApplyTheseTermstoYourNewPrograms 813

GNULesserGeneralPublicLicense 814

Preamble 814

TERMSANDCONDITIONSFORCOPYING,DISTRIBUTIONANDMODIFICATION 816

Section0 816

Section1 816

Section2 817

Section3 817

Section4 818

Section5 818

Section6 819

Section7 820

Section8 820


Section9 820

Section10 820

Section11 821

Section12 821

Section13 821

Section14 822



HowtoApplyTheseTermstoYourNewLibraries 822

Licenseattributions 823

Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License 824

About us 830

Contactingus 830

Technicalsupportresources 830


Preface

Welcometothesyslog-ngOpenSourceEdition3.16AdministratorGuide!

Thisdocumentdescribeshowtoconfigureandmanagesyslog-ng.Backgroundinformationforthetechnologyandconceptsusedbytheproductisalsodiscussed.

Summary of contents

Introductiontosyslog-ngdescribesthemainfunctionalityandpurposeofsyslog-ngOSE.

Theconceptsofsyslog-ngdiscussesthetechnicalconceptsandphilosophiesbehindsyslog-ngOSE.

Installingsyslog-ngdescribeshowtoinstallsyslog-ngOSEonvariousUNIX-basedplatformsusingtheprecompiledbinaries.

Thesyslog-ngOSEquick-startguideprovidesabrieflyexplainshowtoperformthemostcommonlogcollectingtaskswithsyslog-ngOSE.

Thesyslog-ngOSEconfigurationfilediscussestheconfigurationfileformatandsyntaxindetail,andexplainshowtomanagelarge-scaleconfigurationsusingincludedfilesandreusableconfigurationsnippets.

source:Read,receive,andcollectlogmessagesexplainshowtocollectandreceivelogmessagesfromvarioussources.

destination:Forward,send,andstorelogmessagesdescribesthedifferentmethodstostoreandforwardlogmessages.

log:Filterandroutelogmessagesusinglogpaths,flags,andfiltersexplainshowtorouteandsortlogmessages,andhowtousefilterstoselectspecificmessages.

Globaloptionsofsyslog-ngOSEliststheglobaloptionsofsyslog-ngOSEandexplainshowtousethem.

TLS-encryptedmessagetransfershowshowtosecureandauthenticatelogtransportusingTLSencryption.

templateandrewrite:Format,modify,andmanipulatelogmessagesdescribeshowtocustomizemessageformatusingtemplatesandmacros,howtorewriteandmodifymessages,andhowtouseregularexpressions.

parser:Parseandsegmentstructuredmessagesdescribeshowtosegmentandprocessstructuredmessageslikecomma-separatedvalues.

db-parser:Processmessagecontentwithapatterndatabase(patterndb)explainshowtoidentifyandprocesslogmessagesusingapatterndatabase.

Correlatinglogmessagesexplainshowtocorrelatelogmessagesthatmatchasetoffiltersorthatareidentifiedusingapatterndatabase.

syslog-ng OSE 3.16 Administration Guide

Preface18

Enrichinglogmessageswithexternaldataexplainshowtoimportdatafromexternalsourcestoincludeinthelogmessages,thusextending,enriching,andcomplementingthedatafoundinthelogmessage.

Statisticsofsyslog-ngdetailstheavailablestatisticsthatsyslog-ngOSEcollectsabouttheprocessedlogmessages.

Multithreadingandscalinginsyslog-ngOSEdescribeshowtoconfiguresyslog-ngOSEtousemultipleprocessors,andhowtooptimizeitsperformance.

Troubleshootingsyslog-ngofferstipstosolvingproblems.

Bestpracticesandexamplesgivesrecommendationstoconfigurespecialfeaturesofsyslog-ngOSE.

Thesyslog-ngmanualpagescontainsthemanualpagesofthesyslog-ngOSEapplication.

Third-partycontributionsincludesthetextofthelicensesapplicabletosyslog-ngOpenSourceEdition.

CreativeCommonsAttributionNon-commercialNoDerivatives(by-nc-nd)LicenseincludesthetextoftheCreativeCommonsAttributionNon-commercialNoDerivatives(by-nc-nd)LicenseapplicabletoThesyslog-ngOpenSourceEdition3.16AdministratorGuide.

Target audience and prerequisites

Thisguideisintendedforsystemadministratorsandconsultantsresponsiblefordesigningandmaintainingloggingsolutionsandlogcenters.ItisalsousefulforITdecisionmakerslookingforatooltoimplementcentralizedlogginginheterogeneousenvironments.

Thefollowingskillsandknowledgearenecessaryforasuccessfulsyslog-ngadministrator:

l Atleastbasicsystemadministrationknowledge.

l Anunderstandingofnetworks,TCP/IPprotocols,andgeneralnetworkterminology.

l WorkingknowledgeoftheUNIXorLinuxoperatingsystem.

l In-depthknowledgeoftheloggingprocessofvariousplatformsandapplications.

l Anunderstandingofthelegacysyslog(BSD-syslog)protocolandthenewsyslog(IETF-syslog)protocolstandard.

Products covered in this guide

Thisguidedescribestheuseofthefollowingproducts:

l syslog-ngOpenSourceEdition(syslog-ngOSE)3.16.1andlater


Preface19

https://www.ietf.org/rfc/rfc3164.txthttps://tools.ietf.org/html/rfc5424https://tools.ietf.org/html/rfc5424

Summary of changes

ThissectionliststhechangesofThesyslog-ngOpenSourceEditionAdministratorGuide.

Version 3.15 - 3.16

Changes in product:

l Anewdestinationdriver,telegram(),hasbeenadded.Thetelegram()destinationsendslogmessagestoTelegram,whichisasecure,cloud-basedmobileanddesktopmessagingapp.Formoreinformation,seeTelegram:SendingmessagestoTelegram.

l Anewtemplatefunction,urlencode,hasbeenadded.Youcanusetheurlencodetemplatefunctiontogetherwiththetelegram()destinationtosendsyslogmessagestoTelegram.Formoreinformation,seeurlencode.

l Toensurethatamoduleisloaded,[email protected],seeLoadingmodules.

l Theadd-contextual-data()hasbeenextendedwiththeignore-case()option.Formoreinformation,seeOptionsadd-contextual-data().

l Thehook-commands()hasbeenadded,whichmakesitpossibletoexecuteexternalprogramswhentheyareinitializedortorndown.Thehook-commands()canbeusedforbothsourceanddestinationdrivers.Formoreinformation,seehook-commands().

Version 3.14 - 3.15

Changes in product:

l Itisnowpossibletouseif {},elif {},andelse {}blockstoconfigureconditionalexpressions.Fordetails,seeif-else-elif:Conditionalexpressions.

l Anewlogpathflag,drop-unmatched,hasbeenadded.Thenewflagcausesmessagestobedroppedalongalogpathwhentheydonotmatchafilterorarediscardedbyaparser.Fordetails,seeLogpathflags.

l SupportforElasticsearch'sShieldhasbeenremoved.

l SupportforPOSIXregularexpressionshasbeenremoved.


Preface20

https://core.telegram.org/https://core.telegram.org/

Version 3.13 - 3.14

Changes in product:

l Youcanusepassword-protectedprivatekeysinthenetwork()andsyslog()sourceanddestinationdrivers.Fordetails,seePassword-protectedkeys.

l Tobettercontroltowhichlogmessagesyouaddcontextualdata,youcanusefiltersasselectors.Inthiscase,thefirstcolumnoftheCSVdatabasefilemustcontainthenameofafilter.Foreachmessage,syslog-ngOSEevaluatesthefiltersintheordertheyappearinthedatabasefile.Ifafiltermatchesthemessage,syslog-ngOSEaddsthename-valuepairrelatedtothefilter.Fordetails,seeUsingfiltersasselector.

Version 3.12 - 3.13

Changes in product:

l Anewsourcedriver,stdin(),hasbeenadded.Thestdin()drivercollectsmessagesfromthestandardinputstream.Formoreinformation,seestdin:Collectingmessagesfromthestandardinputstream.

l Anewdestination,SendinglogstoGraylog,andatemplatetosendsyslogmessagestoGraylog,format-gelf,hasbeenadded.

l Anewtemplatefunction,getent,hasbeenadded.YoucanusethegetenttemplatefunctiontolookupentriesfromtheNameServiceSwitchlibraries.Formoreinformation,seegetent.

l Thedefaultvaluesofthe--enable-json,--enable-mongodb,and--with-libmongo-clientcompileparametershavechanged.Formoreinformation,seeCompilingoptionsofsyslog-ngOSE.

l Anewcompileoption,--with-module-path,hasbeenadded.Thenewoptionspecifiessyslog-ngOSE'smoduleinstallationdirectory.Formoreinformation,seeCompilingoptionsofsyslog-ngOSE.

l Anewdestinationdriver,osquery(),hasbeenadded.Thenewdriversendslogmessagestoosquery'ssyslogtable.Formoreinformation,seeosquery:Sendinglogmessagestoosquery'ssyslogtable.

l ItisnowpossibletospecifyTLSoptionsinatls()block.Formoreinformation,see:

l amqp()destinationoptions

l HTTPdestinationoptions

l riemann()destinationoptions

l SupportformicrosecondsinRiemanndestinationshasbeenintroduced.Formoreinformation,seeevent-time().


Preface21

l Moduleauto-loadingnowalsoworksforthesystem()source.Formoreinformation,see--default-modules.

Changes in documentation:

l Anewsectiondescribingcommonerrormessageshasbeenaddedtothedocument.Formoreinformation,seeErrormessages.

l Severalcorrectionsandeditorialchanges.

Version 3.11 - 3.12

Changes in product:

l Anewsystemd-journal()sourceoption,calledread-old-records(),hasbeenadded.Formoreinformation,seeread-old-records().

l Anoptioncalledjvm-options()hasbeenadded,whichallowsyoutofine-tuneJavaVirtualMachinesettingswhenconfiguringElasticsearch,HDFS,andApacheKafkadestinations,orwebservicestowhichyousendlogmessagesviatheHTTPprotocol.Fordetails,see:

l Elasticsearchdestinationoptions

l Elasticsearch2destinationoptions

l HDFSdestinationoptions

l HTTPdestinationoptions

l Kafkadestinationoptions

l Globaloptions

l AnewHDFSdestinationoption,calledhdfs-append-enabled()hasbeenadded.Forfurtherinformation,seehdfs-append-enabled().

l Macrosarenowsupportedinthehdfs-file()option.Fordetails,seehdfs-file().

l ThefollowingnewTLSoptionshavebeenadded:

l dhparam-file()

l ecdh-curve-list()

l pkcs12-file().

l Anewparser,capableofprocessinginputinXMLformat,hasbeenadded.Formoreinformation,seeTheXMLparser.


l Addedsectionaboutcommercialversionofsyslog-ng.Formoreinformation,seeCommercialversionofsyslog-ng.

l Addedwarningabouttherequirementtodeletethepersistfileoncethedir()option


Preface22

ofdisk-buffer()hasbeenmodifiedoranewonehasbeenadded.Formoreinformation,seedestination:Forward,send,andstorelogmessages.

l ClarifiedinformationaboutthePythonparser'sdeinit()method.Itrunsnotonlyatasyslog-nggracefulstop,butatareloadtoo.Fordetails,seeMethodsofthepython()parser.


Version 3.10 - 3.11

Changes in product:

l LookingupGeoIP2datafromIPaddresseshasbeenaddedtothedocument.

l http:PostingmessagesoverHTTPwithoutJavahasbeenupgradedwithnewimprovements.

l Thegeoip()parserisnowdeprecated.LookingupGeoIPdatafromIPaddresses(DEPRECATED).

l Thetemplate()optionhasbeenaddedtotheApacheAccessLogParser.Fordetails,see:TheApacheAccessLogParser.

l SSL-relatedoptionshavebeenaddedtoamqp()destination.Fordetails,see:amqp()destinationoptions.

l Theprefix()optionhasbeenaddedtotheCiscoparser.Fordetails,see:TheCiscoParser.

l Thedrop-unmatched()optionhasbeenaddedtothedb-parser()statement.Fordetails,see:Usingpatterndatabases.

l Theevent-time()optionhasbeenaddedtotheRiemanndestination.Fordetails,see:riemann:MonitoringyourdatawithRiemann.


l Anewexamplehasbeenaddedtotheosquery()source.Fordetails,see:osquery:Collectandparseosqueryresultlogs.



Preface23

Version 3.9 - 3.10

Changes in product:

l wildcard-file: Collectingmessages frommultiple text files has been added tothe document.

l snmptrap:ReadNet-SNMPtrapshasbeenaddedtothedocument.

l osquery:Collectandparseosqueryresultlogshasbeenaddedtothedocument.

l Theelasticsearch2()destinationnowsupportsHTTPSmode,includingencryption,andalsopassword-andcertificate-basedauthentication.Fordetails,seeelasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigher.

l Thehttp()destinationnowsupportsencryption,andalsopassword-andcertificate-basedauthentication.Fordetails,seeHTTPdestinationoptions.

l Thehdfs()destinationnowsupportsKerberosauthentication.Fordetails,seeKerberosauthenticationwithsyslog-nghdfs()destination.

l ThePythonParserhasbeenaddedtothedocument.

l TheCiscoParserhasbeenaddedtothedocument.

l map-value-pairs: Rename value-pairs to normalize logs has been added to thedocument.

l Thelist-*templatefunctionsallowyoutomanipulatecomma-separatedlists.Fordetails,seeListmanipulation.

l Thenewbasename()anddirname()templatefunctionsallowyoutoeasilyseparatethepathandfilenames.Fordetails,seeTemplatefunctionsofsyslog-ngOSE.

l stardatehasbeenaddedtothedocument.

l create-statement-append()hasbeenaddedtothedocument.

l Thedefaultvalueofthelog-msg-size()optionhasbeenincreasedto64k.Thatwaysyslog-ngOSEwillnottruncatelonglogmessages,whicharegettingincreasinglycommon.


l Splunk:SendinglogmessagestoSplunkhasbeenaddedtothedocument.

l Aboutdiskqueuefileshasbeenaddedtothedocument.

l AnexamplefailurescripthasbeenaddedtoRunningafailurescript.



Preface24

Version 3.8 - 3.9

Changes in product:

l WhenusingTLS-transport,youcannowusecertainfieldsoftheX.509certificatesasmacros.Fordetails,see.TLS.X509.

l Theelastic2()destinationdrivernowsupportsSearchGuard,analternativesecuritysolutionforElasticsearch.Fordetails,seeSearchGuardandsyslog-ngOSE.

l .TLS.X509hasbeenaddedtothedocument.

l Unsettingmessagefieldshasbeenupdatedwithgroupunset().


l Correctionsandeditorialchanges.

Version 3.7 - 3.8

Changes in product:

l Enrichinglogmessageswithexternaldatahasbeenaddedtothedocument.

l Correlatinglogmessageshasbeenaddedtothedocument.

l elasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigherhasbeenaddedtothedocument.

l http:PostingmessagesoverHTTPwithoutJavahasbeenaddedtothedocument.

l logmatic:UsingLogmatic.iohasbeenaddedtothedocument.

l loggly:UsingLogglyhasbeenaddedtothedocument.

l Disk-basedbufferinghasbeenaddedtosyslog-ngOSE.Fordetails,seeUsingdisk-basedandmemorybuffering.

l What'snewinthesyslog-ngpatterndatabaseformatV5,,hasbeenaddedtoElement:create-contexthasbeenaddedtodb-parser:Processmessagecontentwithapatterndatabase(patterndb).

l Parsingdatesandtimestampshasbeenaddedtoparser:Parseandsegmentstructuredmessages.

l TheApacheAccessLogParserhasbeenaddedtoparser:Parseandsegmentstructuredmessages.

l Newoptionsoftheset()rewriteoperatorhavebeenaddedtoSettingmessagefieldstospecificvalues.

l ArewriteoperatortounsetfieldshasbeenaddedtoUnsettingmessagefields.


Preface25

https://github.com/floragunncom/search-guard

l Atemplatefunctionthatformatsname-valuepairsasArcSightCommonEventFormatextensionhasbeenaddedtoformat-cef-extension.

l NumericaltemplatefunctionsthatworkonnumericalvaluesofacorrelationcontexthavebeenaddedtoNumericaloperations.

l Theinherit-environment()optionhasbeenaddedtoprogram:Receivingmessagesfromexternalapplicationsandprogram:Sendingmessagestoexternalapplications.

l @NLSTRING@hasbeenaddedtoUsingpatternparsers.


l LookingupGeoIPdatafromIPaddresses(DEPRECATED)hasbeenmovedtoEnrichinglogmessageswithexternaldata.


Version 3.6 - 3.7

Changes in product:

l mbox: Converting local e-mailmessages to logmessages has been added tothe document.

l Thekeep-alive()optionhasbeenaddedtotheprogram()destination.

l The Linux Audit Parser has been added to parser: Parse and segmentstructuredmessages.

l pythonhasbeenaddedtoTemplatefunctionsofsyslog-ngOSE.

l PostingmessagesoverHTTPhasbeenaddedtothedocument.

l Write your own custom destination in Java or Python has been added to thedocument.

l Looking up GeoIP data from IP addresses (DEPRECATED) has been added tothe document.

l Elasticsearchdestinationoptionshasbeenaddedtothedocument.

l kafka:PublishingmessagestoApacheKafkahasbeenaddedtothedocument.

l hdfs:StoringmessagesontheHadoopDistributedFileSystem(HDFS)hasbeenaddedtothedocument.

l Parsingkey=valuepairshasbeenaddedtothedocument.

l format-cimhasbeenaddedtothedocument.

l Simpletemplatescanbedefinedwithoutbraces.Templatescanalsoreferenceothertemplates.Fordetails,seeTemplatesandmacros.

l Customtemplatefunctionscanbedefinedinthesyslog-ngOSEconfiguration.For


Preface26

details,seeUsingtemplatefunctions.

l CSV-parserscanusestringsasdelimiters.Fordetails,seedelimiters().

l IPv6addressescanbefilteredusinganewfilter.Fordetails,seenetmask6().

l Theloggenutilitycansendmessagesindefinitelyusingthe--permanentoption.

l Thessl-options()optionhasbeedaddedtoTLSoptions.

l TLS-supporthasbeenaddedtoriemann()destinationoptions.

l Theextract-solaris-msgid()parserhasbeedaddedtosun-streams:CollectingmessagesonSunSolaris.

l Thecontextoptionofinherit-propertieshasbeedaddedtoActionsandmessagecorrelation.

l flush-lines()hasbeenaddedtothedocument.

l Thesanitize-utf8flaghasbeenaddedtothelistofsourceflags.

l Theformat-welffunctionhasbeenaddedtoTemplatefunctionsofsyslog-ngOSE.

l The pass-unix-credentials() option has been added to Global options of syslog-ng OSE.

l Theuse-uniqid()optionhasbeenaddedtoGlobaloptionsofsyslog-ngOSE.

l TheUNIQIDmacrohasbeenaddedtoMacrosofsyslog-ngOSE.

l TheJSON-parsernowhandlesspecialcharactersinobjectnames.Fordetails,seeextract-prefix().

l Thesyslog-debuntoolusedtogeneratesyslog-ngOSEdebugbundleshasbeendocumented.Fordetails,seeThesyslog-ng-debunmanualpage.

l The --control option has been added to the The syslog-ngmanual pagemanual page.

l Version3.7andnewerautomaticallyincludestheplugin.conffilesfromthe/scl/*/directories,makingiteasiertouseanddistributeconfigurationblocks.

l The--enable-all-modulescompileroptionhasbeedaddedtoCompilingoptionsofsyslog-ngOSE.

l Thecreate-dirs()optionhasbeenaddedtounix-stream()andunix-dgram()destinationoptions.


l Generatingconfigurationblocksfromascripthasbeenaddedtothedocument.

l Example:Sendingalertwhenaclientdisappearshasbeenaddedtothedocument.

l Thetcp(),tcp6(),udp(),udp6()sourceanddestinationdrivershavebeendeprecated,asalloftheirfunctionalitycanbeachievedwiththenetwork()driver.Forhelponmigratingtothenetwork()driver,seeChangeanoldsourcedrivertothenetwork()driverandChangeanolddestinationdrivertothenetwork()driver.


Preface27

l ThebeginningofTroubleshootingsyslog-nghasbeenextendedwithbasictroubleshootinginformation.

l Thedescriptionofthechain-hostnames()globaloptionhasbeenclarifiedandextended.Fordetails,seechain-hostnames().

l Othereditorialcorrections.

Version 3.5 - 3.6

Changes in product:


l riemann:MonitoringyourdatawithRiemannhasbeenaddedtothedocument.

l nodejs:ReceivingJSONmessages fromnodejsapplicationshasbeenadded tothedocument.

l systemd-journal:Collectingmessagesfromthesystemd-journalsystemlogstoragehasbeenaddedtothedocument.

l systemd-syslog:Collectingsystemdmessagesusingasockethasbeenaddedtothedocument.

l use-rcptid()hasbeenaddedtothedocument.

l Settingmultiplemessagefieldstospecificvalueshasbeenaddedtothedocument.

l TheretriesandthrottleoptionsareavailablefortheSMTP,MongoDB,AMQP,andRedisdestinations.

l Thedescriptionofthemulti-line-modeoptionhasbeenupdated.

l UNIXcredentialsandothermetadatahasbeenaddedtothedocument.

l RUNIDhasbeenaddedtoMacrosofsyslog-ngOSE.

l Theextract-prefixoptionhasbeenaddedtoTheJSONparserTheJSONparser.

l Thegraphite-output,orandpaddingtemplatefunctionshavebeenaddedtoTemplatefunctionsofsyslog-ngOSE.

l PCREisnowarequireddependencyofsyslog-ngOSE,andbydefault,syslog-ngOSEusesPCRE-styleregularexpressions.Therefore,the--enable-pcrecompliationoptionhasbeenremoved.

l graphite:SendingmetricstoGraphitehasbeenaddedtothedocument.

l pseudofile()hasbeenaddedtothedocument.

l Thecustom-domain()andstats-lifetime()optionshavebeenaddedtoGlobaloptions.

l Theretry_sql_insertsoptionhasbeenrenamedtoretriestoincreaseconsistency.

l on-error()canbesetlocallyforMongoDBdestinationsaswell.Also,MongoDBdestinationssupporttheusernameandpasswordoptions,andconnectingtotheserver


Preface28

usingUNIXdomainsockets.Fordetails,seemongodb:StoringmessagesinaMongoDBdatabase.

l Howsyslog-ngOSEconnectstheMongoDBserverhasbeenaddedtothedocument.

l Severaltyposandsyntaxerrorsinexampleshavebeencorrected.

Feedback

Anyfeedbackisgreatlyappreciated,especiallyonwhatelsethisdocumentshouldcover.Generalcomments,errorsfoundinthetext,andanysuggestionsabouthowtoimprovethedocumentationisalsowelcomeatdocumentation@balabit.com.

ThesourceofthisguideisavailableonGitHub.Incaseofthesyslog-ngOpenSourceEditionguides,youcanalso:

l Openanissue

Acknowledgments

OneIdentitywouldliketoexpressitsgratitudetothesyslog-ngusersandthesyslog-ngcommunityfortheirinvaluablehelpandsupport.


Preface29

https://github.com/balabit/syslog-ng-ose-guideshttps://github.com/balabit/syslog-ng-ose-guides/issues

3

Introduction to syslog-ng

Thischapterintroducesthesyslog-ngOpenSourceEditionapplicationinanon-technicalmanner,discussinghowandwhyisituseful,andthebenefitsitofferstoanexistingITinfrastructure.

What syslog-ng is

Thesyslog-ngapplicationisaflexibleandhighlyscalablesystemloggingapplicationthatisidealforcreatingcentralizedandtrustedloggingsolutions.Amongothers,syslog-ngOSEallowsyouthefollowing.

Secure and reliable log transfer

Thesyslog-ngOSEapplicationenablesyoutosendthelogmessagesofyourhoststoremoteserversusingthelatestprotocolstandards.Youcancollectandstoreyourlogdatacentrallyondedicatedlogservers.TransferlogmessagesusingtheTCPprotocolensuresthatnomessagesarelost.

Disk-based message buffering

Tominimizetheriskoflosingimportantlogmessages,thesyslog-ngOSEapplicationcanstoremessagesonthelocalharddiskifthecentrallogserverorthenetworkconnectionbecomesunavailable.Thesyslog-ngapplicationautomaticallysendsthestoredmessagestotheserverwhentheconnectionisreestablished,inthesameorderthemessageswerereceived.Thediskbufferispersistentnomessagesarelostevenifsyslog-ngisrestarted.

Secure logging using TLS

Logmessagesmaycontainsensitiveinformationthatshouldnotbeaccessedbythirdparties.Therefore,syslog-ngOSEsupportstheTransportLayerSecurity(TLS)protocolto


Introduction to syslog-ng30

encryptthecommunication.TLSalsoallowsyoutoauthenticateyourclientsandthelogserverusingX.509certificates.

Flexible data extraction and processing

Mostlogmessagesareinherentlyunstructured,whichmakesthemdifficulttoprocess.Toovercomethisproblem,syslog-ngOSEcomeswithasetofbuilt-inparsers,whichyoucancombinetobuildverycomplexthings.

Filter and classify

Thesyslog-ngOSEapplicationcansorttheincominglogmessagesbasedontheircontentandvariousparameterslikethesourcehost,application,andpriority.Youcancreatedirectories,files,anddatabasetablesdynamicallyusingmacros.Complexfilteringusingregularexpressionsandbooleanoperatorsoffersalmostunlimitedflexibilitytoforwardonlytheimportantlogmessagestotheselecteddestinations.

Parse and rewrite

Thesyslog-ngOSEapplicationcansegmentlogmessagestonamedfieldsorcolumns,andalsomodifythevaluesofthesefields.YoucanprocessJSONmessages,key-valuepairs,andmore.

Togetthemostinformationoutofyourlogdata,syslog-ngOSEallowsyoutocorrelatelogmessagesandaggregatetheextractedinformationintoasinglemessage.Youcanalsouseexternalinformationtoenrichyourlogdata.

Big data clusters

Thelogdatathatyourorganizationhastoprocess,store,andreviewincreasesdaily,somanyorganizationsusebigdatasolutionsfortheirlogs.Toaccomodatethishugeamountofdata,syslog-ngOSEnativelysupportsstoringlogmessagesinHDFSfilesandElasticsearchclusters.

Message queue support

Largeorganizationsincreasinglyrelyonqueuinginfrastructuretotransfertheirdata.syslog-ngOSEsupportsApacheKafka,theAdvancedMessageQueuingProtocol(AMQP),andtheSimpleTextOrientedMessagingProtocol(STOMP).



SQL, NoSQL, and monitoring

Storing your log messages in a database allows you to easily search and query themessages and interoperate with log analyzing applications. The syslog-ng applicationsupports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, andSQLite.

syslog-ngOSEalsoallowsyoutoextracttheinformationyouneedfromyourlogdata,anddirectlysendittoyourGraphite,Redis,orRiemannmonitoringsystem.

Wide protocol and platform support

syslog protocol standards

syslog-ngnotonlysupportslegacyBSDsyslog(RFC3164)andtheenhancedRFC5424protocolsbutalsoJavaScriptObjectNotation(JSON)andjournaldmessageformats.

Heterogeneous environments

Thesyslog-ngOSEapplicationistheidealchoicetocollectlogsinmassivelyheterogeneousenvironmentsusingseveraldifferentoperatingsystemsandhardwareplatforms,includingLinux,Unix,BSD,SunSolaris,HP-UX,Tru64,andAIX.

IPv4 and IPv6 support

Thesyslog-ngapplicationcanoperateinbothIPv4andIPv6networkenvironments,andcanreceiveandsendmessagestobothtypesofnetworks.

What syslog-ng is not

Thesyslog-ngapplicationisnotloganalysissoftware.Itcanfilterlogmessagesandselectonlytheonesmatchingcertaincriteria.Itcanevenconvertthemessagesandrestructurethemtoapredefinedformat,orparsethemessagesandsegmentthemintodifferentfields.Butsyslog-ngcannotinterpretandanalyzethemeaningbehindthemessages,orrecognizepatternsintheoccurrenceofdifferentmessages.

Why is syslog-ng needed?

Logmessagescontaininformationabouttheeventshappeningonthehosts.Monitoringsystemeventsisessentialforsecurityandsystemhealthmonitoringreasons.



Theoriginalsyslogprotocolseparatesmessagesbasedonthepriorityofthemessageandthefacilitysendingthemessage.Thesetwoparametersaloneareofteninadequatetoconsistentlyclassifymessages,asmanyapplicationsmightusethesamefacility,andthefacilityitselfisnotevenincludedinthelogmessage.Tomakethingsworse,manylogmessagescontainunimportantinformation.Thesyslog-ngapplicationhelpsyoutoselectonlythereallyinterestingmessages,andforwardthemtoacentralserver.

Companypoliciesorotherregulationsoftenrequirelogmessagestobearchived.Storingtheimportantmessagesinacentrallocationgreatlysimplifiesthisprocess.

What is new in syslog-ng Open Source Edition 3.16?

Version3.16ofsyslog-ngOpenSourceEditionincludesthefollowingmainfeatures.

Easily receive and parse messages from remote hosts

Thedefault-network-drivers()sourceisaspecialsourcethatusesmultiplesourcedriverstoreceiveandparseseveraldifferenttypesofsyslogmessagesfromthenetwork.Fordetails,see"default-network-drivers()sourceoptions"intheAdministrationGuide.

Transfer log messages and their key-value pairs between syslog-ng nodes

TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesfromtheinitialreceivingsyslog-ngcomponentrightuptothecentrallogserver,throughanynumberofhops.Itdoesnotmatterifyouparsethemessagesontheclient,onarelay,oronthecentralserver,theirstructuredresultswillbeavailablewhereyoustorethemessages.Optionally,youcanalsoforwardtheoriginalrawmessageasthefirstsyslog-ngcomponentinyourinfrastructurehasreceivedit,whichisimportantifyouwanttoforwardamessageforexampletoaSIEMsystem.Tomakeuseoftheenterprise-widemessagemodel,youhavetousethesyslog-ng()destinationonthesenderside,andthedefault-network-drivers()sourceonthereceiverside.

Clearer configuration using if, else, elif conditions

Youcanuseif {},elif {},andelse {}blockstoconfigureconditionalexpressions.Fordetails,seeAdministrationGuide.

Message parsing

syslog-ngOSEversion3.16includesparsersforthesudoandiptablesapplications.

Foramoredetailedlist,seeVersion3.14-3.15andthesyslog-ngReleasespage.



https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://github.com/balabit/syslog-ng/releases

Execute external programs during startup

Thehook-commands()optionmakesitpossibletoexecuteexternalprogramswhentherelevantdriverisinitializedortorndown.Itcanbeusedwithallsourceanddestinationdriverswiththeexceptionoftheusertty()andinternal()drivers.Fordetails,seetheoptiondescriptionsintheAdministrationGuide,andthehook-commandsblogpost.

Who uses syslog-ng?

Thesyslog-ngapplicationisusedworldwidebycompaniesandinstitutionswhocollectandmanagethelogsofseveralhosts,andwanttostoretheminacentralized,organizedway.Usingsyslog-ngisparticularlyadvantageousfor:

l InternetServiceProviders

l Financialinstitutionsandcompaniesrequiringpolicycompliance

l Server,web,andapplicationhostingcompanies

l Datacenters

l Wideareanetwork(WAN)operators

l Serverfarmadministrators.

Supported platformsThesyslog-ngOpenSourceEditionapplicationishighlyportableandisknowntorunonawiderangeofhardwarearchitectures(x86,x86_64,SUNSparc,PowerPC32and64,Alpha)andoperatingsystems,includingLinux,BSD,Solaris,IBMAIX,HP-UX,MacOSX,Cygwin,Tru64,andothers.

l Thesourcecodeofsyslog-ngOpenSourceEditionisreleasedundertheGPLv2licenseandisavailableonGitHub.

l Seethelistofprecompiledsyslog-ngOSEbinarypackages.



https://www.syslog-ng.com/community/b/blog/posts/hook-commands-easy-driver-setuphttps://github.com/balabit/syslog-nghttps://syslog-ng.org/3rd-party-binaries/

4

The concepts of syslog-ng

Thischapterdiscussesthetechnicalconceptsofsyslog-ng.

The philosophy of syslog-ng

Typically,syslog-ngisusedtomanagelogmessagesandimplementcentralizedlogging,wheretheaimistocollectthelogmessagesofseveraldevicesonasingle,centrallogserver.Thedifferentdevicescalledsyslog-ngclientsallrunsyslog-ng,andcollectthelogmessagesfromthevariousapplications,files,andothersources.Theclientssendallimportantlogmessagestotheremotesyslog-ngserver,whichsortsandstoresthem.

Logging with syslog-ngThesyslog-ngapplicationreadsincomingmessagesandforwardsthemtotheselecteddestinations.Thesyslog-ngapplicationcanreceivemessagesfromfiles,remotehosts,andothersources.

Logmessagesentersyslog-nginoneofthedefinedsources,andaresenttooneormoredestinations.

Sourcesanddestinationsareindependentobjects,log pathsdefinewhatsyslog-ngdoeswithamessage,connectingthesourcestothedestinations.Alogpathconsistsofoneormoresourcesandoneormoredestinations:messagesarrivingfromasourcearesenttoeverydestinationlistedinthelogpath.Alogpathdefinedinsyslog-ngiscalledalog statement.

Optionally,logpathscanincludefilters.Filtersarerulesthatselectonlycertainmessages,forexample,selectingonlymessagessentbyaspecificapplication.Ifalogpathincludesfilters,syslog-ngsendsonlythemessagessatisfyingthefilterrulestothedestinationssetinthelogpath.

Otheroptionalelements thatcanappear in logstatementsareparsersand rewriting rules.Parserssegmentmessages intodifferent fields tohelpprocessing themessages,while rewrite rulesmodify themessagesbyadding, replacing,or removingpartsofthemessages.


The concepts of syslog-ng35

The route of a log message in syslog-ng

Purpose:

Thefollowingprocedureillustratestherouteofalogmessagefromitssourceonthesyslog-ngclienttoitsfinaldestinationonthecentralsyslog-ngserver.

Figure 1: The route of a log message

Steps:

1. Adeviceorapplicationsendsalogmessagetoasourceonthesyslog-ngclient.Forexample,anApachewebserverrunningonLinuxentersamessageintothe/var/log/apachefile.

2. Thesyslog-ngclientrunningonthewebserverreadsthemessagefromits/var/log/apachesource.

3. Thesyslog-ngclientprocessesthefirstlogstatementthatincludesthe/var/log/apachesource.

4. Thesyslog-ngclientperformsoptionaloperations(messagefiltering,parsing,andrewriting)onthemessage,forexample,itcomparesthemessagetothefiltersofthelogstatement(ifany).Ifthemessagecomplieswithallfilterrules,syslog-ngsendsthemessagetothedestinationssetinthelogstatement,forexample,totheremotesyslog-ngserver.



CAUTION:

Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

NOTE:

Thesyslog-ngclientsendsamessagetoallmatchingdestinationsbydefault.Asaresult,amessagemaybesenttoadestinationmorethanonce,ifthedestinationisusedinmultiplelogstatements.Topreventsuchsituations,usethefinalflaginthedestinationstatements.Fordetails,seeLogstatementflags.

5. Thesyslog-ngclientprocessesthenextlogstatementthatincludesthe/var/log/apachesource,repeatingSteps3-4.

6. Themessagesentby thesyslog-ngclientarrives fromasourceset in thesyslog-ngserver.

7. Thesyslog-ngserverreadsthemessagefromitssourceandprocessesthefirstlogstatementthatincludesthatsource.

8. Thesyslog-ngserverperformsoptionaloperations(messagefiltering,parsing,andrewriting)onthemessage,forexample,itcomparesthemessagetothefiltersofthelogstatement(ifany).Ifthemessagecomplieswithallfilterrules,syslog-ngsendsthemessagetothedestinationssetinthelogstatement.

CAUTION:

Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

9. Thesyslog-ngserverprocessesthenextlogstatement,repeatingSteps7-9.

NOTE:

Thesyslog-ngapplicationcanstopreadingmessagesfromitssourcesifthedestinationscannotprocessthesentmessages.Thisfeatureiscalledflow-controlandisdetailedinManagingincomingandoutgoingmessageswithflow-control.

Modes of operationThesyslog-ngOpenSourceEditionapplicationhasthreetypicaloperationscenarios:Client,Server,andRelay.

Client mode



Figure 2: Client-mode operation

Inclientmode,syslog-ngcollectsthelocallogsgeneratedbythehostandforwardsthemthroughanetworkconnectiontothecentralsyslog-ngserverortoarelay.Clientsoftenalsologthemessageslocallyintofiles.

Relay modeFigure 3: Relay-mode operation

Inrelaymode,syslog-ngreceiveslogsthroughthenetworkfromsyslog-ngclientsandforwardsthemtothecentralsyslog-ngserverusinganetworkconnection.Relaysalsologthemessagesfromtherelayhostintoalocalfile,orforwardthesemessagestothecentralsyslog-ngserver.

Server mode



Figure 4: Server-mode operation

Inservermode,syslog-ngactsasacentrallog-collectingserver.Itreceivesmessagesfromsyslog-ngclientsandrelaysoverthenetwork,andstoresthemlocallyinfiles,orpassesthemtootherapplications,forexampleloganalyzers.

Global objectsThesyslog-ngapplicationusesthefollowingobjects:

l Source driver:Acommunicationmethodusedtoreceivelogmessages.Forexample,syslog-ngcanreceivemessagesfromaremotehostviaTCP/IP,orreadthemessagesofalocalapplicationfromafile.Fordetailsonsourcedrivers,seesource:Read,receive,andcollectlogmessages.

l Source:Anamedcollectionofconfiguredsourcedrivers.

l Destination driver:Acommunicationmethodusedtosendlogmessages.Forexample,syslog-ngcansendmessagestoaremotehostviaTCP/IP,orwritethemessagesintoafileordatabase.Fordetailsondestinationdrivers,seedestination:Forward,send,andstorelogmessages.

l Destination:Anamedcollectionofconfigureddestinationdrivers.

l Filter:Anexpressiontoselectmessages.Forexample,asimplefiltercanselectthe



messagesreceivedfromaspecifichost.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.

l Macro:Anidentifierthatreferstoapartofthelogmessage.Forexample,the${HOST}macroreturnsthenameofthehostthatsentthemessage.Macrosareoftenusedintemplatesandfilenames.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.

l Parser:Parsersareobjectsthatparsetheincomingmessages,orpartsofamessage.Forexample,thecsv-parser()cansegmentmessagesintoseparatecolumnsatapredefinedseparatorcharacter(forexampleacomma).Everycolumnhasauniquenamethatcanbeusedasamacro.Fordetails,seeparser:Parseandsegmentstructuredmessagesanddb-parser:Processmessagecontentwithapatterndatabase(patterndb).

l Rewrite rule:Arulemodifiesapartofthemessage,forexample,replacesastring,orsetsafieldtoaspecifiedvalue.Fordetails,seeModifyingmessagesusingrewriterules.

l Log paths:Acombinationofsources,destinations,andotherobjectslikefilters,parsers,andrewriterules.Thesyslog-ngapplicationsendsmessagesarrivingfromthesourcesofthelogpathstothedefineddestinations,andperformsfiltering,parsing,andrewritingofthemessages.Logpathsarealsocalledlogstatements.Logstatementscanincludeother(embedded)logstatementsandjunctionstocreatecomplexlogpaths.Fordetails,seelog:Filterandroutelogmessagesusinglogpaths,flags,andfilters.

l Template:Atemplateisasetofmacrosthatcanbeusedtorestructurelogmessagesorautomaticallygeneratefilenames.Forexample,atemplatecanaddthehostnameandthedatetothebeginningofeverylogmessage.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.

l Option:Optionssetglobalparametersofsyslog-ng,liketheparametersofnameresolutionandtimezonehandling.Fordetails,seeGlobaloptionsofsyslog-ngOSE.

Fordetailsontheaboveobjects,seeTheconfigurationsyntaxindetail.

Timezones and daylight savingThesyslog-ngapplicationreceivesthetimezoneanddaylightsavinginformationfromtheoperatingsystemitisinstalledon.Iftheoperatingsystemhandlesdaylightsavingcorrectly,sodoessyslog-ng.

Thesyslog-ngapplicationsupportsmessagesoriginatingfromdifferenttimezones.Theoriginalsyslogprotocol(RFC3164)doesnotincludetimezoneinformation,butsyslog-ngprovidesasolutionbyextendingthesyslogprotocoltoincludethetimezoneinthelogmessages.Thesyslog-ngapplicationalsoenablesadministratorstosupplytimezoneinformationforlegacydeviceswhichdonotsupporttheprotocolextension.



How syslog-ng OSE assigns timezone to the message

Whensyslog-ngOSEreceivesamessage,itassignstimezoneinformationtothemessageusingthefollowingalgorithm.

1. Thesenderapplication(forexamplethesyslog-ngclient)orhostspecifiesthetimezoneofthemessages.Iftheincomingmessageincludesatimezoneitisassociatedwiththemessage.Otherwise,thelocaltimezoneisassumed.

2. Specifythetime-zone()parameterforthesourcedriverthatreadsthemessage.Thistimezonewillbeassociatedwiththemessagesonlyifnotimezoneisspecifiedwithinthemessageitself.Eachsourcedefaultstothevalueoftherecv-time-zone()globaloption.Itisnotpossibletooverrideonlythetimezoneinformationoftheincomingmessage,butsettingthekeep-timestamp()optiontonoallowssyslog-ngOSEtoreplacethefulltimestamp(timezoneincluded)withthetimethemessagewasreceived.

NOTE:

Whenprocessingamessagethatdoesnotcontaintimezoneinformation,thesyslog-ngOSEapplicationwillusethetimezoneanddaylight-savingthatwaseffectivewhenthetimestampwasgenerated.Forexample,thecurrenttimeis2011-03-11(March11,2011)intheEU/Budapesttimezone.Whendaylight-savingisactive(summertime),theoffsetis+02:00.Whendaylight-savingisinactive(wintertime)thetimezoneoffsetis+01:00.Ifthetimestampofanincomingmessageis2011-01-01,thetimezoneassociatedwiththemessagewillbe+01:00,butthetimestampwillbeconverted,because2011-01-01meantwintertimewhendaylightsavingisnotactivebutthecurrenttimezoneis+02:00.

3. Specifythetimezoneinthedestinationdriverusingthetime-zone()parameter.Eachdestinationdrivermighthaveanassociatedtimezonevalue:syslog-ngconvertsmessagetimestampstothistimezonebeforesendingthemessagetoitsdestination(fileornetworksocket).Eachdestinationdefaultstothevalueofthesend-time-zone()globaloption.

NOTE:

Amessagecanbesenttomultipledestinationzones.Thesyslog-ngapplicationconvertsthetimezoneinformationproperlyforeveryindividualdestinationzone.

CAUTION:

If syslog-ng OSE sends the message is to the destination using the legacy-syslog protocol (RFC3164) which does not support timezone information in its timestamps, the timezone information cannot be encapsulated into the sent timestamp, so syslog-ng OSE will convert the hour:min values based on the explicitly specified timezone.

4. Ifthetimezoneisnotspecified,localtimezoneisused.



5. Whenmacroexpansionsareusedinthedestinationfilenames,thelocaltimezoneisused.(Also,ifthetimestampofthereceivedmessagedoesnotcontaintheyearofthemessage,syslog-ngOSEusesthelocalyear.)

A note on timezones and timestampsIf the clients run syslog-ng, then use the ISO timestamp, because it includestimezone information. Thatway you do not need to adjust the recv-time-zone()parameter of syslog-ng.

Ifyouwantsyslog-ngtooutputtimestampsinUnix(POSIX)timeformat,usetheS_UNIXTIMEandR_UNIXTIMEmacros.Youdonotneedtochangeanyofthetimezonerelatedparameters,becausethetimestampinformationofincomingmessagesisconvertedtoUnixtimeinternally,andUnixtimeisatimezone-independenttimerepresentation.(Actually,UnixtimemeasuresthenumberofsecondselapsedsincemidnightofCoordinatedUniversalTime(UTC)January1,1970,butdoesnotcountleapseconds.)

Product licensing

Startingwithversion3.2,thesyslog-ngOpenSourceEditionapplicationislicensedunderacombinedLGPL+GPLlicense.Thecoreofsyslog-ngOSEislicensedundertheGNULesserGeneralPublicLicenseVersion2.1license,whiletherestofthecodebaseislicensedundertheGNUGeneralPublicLicenseVersion2license.

NOTE:

Practically,thecodestoredunderthelibdirectoryofthesourcecodepackageisunderLGPL,therestisGPL.

FordetailsabouttheLGPLandGPLlicenses,seeGNULesserGeneralPublicLicenseandGNUGeneralPublicLicense,respectively.

High availability supportMultiplesyslog-ngserverscanberuninfail-overmode.Thesyslog-ngapplicationdoesnotincludeanyinternalsupportforthis,asclusteringsupportmustbeimplementedontheoperatingsystemlevel.AtoolthatcanbeusedtocreateUNIXclustersisHeartbeat(fordetails,seethispage).

The structure of a log messageThefollowingsectionsdescribethestructureoflogmessages.Currentlytherearetwostandardsyslogmessageformats:



http://www.linux-ha.org/wiki/Main_Page/

l TheoldstandarddescribedinRFC3164(alsocalledtheBSD-syslogorthelegacy-syslogprotocol):seeBSD-syslogorlegacy-syslogmessages

l ThenewstandarddescribedinRFC5424(alsocalledtheIETF-syslogprotocol):seeIETF-syslogmessages

l TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesbetweensyslog-ngnodes:seeEnterprise-widemessagemodel(EWMM)

l Howmessagesarerepresentedinsyslog-ngOSE:seeMessagerepresentationinsyslog-ngOSE.

BSD-syslog or legacy-syslog messagesThissectiondescribestheformatofasyslogmessage,accordingtothelegacy-syslogorBSD-syslogprotocol.Asyslogmessageconsistsofthefollowingparts:

l PRI

l HEADER

l MSG

Thetotalmessagecannotbelongerthan1024bytes.

Thefollowingisasamplesyslogmessage:

Feb 25 14:09:07 webserver syslogd: restart

Themessagecorrespondstothefollowingformat:

timestamp hostname application: message

Thedifferentpartsofthemessageareexplainedinthefollowingsections.

NOTE:

Thesyslog-ngapplicationsupportslongermessagesaswell.Fordetails,seethelog-msg-size()optioninGlobaloptions.However,itisnotrecommendedtoenablemessageslargerthanthepacketsizewhenusingUDPdestinations.

The PRI message part

ThePRIpartofthesyslogmessage(knownasPriorityvalue)representstheFacilityandSeverityofthemessage.Facilityrepresentsthepartofthesystemsendingthemessage,whileseveritymarksitsimportance.ThePriorityvalueiscalculatedbyfirstmultiplyingtheFacilitynumberby8andthenaddingthenumericalvalueoftheSeverity.Thepossiblefacilityandseverityvaluesarepresentedbelow.

NOTE:

Facilitycodesmayslightlyvarybetweendifferentplatforms.Thesyslog-ngapplicationacceptsfacilitycodesasnumericalvaluesaswell.



https://tools.ietf.org/search/rfc3164https://tools.ietf.org/search/rfc3164

Numerical Code Facility

0 kernelmessages

1 user-levelmessages

2 mailsystem

3 systemdaemons

4 security/authorizationmessages

5 messagesgeneratedinternallybysyslogd

6 lineprintersubsystem

7 networknewssubsystem

8 UUCPsubsystem

9 clockdaemon


11 FTPdaemon

12 NTPsubsystem

13 logaudit

14 logalert

15 clockdaemon

16-23 locallyusedfacilities(local0-local7)

Table 1: syslog Message Facilities

Thefollowingtableliststheseverityvalues.

Numerical Code Severity

0 Emergency:systemisunusable

1 Alert:actionmustbetakenimmediately

2 Critical:criticalconditions

3 Error:errorconditions

4 Warning:warningconditions

5 Notice:normalbutsignificantcondition

6 Informational:informationalmessages

7 Debug:debug-levelmessages

Table 2: syslog Message Severities



The HEADER message partTheHEADERpartcontainsatimestampandthehostname(withoutthedomainname)ortheIPaddressofthedevice.ThetimestampfieldisthelocaltimeintheMmm dd hh:mm:ssformat,where:

l MmmistheEnglishabbreviationofthemonth:Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec.

l ddisthedayofthemonthontwodigits.Ifthedayofthemonthislessthan10,thefirstdigitisreplacedwithaspace.(ForexampleAug 7.)

l hh:mm:ssisthelocaltime.Thehour(hh)isrepresentedina24-hourformat.Validentriesarebetween00and23,inclusive.Theminute(mm)andsecond(ss)entriesarebetween00and59inclusive.

NOTE:

Thesyslog-ngapplicationsupportsothertimestampformatsaswell,likeISO,orthePIXextendedformat.Fordetails,seethets-format()optioninGlobaloptions.

The MSG message part

TheMSGpartcontainsthenameoftheprogramorprocessthatgeneratedthemessage,andthetextofthemessageitself.TheMSGpartisusuallyinthefollowingformat:program[pid]: message text.

IETF-syslog messagesThissectiondescribestheformatofasyslogmessage,accordingtotheIETF-syslogprotocol.Asyslogmessageconsistsofthefollowingparts:

l HEADER(includesthePRIaswell)

l STRUCTURED-DATA

l MSG

Thefollowingisasamplesyslogmessage(source:https://tools.ietf.org/html/rfc5424):

1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

Themessagecorrespondstothefollowingformat:

VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

Inthisexample,theFacilityhasthevalueof4,severityis2,soPRIis34.TheVERSIONis1.Themessagewascreatedon11October2003at10:14:15pmUTC,3millisecondsintothenextsecond.Themessageoriginatedfromahostthatidentifiesitselfas"mymachine.example.com".TheAPP-NAMEis"su"andthePROCIDisunknown.The



https://tools.ietf.org/html/rfc5424https://tools.ietf.org/html/rfc5424

MSGIDis"ID47".TheMSGis"'suroot'failedforlonvick...",encodedinUTF-8.TheencodingisdefinedbytheBOM:

Thebyteordermark(BOM)isaUnicodecharacterusedtosignalthebyte-orderofthemessagetext.

ThereisnoSTRUCTURED-DATApresentinthemessage,thisisindicatedby"-"intheSTRUCTURED-DATAfield.TheMSGis"'suroot'failedforlonvick...".

TheHEADERpartofthemessagemustbeinplainASCIIformat,theparametervaluesoftheSTRUCTURED-DATApartmustbeinUTF-8,whiletheMSGpartshouldbeinUTF-8.Thedifferentpartsofthemessageareexplainedinthefollowingsections.

The PRI message part

ThePRIpartofthesyslogmessage(knownasPriorityvalue)representstheFacilityandSeverityofthemessage.Facilityrepresentsthepartofthesystemsendingthemessage,whileseveritymarksitsimportance.ThePriorityvalueiscalculatedbyfirstmultiplyingtheFacilitynumberby8andthenaddingthenumericalvalueoftheSeverity.Thepossiblefacilityandseverityvaluesarepresentedbelow.

NOTE:

Facilitycodesmayslightlyvarybetweendifferentplatforms.Thesyslog-ngapplicationacceptsfacilitycodesasnumericalvaluesaswell.


0 kernelmessages

1 user-levelmessages

2 mailsystem

3 systemdaemons


5 messagesgeneratedinternallybysyslogd

6 lineprintersubsystem

7 networknewssubsystem

8 UUCPsubsystem

9 clockdaemon


11 FTPdaemon

Table 3: syslog Message Facilities




12 NTPsubsystem

13 logaudit

14 logalert

15 clockdaemon

16-23 locallyusedfacilities(local0-local7)

Thefollowingtableliststheseverityvalues.

Numerical Code Severity

0 Emergency:systemisunusable

1 Alert:actionmustbetakenimmediately

2 Critical:criticalconditions

3 Error:errorconditions

4 Warning:warningconditions

5 Notice:normalbutsignificantcondition

6 Informational:informationalmessages

7 Debug:debug-levelmessages

Table 4: syslog Message Severities

The HEADER message partTheHEADERpartcontainsthefollowingelements:

l VERSION: Version number of the syslog protocol standard. Currently this canonly be 1.

l ISOTIMESTAMP:ThetimewhenthemessagewasgeneratedintheISO8601compatiblestandardtimestampformat(yyyy-mm-ddThh:mm:ss+-ZONE),forexample:2006-06-13T15:58:00.123+01:00.

l HOSTNAME:Themachinethatoriginallysentthemessage.

l APPLICATION:Thedeviceorapplicationthatgeneratedthemessage

l PID:TheprocessnameorprocessIDofthesyslogapplicationthatsentthemessage.It isnotnecessarilytheprocessIDoftheapplicationthatgeneratedthemessage.

l MESSAGEID:TheIDnumberofthemessage.



NOTE:

Thesyslog-ngapplicationsupportsothertimestampformatsaswell,likeISO,orthePIXextendedformat.ThetimestampusedintheIETF-syslogprotocolisderivedfromRFC3339,whichisbasedonISO8601.Fordetails,seethets-format()optioninGlobaloptions.

Thesyslog-ngOSEapplicationwilltruncatethefollowingfields:

l IfAPP-NAMEislongerthan48charactersitwillbetruncatedto48characters.

l IfPROC-IDislongerthan128charactersitwillbetruncatedto128characters.

l IfMSGIDislongerthan32charactersitwillbetruncatedto32characters.

l IfHOSTNAMEislongerthan255charactersitwillbetruncatedto255characters.

The STRUCTURED-DATA message part

TheSTRUCTURED-DATAmessagepartmaycontainmeta-informationaboutthesyslogmessage,orapplication-specificinformationsuchastrafficcountersorIPaddresses.STRUCTURED-DATAconsistsofdatablocksenclosedinbrackets([]).EveryblockincludestheIDoftheblock,andoneormorename=valuepairs.Thesyslog-ngapplicationautomaticallyparsestheSTRUCTURED-DATApartofsyslogmessages,whichcanbereferencedinmacros(fordetails,seeMacrosofsyslog-ngOSE).AnexampleSTRUCTURED-DATAblocklookslike:

[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]

The MSG message part

TheMSGpartcontainsthetextofthemessageitself.TheencodingofthetextmustbeUTF-8iftheBOM1characterispresentinthemessage.IfthemessagedoesnotcontaintheBOMcharacter,theencodingistreatedasunknown.UsuallymessagesarrivingfromlegacysourcesdonotincludetheBOMcharacter.CRLFcharacterswillnotberemovedfromthemessage.

Enterprise-wide message model (EWMM)ThefollowingsectiondescribesthestructureoflogmessagesusingtheEnterprise-widemessagemodelorEWMMmessageformat.

TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesfromtheinitialreceivingsyslog-ngcomponentrightuptothecentrallogserver,through

1Thebyteordermark(BOM)isaUnicodecharacterusedtosignalthebyte-orderofthemessagetext.



anynumberofhops.Itdoesnotmatterifyouparsethemessagesontheclient,onarelay,oronthecentralserver,theirstructuredresultswillbeavailablewhereyoustorethemessages.Optionally,youcanalsoforwardtheoriginalrawmessageasthefirstsyslog-ngcomponentinyourinfrastructurehasreceivedit,whichisimportantifyouwanttoforwardamessageforexampletoaSIEMsystem.Tomakeuseoftheenterprise-widemessagemodel,youhavetousethesyslog-ng()destinationonthesenderside,andthedefault-network-drivers()sourceonthereceiverside.

ThefollwingisasamplelogmessageinEWMMformat.

1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - - {"MESSAGE":"Oct 11 22:14:15 mymachine su: 'su root' failed for username on /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}

Themessagehasthefollowingparts.

l TheheaderofthecomplieswiththeRFC5424messageformat,wherethePROGRAMfieldissetto@syslog-ng,andtheSDATAfieldisempty.

l TheMESSAGEpartisinJSONformat,andcontainstheactualmessage,aswellasanyname-valuepairsthatsyslog-ngOSEhasattachedtoorextractedfromthemessage.The${._TAGS}fieldcontainstheidentifierofthesyslog-ngsourcethathasoriginallyreceivedthemessageonthefirstsyslog-ngnode.

TosendamessageinEWMMformat,youcanusethesyslog-ng()destinationdriver,ortheformat-ewmm()templatefunction.

ToreceiveamessageinEWMMformat,youcanusethedefault-destination-drivers()sourcedriver,ortheewmm-parser()parser.

Message representation in syslog-ng OSEWhenthesyslog-ngOSEapplicationreceivesamessage,itautomaticallyparsesthemessage.Thesyslog-ngOSEapplicationcanautomaticallyparselogmessagesthatconformtotheRFC3164(BSDorlegacy-syslog)ortheRFC5424(IETF-syslog)messageformats.Ifsyslog-ngOSEcannotparseamessage,itresultsinanerror.

TIP:

Incaseyouneedtorelaymessagesthatcannotbeparsedwithoutanymodificationsorchanges,usetheflags(no-parse)optioninthesourcedefinition,andatemplatecontainingonlythe${MESSAGE}macrointhedestinationdefinition.

Toparsenon-syslogmessages,forexample,JSON,CSV,orothermessages,youcanusethebuilt-inparsersofsyslog-ngOSE.Fordetails,seeparser:Parseandsegmentstructuredmessages.

Aparsedsyslogmessagehasthefollowingparts.



l Timestamps

Twotimestampsareassociatedwitheverymessage:oneisthetimestampcontainedwithinthemessage(thatis,whenthesendersentthemessage),theotheristhetimewhensyslog-ngOSEhasactuallyreceivedthemessage.

l Severity

Theseverityofthemessage.

l Facility

Thefacilitythatsentthemessage.

l Tags

Customtextlabelsaddedtothemessagethataremainlyusedforfiltering.Noneofthecurrentmessagetransportprotocolsaddstagstothelogmessages.Tagscanbeaddedtothelogmessageonlywithinsyslog-ngOSE.Thesyslog-ngOSEapplicationautomaticallyaddstheidofthesourceasatagtotheincomingmessages.Othertagscanbeaddedtothemessagebythepatterndatabase,orusingthetags()optionofthesource.

l IP address of the sender

TheIPaddressofthehostthatsentthemessage.NotethattheIPaddressofthesenderisahardmacroandcannotbemodifiedwithinsyslog-ngOSEbuttheassociatedhostnamecanbemodified,forexample,usingrewriterules.

l Hard macrosHardmacroscontaindatathatisdirectlyderivedfromthelogmessage,forexample,the${MONTH}macroderivesitsvaluefromthetimestamp.Themostimportantconsiderationwithhardmacrosisthattheyareread-only,meaningtheycannotbemodifiedusingrewriterulesorothermeans.

l Soft macrosSoftmacros(sometimesalsocalledname-valuepairs)areeitherbuilt-inmacrosautomaticallygeneratedfromthelogmessage(forexample,${HOST}),orcustomuser-createdmacrosgeneratedbyusingthesyslog-ngpatterndatabaseoraCSV-parser.TheSDATAfieldsofRFC5424-formattedlogmessagesbecomesoftmacrosaswell.Incontrastwithhardmacros,softmacrosarewritableandcanbemodifiedwithinsyslog-ngOSE,forexample,usingrewriterules.



NOTE:

Itisalsopossibletosetthevalueofbuilt-insoftmacrosusingparsers,forexample,tosetthe${HOST}macrofromthemessageusingacolumnofaCSV-parser.

Thedataextractedfromthelogmessagesusingnamedpatternparsersinthepatterndatabasearealsosoftmacros.

TIP:

Forthelistofhardandsoftmacros,seeHardvs.softmacros.

Message size and encodingInternally,syslog-ngOSErepresentseverymessageasUTF-8.Themaximallengthofthelogmessagesislimitedbythelog-msg-size()option:ifamessageislongerthanthisvalue,syslog-ngOSEtruncatesthemessageatthelocationitreachesthelog-msg-size()value,anddiscardstherestofthemessage.

Whenencodingissetinasource(usingtheencoding()option)andthemessageislonger(inbytes)thanlog-msg-size()inUTF-8representation,syslog-ngOSEsplitsthemessageatanundefinedlocation(becausetheconversionbetweendifferentencodingsisnottrivial).

Structuring macros, metadata, and other value-pairs

Available in syslog-ng OSE 3.3 and later.

Thesyslog-ngOSEapplicationallowsyoutoselectandconstructname-valuepairsfromanyinformationalreadyavailableaboutthelogmessage,orextractedfromthemessageitself.Youcandirectlyusethisstructuredinformation,forexample,inthefollowingplaces:

l amqp()destination

l format-welf()templatefunction

l mongodb()destination

l stomp()destination

l orinotherdestinationsusingtheformat-json()templatefunction.

Whenusingvalue-pairs,therearethreewaystospecifywhichinformation(thatis,macrosorothername-valuepairs)toincludeintheselection.

l Selectgroupsofmacrosusingthescope()parameter,andoptionallyremovecertainmacrosfromthegroupusingtheexclude()parameter.



l Listspecificmacrostoincludeusingthekey()parameter.

l Definenewname-valuepairstoincludeusingthepair()parameter.

Theseparametersaredetailedinvalue-pairs().

Specifying data types in value-pairsBydefault,syslog-ngOSEhandleseverydataasstrings.However,certaindestinationsanddataformats(forexample,SQL,MongoDB,JSON,AMQP)supportothertypesofdataaswell,forexample,numbersordates.Thesyslog-ngOSEapplicationallowsyoutospecifythedatatypeintemplates(thisisalsocalledtype-hinting).Ifthedestinationdriversupportsdatatypes,itconvertstheincomingdatatothespecifieddatatype.Forexample,thisallowsyoutostoreintegernumbersasnumbersinMongoDB,insteadofstrings.

CAUTION:

Hazard of data loss! If syslog-ng OSE cannot convert the data into the specified type, an error occurs, and syslog-ng OSE drops the message by default. To change how syslog-ng OSE handles data-conversion errors, see on-error().

Tousetype-hinting,enclosethemacroortemplatecontainingthedatawiththetype:(""),forexample:int("$PID").

Currently the mongodb() destination and the format-json template function supportsdata types.

Example: Using type-hintingThefollowingexamplestorestheMESSAGE,PID,DATE,andPROGRAMfieldsofalogmessageinaMongoDBdatabase.TheDATEandPIDpartsarestoredasnumbersinsteadofstrings.

mongodb( value-pairs(pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM")) pair("message", "$MESSAGE")) ) );

ThefollowingexampleformatsthesamefieldsintoJSON.

$(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)

Thesyslog-ngOSEapplicationcurrentlysupportsthefollowingdata-types.



l boolean:Convertsthedatatoabooleanvalue.Anythingthatbeginswithator1isconvertedtotrue,anythingthatbeginswithanfor0isconvertedtofalse.

l datetime: Use it onlywithUNIX timestamps, anything elsewill likely result inan error. Thismeans that currently you can use only the $UNIXTIMEmacro forthis purpose.

l double:Afloating-pointnumber.

l literal:Thedataasaliteralstring,withoutaddinganyquotesorescapecharacters.

l intorint32:32-bitinteger.

l int64:64-bitinteger.

l string:Thedataasastring.

value-pairs()

Type: parameterlistofthevalue-pairs()option

Default: empty string

Description:Thevalue-pairs()optionallowsyoutoselectspecificinformationaboutamessageeasilyusingpredefinedmacrogroups.Theselectedinformationisrepresentedasname-valuepairsandcanbeusedformattedtoJSONformat,ordirectlyusedinamongodb()destination.

Example: Using the value-pairs() optionThefollowingexampleselectseveryavailableinformationaboutthelogmessage,exceptforthedate-relatedmacros(R_*andS_*),selectsthe.SDATA.meta.sequenceIdmacro,anddefinesanewvalue-paircalledMSGHDRthatcontainstheprogramnameandPIDoftheapplicationthatsentthelogmessage.

value-pairs( scope(nv_pairs core syslog all_macros selected_macros everything) exclude("R_*") exclude("S_*") key(".SDATA.meta.sequenceId") pair("MSGHDR" "$PROGRAM[$PID]: ") )

Thefollowingexampleselectsthesameinformationasthepreviousexample,butconvertsitintoJSONformat.

$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \



--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \ --pair MSGHDR="$PROGRAM[$PID]: ")

NOTE:

Everymacroisincludedintheselectiononlyonce,butredundantinformationmayappearifmultiplemacrosincludethesameinformation(forexample,includingseveraldate-relatedmacrosintheselection).

Thevalue-pairs()optionhasthefollowingparameters.Theparametersareevaluatedinthefollowingorder:

1. scope()

2. exclude()

3. key()

4. pair()

exclude()

Type: Space-separatedlistofmacrostoremovefromtheselectioncreatedusingthescope()option.

Default: emptystring

Description:Thisoptionremovesthespecifiedmacrosfromtheselection.Useittoremoveunneededmacrosselectedusingthescope()parameter.

Forexample,thefollowingexampleremovestheSDATAmacrosfromtheselection.

value-pairs( scope(rfc5424 selected_macros) exclude(".SDATA*") )

Thenameofthemacrotoremovecanincludewildcards(*, ?).Regularexpressionsarenotsupported.

key()

Type: Space-separatedlistofmacrostobeincludedinselection


Description:Thisoptionselectsthespecifiedmacros.TheselectedmacroswillbeincludedasMACRONAME = MACROVALUE,thatisusingkey("HOST")willresultinHOST = $HOST.Youcanusewildcards(*, ?)toselectmultiplemacros.Forexample:

value-pairs(



scope(rfc3164) key("HOST") )

value-pairs( scope(rfc3164) key("HOST", "PROGRAM") )

pair()

Type: namevaluepairsin"" ""format


Description:Thisoptiondefinesanewname-valuepairtobeincludedinthemessage.Thevaluepartcanincludemacros,templates,andtemplatefunctionsaswell.Forexample:

value-pairs( scope(rfc3164) pair("TIME" "$HOUR:$MIN") pair("MSGHDR" "$PROGRAM[$PID]: ") )

rekey()

Type: ,


Description:Thisoptionallowsyoutomanipulateandmodifythenameofthevalue-pairs.Youcandefinetransformations,whichareareappliedtotheselectedname-valuepairs.Thefirstparameteroftherekey()optionisaglobpatternthatselectsthename-valuepairstomodify.Ifyouomitthepattern,thetransformationsareappliedtoeverykeyofthescope.Fordetailsonglobs,seeglob.

Ifyouwanttomodifythenamesofseveralmessagefields,seealsomap-value-pairs:Renamevalue-pairstonormalizelogs.

l Ifrekey()isusedwithinakey()option,thename-valuepairsspecifiedintheglobofthekey()optionaretransformed.

l Ifrekey()isusedoutsidethekey()option,everyname-valuepairofthescope()istransformed.

Thefollowingtransformationsareavailable:

l add-prefix("")

l Addsthespecifiedprefixtoeveryname.Forexample,rekey( add-prefix("my-



prefix."))

l replace-prefix("", "")

l Replacesasubstringatthebeginningofthekeywithanotherstring.Onlyprefixescanbereplaced.Forexample,replace-prefix(".class", ".patterndb")changesthebeginningtag.classto.patterndb

Thisoptionwascalledreplace()insyslog-ngOSEversion3.4.

l shift("")

l Cutsthespecifiednumberofcharactersfromthebeginningofthename.

Example: Using the rekey() option

Thefollowingsampleselectseveryvalue-pairthatbeginswith.cee.,deletesthisprefixbycutting4charactersfromthenames,andaddsanewprefix(events.).

value-pairs( key(".cee.*" rekey( shift(4) add-prefix("events.") ) ) )

Therekey()optioncanbeusedwiththeformat-jsontemplate-functionaswell,usingthefollowingsyntax:

$(format-json --rekey .cee.* --add-prefix events.)

scope()

Type: space-separatedlistofmacrogroupstoincludeinselection


Description:Thisoptionselectspredefinedgroupsofmacros.Thefollowinggroupsareavailable:

l nv-pairs:Everysoftmacro(name-valuepair)associatedwiththemessage,excepttheonesthatstartwithadot(.)character.Macrosstartingwithadotcharacteraregeneratedwithinsyslog-ngOSEandarenotoriginallypartofthemessage,thereforearenotincludedinthisgroup.

l dot-nv-pairs:Everysoftmacro(name-valuepair)associatedwiththemessagewhichstartswithadot(.)character.Forexample,.classifier.rule_idand.sdata.*.Macrosstartingwithadotcharacteraregeneratedwithinsyslog-ngOSEandarenotoriginallypartofthemessage.



l all-nv-pairs:Includeeverysoftmacro(name-valuepair).Equivalenttousingbothnv-pairsanddot-nv-pairs.

l rfc3164:ThemacrosthatcorrespondtotheRFC3164(legacyorBSD-syslog)messageformat:$FACILITY,$PRIORITY,$HOST,$PROGRAM,$PID,$MESSAGE,and$DATE.

l rfc5424:ThemacrosthatcorrespondtotheRFC5424(IETF-syslog)messageformat:$FACILITY,$PRIORITY,$HOST,$PROGRAM,$PID,$MESSAGE,$MSGID,$R_DATE,andthemetadatafromthestructured-data(SDATA)partofRFC5424-formattedmessages,thatis,everymacrothatstartswith.SDATA..

Therfc5424groupalsohasthefollowingalias:syslog-proto.Notethatthevalueof$R_DATEwillbelistedundertheDATEkey.

Therfc5424groupdoesnotcontainanymetadataaboutthemessage,onlyinformationthatwaspresentintheoriginalmessage.Toincludethemostcommonlyusedmetadata(forexample,the$SOURCEIPmacro),usetheselected-macrosgroupinstead.

l all-macros:Includeeveryhardmacro.Thisgroupismainlyusefulfordebugging,asitcontainsredundantinformation(forexample,thedate-relatedmacrosincludethedate-relatedinformationseveraltimesinvariousformats).

l selected-macros:Includethemacrosoftherfc3164groups,andthemostcommonlyusedmetadataaboutthelogmessage:the$TAGS,$SOURCEIP,and$SEQNUMmacros.

l sdata:Themetadatafromthestructured-data(SDATA)partofRFC5424-formattedmessages,thatis,everymacrothatstartswith.SDATA.

l everything:Includeeveryhardandsoftmacros.Thisgroupismainlyusefulfordebugging,asitcontainsredundantinformation(forexample,thedate-relatedmacrosincludethedate-relatedinformationseveraltimesinvariousformats).

Forexample:

value-pairs( scope(rfc3164 selected-macros) )

Things to consider when forwarding messages between syslog-ng OSE hosts

Whenyousendyourlogmessagesfromasyslog-ngOSEclientthroughthenetworktoasyslog-ngOSEserver,youcanusedifferentprotocolsandoptions.Everycombinationhasitsadvantagesanddisadvantages.Themostimportantthingistousematchingprotocolsandoptions,sotheserverhandlestheincominglogmessagesproperly.

Insyslog-ngOSEyoucanchangemanyaspectsofthenetworkcommunication.Firstofall,thereisthestructureofthemessagesitself.Currently,syslog-ngOSEsupportstwostandardsyslogprotocols:theBSD(RFC3164)andthesyslog(RFC5424)messageformat.


The co

Administration Guide - support-public.cfm.quest.com · Introduction to syslog-ng 30 What syslog-ng...

Documents

Transcript of Administration Guide - support-public.cfm.quest.com · Introduction to syslog-ng 30 What syslog-ng...