BRKNMS-2031

© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public

Syslog Design, Methodology and Best Practices Clayton Dukes

Cisco Systems, Advanced Services

BRKNMS-2031

@cdnw

2


Session Abstract

“This session will help define the design and methodology for implementing a robust syslog solution using open source tools on Linux platforms.

It provides leading practices for deployment of a set of tools and applications to support effective collection, storage, and analysis of syslog messages.

This session provides examples using messages from Cisco IOS Software, but is applicable to all other syslog message types and general event management.”

Tweet about this session: #SYSLOG (but only if you’re saying how awesome it is )

3


Meet the Engineer—Clayton Dukes

Joined Cisco in 2005 as a Network Consulting Engineer

Background was in routing and switching for 6+ years

Moved to Network Management/OSS automation around 1998

Frequent speaker at Networkers

Author of several Cisco.com whitepapers on network management architectures and large-scale syslog deployment

Self Portrait

4


Topics

Why Syslog?

Syslog Basics

The Syslog Message

Relevant IOS commands

Syslog vs. SNMP

Management Techniques/Methodologies

Syslog Analysis

Syslog Architectures

Analysis Tools

Implementation Walk-Through (Using Open Source Tools)

5


Topics

Why Syslog?

Syslog Basics

The Syslog Message

Relevant IOS commands

Syslog vs. SNMP

Management Techniques/Methodologies

Syslog Analysis


Analysis Tools

Implementation Walk-Through (Using Open Source Tools)

6

“Cat6500 IOS 12.2(18)SXF contains about 90 SNMP traps, but has over 6000 syslog event messages.”

Source: Me!

Not this guy


Why Syslog?

Proactive Syslog management benefits both operations personnel and the company as a whole. Successful event management provides:

Reduced downtime through operational effectiveness

Improved Incident Management through real-time detection and self-remediation

Reduced volume of incidents through proper problem management

Reduced severity of business interruptions

Proactive measures to reduce the need for post-mortem troubleshooting

8


Why Syslog?

%CDP-4-DUPLEXMISMATCH: ‒ Duplex Full/Half between connections

%ENVMON-3-FAN_FAILED: ‒ Fan failure – may cause overheating

%ENVMON-3-OVER_HEAT: ‒ Device temperature is > 60C (140F)

%PQUICC-5-COLL: ‒ Excessive collisions - broken or unterminated Ethernet cable

%SYS-3-CPUHOG: ‒ The indicated process has run for too long a period of time without relinquishing

the processor

Ignoring Syslog Doesn’t Mean Your Network Isn’t Failing or Degraded

9


Downtime = Lost Revenue Reducing Downtime Through Proactive Problem Management Reduces Operational Cost

0.09

0.09

0.1

1.2

2.6

4.5

0 1 2 3 4 5

Transportation

Retail

E-Commerce

Media

Banking

Brokerage

Industry Cost of Downtime

Revenue Loss Per Hour (in Millions of Dollars)

10


Syslog Basics

11


The Syslog Protocol

Syslog is a client/server protocol ‒ The syslog sender sends a small (less than 1KB) text message to the syslog

receiver. The receiver is commonly called "syslogd", "syslog daemon" or "syslog server". Syslog messages (RFC 3164) can be sent via UDP (514) and/or TCP*. The data is typically sent in clear text.

Originally developed in the 1980s by Eric Allman as part of the Sendmail project, syslog is now standardized within the syslog working group of the IETF Syslog is supported by a wide variety of devices and receivers across

multiple platforms. Because of this, it can be used to integrate log data from disparate systems into a central repository for real-time and historical analysis.

* TCP support is available with some syslog daemons, such as syslog-ng or rsyslog as well as Cisco IOS Software Releases after 12.4(11)T, 12.2(33)SRB, 12.2(33)SB, and Cisco IOS XE Release 2.1 12.2(33)SXI.

12


RFC’s

13


RFC 3164

http://www.ietf.org/rfc/rfc3164.txt

Written in August 2001 by Chris Lonvick/Cisco Systems

Obsoleted by RFC 5424 (although most vendors still use 3164)

14


RFC 5424

http://tools.ietf.org/html/rfc5424

Written in March 2009 by Rainer Gerhards/Adiscon

Provides a method for adding Structured Data to a message which is meant to make it easier to parse and analyze the message once received.

Has many advantages over RFC 3164, but is also a lot more complex.

Meant to replace RFC 3164, but it doesn’t seem to get used much – perhaps because of the complexity?

15


The Syslog Message

Every syslog message should contain five distinct fields with the following information:

Facility

Severity

Hostname

Timestamp

Message

16


Syslog Message Facility

Syslog messages are broadly categorized on the basis of the sources that generate them such as OS, process or application and are represented in integers ranging from 0-23, Cisco devices use the local facility ranges 16-23 (local0 – local7)

By default, Cisco IOS devices, CatOS switches, and VPN 3000 Concentrators use facility local7 while Cisco Firewalls use local4

17


Syslog Message—Severity

0 - Emergency: System Is Unusable

1 - Alert: Action Must Be Taken Immediately

2 - Critical: Critical Conditions

3 - Error: Error Conditions

4 - Warning: Warning Conditions

5 - Notice: Normal But Significant Condition

6 - Informational: Informational Messages

7 - Debug: Debug-Level Messages

The log source or facility (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7

Leading Practice

Network Devices Should Log Levels 0-6

Level 7 Should Be Used for Console Troubleshooting

18


Syslog Message—Hostname

The hostname field consists of the host name (as configured on the host itself) or the IP address

Note: Don’t be confused by “host name” and “hostname”. “Hostname” Is typically associated with a DNS lookup. If the syslog message contains a “host name”, it may be (and often is) different than the actual DNS hostname of the device.

19


Syslog Message—Timestamp

The local time, in MMM DD HH:MM:SS format, of the device when the message was generated

The * and . characters preceding a syslog message are indicators of a problem with NTP. ‒ * Means that time is not authoritative: the software clock is not

in sync or has never been set.

‒ . Means that time is authoritative, but NTP is not synchronized: the software clock was in sync, but has since lost contact with all configured NTP servers

*Jun 28 08:50:47.359 EDT: %SYS-5-CONFIG_I: Configured from console by Skeeter McGillicutty (10.10.86.123)

For the Timestamp Information to Be

Accurate, It Is Good Administrative Practice

to Configure All the Devices to Use the

Network Time Protocol (NTP)

Leading Practice

20


Syslog Message—Message Text

This is the text of the syslog message, along with some additional information about the process that generated it

Messages generated by most Cisco devices begin with a percent sign (%) and use the following format:

‒%FACILITY-SEVERITY-MNEMONIC: Message-text

‒ The mnemonic is a device-specific code that uniquely identifies the message such as “up”, “down”, “changed”, “config”, etc.

‒ The term “facility” in Cisco Mnemonics are not the same as the IETF definition of “facility” (such as local7). Cisco facilities are a free-form method of identifying the source message type such as SYS, IP, LDP, L2, MEM, FILESYS, DOT11, LINEPROTO, etc. (the list is very large)

*Sep 16 08:50:47.359 EDT: %SYS-5-CONFIG_I: Configured from console by vty0 (172.18.86.123)

21


Relevant IOS Commands service timestamps debug datetime localtime show-timezone

service timestamps log datetime localtime show-timezone

clock timezone GMT 0

!

logging source-interface Loopback0

logging buffered 65536

logging host 192.168.100.20



logging trap informational

!

ntp server 143.232.55.5

ntp server 204.34.198.40

ntp peer 192.168.100.2

ntp peer 192.168.100.3

ntp update-calendar

22


Configuration Command Detail—Time service timestamps debug datetime localtime show-timezone

service timestamps log datetime localtime show-timezone

clock timezone GMT 0

Time stamps can be added to either debugging or logging messages independently

Datetime

Adds time stamps in the format MMM DD HH:MM:SS, indicating the date and time according to the system clock

Uptime

Adds time stamps in the format HHHH:MM:SS, indicating the time since the system was rebooted

23


Configuration Command Detail—Logging

logging source-interface loopback0

The “logging source-interface” command instructs the system to generate logging to the remote system from this source interface

Ensures that all messages appear to come from the same IP and makes it easier to track in the destination syslog receiver

Allows you to create a DNS entry for that source interface

24


Logging (Cont.)

logging buffered 65536

Used to reserve a memory buffer for logging to the console of the device

Since today’s devices have plenty of memory, feel free to set this number higher than the old 16k buffer, but be aware that there is a point of diminishing returns

The typical recommendation is to have 256k buffers on core devices and 64k elsewhere

Note: Console refers to the output of the screen when attached to the device either by serial or via telnet/ssh using the “Terminal Monitor” command.

25


Logging (Cont.)

logging host <ip address 1>



Sets the remote syslog daemon to send messages to

Use a maximum of four syslog servers

The syslog server can then be configured to forward messages to other Network Management Systems

Devices Should Be Configured with a Maximum of 3-4

Destination Servers

Leading Practice

26


Logging (Cont.)

logging trap informational

Sets the syslog server logging level (emerg through debug)

The term “Trap” here has nothing to do with SNMP Traps

“Trap” in this context is simply telling the device to log the specified severity levels.

It’s NOT a Trap!

27


Configuration Command Detail—NTP

ntp server <ip address 1> ntp server <ip address 2> ntp peer <ip address 3> ntp peer <ip address 4> ntp update-calendar

The “ntp update-calendar” command is used to synchronize the time of the internal clock with the clock of the NTP reference server

28


NTP Recommendations

Use a minimum of two reference clocks (GPS and Internet derived are popular)

“Peer” time between the reference clocks

29


Additional/Useful Logging Statements

logging count

Enables local device metrics on syslog messages

http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_logging_count_ps6350_TSD_Products_Configuration_Guide_Chapter.html#wp1025043

30


Syslog vs. SNMP Notifications

Can’t I just turn on SNMP traps? ‒ The simple answer is: no. In general there are significantly

more syslog messages available within IOS as compared to SNMP Trap messages

‒ Cat6500 IOS 12.2(18)SXF contains about 90 SNMP traps, but has over 6000 syslog event messages

If You Had to Pick SNMP Traps or Syslog, Go with Syslog; However, a Truly Robust and Full-Featured Event Management Solution Would Take Advantage of All Fault Indicators

31


Management Techniques #SYSLOG

32


Traditional Syslog Management

File-based storage ‒Traditionally, syslog daemons would store all incoming messages to one or more files for later parsing. This led to a very reactive use of syslog for after-the-fact troubleshooting and could not scale beyond very few devices

grep and tail ‒Great tools in their own right, but hardly useful for scraping through gigabytes of log data…better tools are necessary

33


A Better Way

Store all incoming messages in a database

Provides speed and scalability

Capable of storing thousands of messages per second

Allows for trending and metrics

34


How?

Syslog-ng ‒“An open source implementation of the syslog protocol for UNIX and UNIX-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport”

syslog-ng is a unix/linux daemon—it listens on a specified port for incoming data and forwards the information to a specified destination

35


How?

Rsyslog

An alternative daemon that works on the same principals as syslog-ng such as the ability to multiplex messages, use filters, pipe to programs, etc.

Not quite as “mature” as syslog-ng, but it does have a lot of community support and many of syslog-ng’s “pro” features for free.

36


Now We’ve Introduced a New Problem!

How do we manage such a large amount of data?

How do we detect errors from a single device?

New processes need to be developed to detect device errors, degradation, change notifications, etc.

37


Syslog Analysis

Database metrics

Collect metrics to show fault indicators and performance degradation

Top hosts/messages/severities, etc.

Messages per second/minute/day, etc.

38


Syslog Analysis (Cont.)

Database metrics

Integrate syslog data with performance managers to trigger baseline thresholds

Example: Collecting the number of average messages per second a single device generates and alerting on variations outside the derived baseline

Integrate with Inventory systems!

If a device is talking to you, there’s a good chance it exists

Example: new devices being added to the network will have to wait until the next polling cycle by discovery systems, but if syslog is turned on in that device, your syslog manager will pick it up almost immediately.

39



40



Syslog-ng

Design Guidelines

Single Server Deployment

Multi-Server Deployment

Server Sizing

41


Syslog-ng

This design is based on the use of syslog-ng

Syslog-ng allows for collection and “forking” of syslog messages to many hosts

This allows log data to be collected and distributed in a much more robust fashion

42


Traditional Logging Architecture

Traditional logging collection requires that many logging destinations be stored in each device

43


Syslog-ng Logging Architecture

Syslog-ng collectors allow for only a few logging hosts to be configured in your devices but then replicates these messages to end hosts

44


Design Guidelines

Collection stations

Design your syslog architecture in a distributed, hierarchical fashion

Syslog collectors should lie as close to their networks as possible

Some filtering may be done at the collection level to weed out unnecessary log data

These collectors should forward filtered messages to a centralized server/database for further filtering and processing

45


Design Guidelines (Cont.)

Syslog Event Manager

Deploy a performance management tool such as Cacti to establish a baseline of your logs

Assign people (or groups) to monitor daily Top X events/hosts/messages, etc. and remediate common problems such as fan failures, duplex mismatch, redundant power fails, etc.

Log Rotation and Retention

Establish a log retention and rotation policy

Include logs and log archives in a standard backup process

46


Single Server Deployment

Can handle ~100 million messages per day

Dependent on server CPU(s), disk(s) and memory

One million logs = ~300-500MB of DB disk space a day, so size accordingly!

47


Multi-Server Deployment

Assured delivery via TCP

Can handle large amounts of messages

Requires high end (master) server

Distributed collectors can be small servers used to filter and forward

UDP

UDP TCP

TCP

48


Server Sizing

Please be aware that this is only a rough estimate

Every network is unique, there truly is no way to tell how many events/day you can expect until you start looking.

These are just a few devices that I’ve checked recently that may help you get an idea.

49


Estimated EPD and Storage Needs

Device Type Device Count

Average Msgs/Week Per day Per Hour Per Min

Per second DB Size

External Indexes

Cisco Call Manager 1 151,200.00 21,600.00 900.00 15.00 0.25 21.09M 22.58M

IP SLA (2000 Probes P2P) 1 302,400.00 43,200.00 1,800.00 30.00 0.50 42.19M 45.15M

VMWare 1 3,780,000.00 540,000.00 22,500.00 375.00 6.25 527.34M 564.42M

Cisco ASA Firewall 1 6,652,800.00 950,400.00 39,600.00 660.00 11.00 928.13M 993.38M

Generic Router/Switch 1 211,680.00 30,240.00 1,260.00 21.00 0.35 29.53M 31.61M

Totals 5 10,886,400 1,555,200 64,800 1,080 18 1.48G 1.59G

50


High Level Starter Design

Sample Daily Activities

Identify Hardware related/Restart/Reboot events

Identify configuration changes (and forward to compliance manager)

Identify SNMP Authentication Failures

Identify large numbers of failed logins

Sample Daily Reports

Top 10 Hosts

Top 10 Mnemonics

Top 10 Severities

Top 10 Programs

These are low hanging fruits, take advantage of them!

Syslog Receiver

Syslog Reporter

Syslog Protocol

51


Syslog Protocol

High Level Advanced Design

Syslog Receiver

Syslog Reporter

Inventory Mgmt

NCCM (Compliance)

Device Synchronization

Filtered Change Notifications

MoM/Fault Mgmt

Event Correlation

Fault Notification

Performance Mgmt

Event Deduplication

DB Poller

Incident/Ticketing

Baseline Threshold

Alerts

52


Always filter unwanted messages versus wanted

Allows for proper metric trending ‒ You may not care that a VPN session has terminated, but do you care that 1000

of them are terminating every minute?

Saves you the embarrassment of having to explain to upper management why you MISSED the message that caused the outage.

Filtering Events

My Organization Has a LOT of Events.

Can You Please Tell Me What to Look For?

53


Design an architecture to build filters based on actionable events

Filtering Events

But… My Organization Has a LOT of Events.

54


Actionable Events

When an Event Is Received, Two Immediate Questions Need to Be Asked:

Have we seen the event before?

Is an action required for the event?

55


Advanced Design – Actionable Events

Syslog Protocol

Event Initial Analysis Start

Syslog (Event)

Database Query

Response

Yes

Actionable Event?

No

No Action: Leave Event in DB for Later Forensics and Retention

Yes

Immediate Action

Required?

Open Incident

Open Problem

Yes

No

Known Event?

Periodically Move Old Events to Offline (Disk) Storage

No

Mark as Known, Determine Action

56


Analysis Tools

57


Analysis Tools

Syslog-ng Store Box (SSB)

LogLogic

Splunk

LogZilla

Cisco Prime (LMS)

58


Syslog-ng Store Box

Turnkey solution for deploying syslog receivers using the pro version of syslog-ng

High-capacity log server with high-availability support

Able to collect logs from many different platforms

Made by the authors of syslog-ng

http://www.balabit.com/network-security/syslog-ng/log-server-appliance/

59


LogLogic http://loglogic.com

Commercial Solution

Capable of handling large amounts of data (70-100k mps)

Highly Scalable

60


Splunk http://splunk.com

Searches and navigates IT data (not just Cisco devices) from applications, servers and network devices in real-time

Free version available (limited to 500MB of storage)

61


Highly Scalable Indexing Service Provides Results in Seconds, Not Minutes.

Process Messages for up to 50,000 Hosts

Modular open source code allows for easy customization

VMWare Ready Turnkey Hardware Solutions

Available Optional Cloud-based service

offering allows users to send logs to our datacenter without having to set up their own server

http://www.logzilla.pro

@logzilla

62


Cisco Prime LMS (RME Component) http://www.cisco.com/go/lms

Supports filtering of unwanted syslog messages

Can trigger user-defined scripts in response to specific syslog messages

Provides reports to quickly view syslog events by severity, device, or message.

Does not scale to large numbers, just handy for alerting if you already own it.

Note: Some message filters are enabled by default, including: Link Up/Down, ASA, DEBUG, and IOS Firewall Audit Trail messages. Large amounts of messages may lock up a server, so plan wisely!

63


Case Study Cambridge University

64


Goals

Improve confidence of log integrity in the event of a security compromise. How can we be certain that we’ll receive the event?

Provide mechanism to swiftly analyze logs on all systems. Utilize daily reports to find “top talkers”

Provide mechanism to have instant reports on log-on activity on all systems, and any other ad-hoc reporting required.

Information Provided by Andrew Baughan - Cambridge IT Manager

65


Goals (Cont.)

Solution must be robust, and not introduce a significant overhead on target systems.

If possible, provide a mechanism to store all system logs for fault analysis and baseline statistical analysis for host data.

Provide a system capable of handling 100 Million events per day


66


Solution Selection

The two technologies chosen for this solution were syslog-ng and LogZilla

Because syslog-ng writes the log data to both the client (localhost) and the LogZilla server simultaneously, everything reaches the LogZilla server and cannot be simply deleted or modified on the local host.


67


Solution Selections (Cont.)

Since all logs are held in a central database, with a feature rich user interface, patterns of events across a large number of systems can be quickly identified and acted upon once the "signature" of a compromise is known.

It is easy to write scheduled jobs on the data stored within the MySQL database used as the back-end to LogZilla.

Open source makes it easy to customize to our needs if we ever need to.


68


Solution Selections (Cont.)

The syslog-ng software does not add significant overhead to the target systems.

Higher priority events can be filtered out from the lower priority background events for improved reliability during high traffic periods on the log server.

An increase to the allocated storage allowed a longer retention period, with all events being filtered to the log host.


69


Server Hardware

1 x Dell PowerEdge R410 ‒ 2 x Intel Xeon E5504 @ 2 GHz

‒ 4 x 4GB DIMMs

‒ 1 x SAS 6/iR, internal raid card for Hotplug drives

‒ 2 x 160GB SATA 7.2k 3.5 inch Hotplug HDD (mirrored system disk)

‒ 1 x PERC H800 RAID Adapter for External JBOD, 512MB, PCIe

‒ 1 x iDRAC6 Enterprise

‒ 1 x 16X DVD+/-RW ROM Drive SATA

‒ 1 x Redundant Power Supply (2 PSU) 500W

‒ 1 x Sliding Ready Rack Rails


70


Storage Hardware

1 x PV MD1000 SAS Chassis ‒ 9 x 500GB NearLine SAS 6Gbps 7.2k 3.5" HD

‒ 1 x Rapid Rack Rails

‒ 1 x 2M External SAS Connector Cable

‒ 1 x PV MD1000 Bezel


71


Hardware Notes

Two internal disks are in hardware mirror (R1), ext3 file-system.

The external disks are in a hardware RAID6 configuration with one hot spare, ext3 file-system.

The memory was a bit low for our needs, so it will most likely get doubled fairly soon, otherwise no substantial bottlenecks that a bit of tweaking cannot fix (I/O caching for example has been tweaked).


72


Comments

Primary concern for large deployments should be DISK I/O and Memory.

When available use SSD disks (at least for write caching) ‒ Google “Cachecade Pro”

You can never have too much memory. Use a min of 32GB, but 96GB or even 128GB would be better. ‒ This determines how long it takes for SQL to do table sorting. The more memory

you have, the less it swaps to disk.

‒ Learn how to tweak MySQL, it can definitely scale, but people tend to not understand all the “knobs” to turn.

73


Implementation

Syslog Collector (using syslog-ng)

Search and Reporting Tool (using LogZilla)

74


Implementation



75


Hardware

My laptop and VMWare

Realistically, start with a dual or quad-core box with 4-8G ram and work up from there unless you expect a large amount of logs (> 10mil/day)

Software installed:

Ubuntu v12.04 Server (64bit is a must!) – Why?

Basic server with a LAMP stack

Updated to latest patches

Syslog-ng

Syslog Collector Server Environment

LAMP = Linux, Apache, PHP, MySQL

76


root@log# aptitude install syslog-ng Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libevtlog0 The following packages will be REMOVED: rsyslog ubuntu-minimal The following NEW packages will be installed: libevtlog0 syslog-ng Do you want to continue [Y/n]? Y Setting up syslog-ng (2.0.9-4.1) ... * Starting system logging syslog-ng [ OK ]

Remove ubuntu minimal? Yes – Don’t Worry!

Use sudo if you’re not logged in as root “sudo aptitude…”

Installing Syslog-ng Syslog-ng is available in the apt repositories:

77


Installing Syslog-ng

That was difficult, wasn’t it?

78


Configuring Syslog-ng

The syslog-ng configuration file is typically stored in ‒ /etc/syslog-ng/syslog-ng.conf

There are five “steps” to building a syslog-ng configuration

Main configuration options

Source definitions

Filter definitions

Destination definitions

Statement to apply the defined source, filter and destination

79




Source definitions

Filter definitions


Statement to apply the defined source, filter, and destination

80


Main Configuration Sample

Sets “global” options that apply to everything, for example:

use_dns(yes);

use_fqdn(yes);

keep_hostname(yes);

chain_hostnames(no);

81


Sample Main Configuration

options {

log_fifo_size(16384);

time_reopen(10);

use_dns(yes);

dns_cache(yes);

use_fqdn(yes);

keep_hostname(yes);

};

82




Source definitions

Filter definitions



83


Defines sources of information to receive messages from

source s_all { internal(); unix-stream("/dev/log"); udp(); };

“s_all” Can Be Named Whatever You Want, Just Be Sure to Use It Consistently

Syslog-ng Sources

Other UDP/TCP options are available, such as:

udp( ip(127.0.0.1) port(514) );

tcp( ip(0.0.0.0) port(5000) );

Only Allow UDP Messages from Localhost

Allow TCP Messages from All Hosts on Port 5000.

84




Source definitions

Filter definitions



85


Syslog-ng Filters

Defines a filter to be applied

Types of filters are: Facility—Match on a facility code (kern, local7, etc.)

Level—Match on a level code (error, notice, emerg, etc.)

Program—Match messages by using a regular expression against the program field

Host—Match messages by using a regular expression against the host field host("^cam(1|2|3|4|5)\.somehost\.tld$"); };

Match—Match a regular expression to the message itself

Filter—Call another filter rule and evaluate its value

Netmask—Determine if the sender’s IP is in the specified IP subnet

“Level” Is Synonymous with “Priority” or “Severity”

86




Source definitions

Filter definitions



87


Syslog-ng Destinations

Destinations Define Where to “Fork” Messages to, Such as: ‒ Files

‒ Programs

‒ Remote Hosts

‒ PIPE/FIFO

‒ Databases

88


Sample Destination Definitions

File

destination df_syslog { file("/var/log/syslog"); };

destination df_disk { file("/var/log/HOSTS/$YEAR/$MONTH/$DAY/$HOST" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); };

Program

destination d_mydest { program("/var/scripts/db_insert.pl"

template("$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n")

template_escape(yes) );};

Remote Host

destination d_othersyslogbox { udp("170.19.86.100" port (514));};

You can use any name, just be sure to use the same name when applying it later on

89


Syslog-ng Destinations (Cont.)

PIPE (or FIFO) destination d_mysql {

pipe("/tmp/mysql.pipe" template(

"INSERT INTO logs (host, facility, level, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$LEVEL', '$S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes));

};

Note: for direct DB inserts, it’s better to use the new SQL insert function built into syslog-ng 3.x, but ONLY when you don’t use a pre-processor like PERL (for event correlation, deduplication, etc.)

90


Syslog-ng Direct Database Inserts

The following example is for MySQL, but other types may be used such as sqllite, pgsql, mssql and oracle.

@version: 3.0 destination d_mysql { sql(type(mysql) host("localhost") username("syslog") password("syslog") database("syslog") table("logs") columns("host", "facility", "level", "datetime", "program", "msg") values("$HOST_FROM", "$FACILITY", "$LEVEL", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "$PROGRAM", "$MSG“) indexes("host", "facility", “level", "datetime", "program")); };

@ at the beginning, syslog-ng 3.02 and up complains if this is not present

91


Using Direct Database Inserts

Plan on implementing this in a large scale environment?

If you have more than 1000 events per second, MySQL will bottleneck causing dropped events.

Instead, use a “program” call to Perl to process the incoming data and use bulk insert methods. This will handle up to ~35k mps (maybe more, but that’s what I’ve tested it to).

destination d_mydest { program("/var/www/myprog/scripts/myscript.pl”

92




Source definitions

Filter definitions



93


Once you have a defined source, filter (optional) and destination, you must “apply” them in a statement: log {

source(my_source);

filter(my_filter);

destination(my_dest);

};

Apply the Definitions

94


source s_all {

internal();

unix-stream("/dev/log");

file("/proc/kmsg" log_prefix("kernel: "));

udp();

tcp(port(2000));};

filter my_filter {

host("^router (1|2|3|4|5)\.cisco\.com$");};

Sources

Filter (Optional)

Final Sample Configuration

95


destination my_dest {

file("/var/log/logzilla/syslog.log"

template("$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n"));};

destination my_dest_hosts {

udp("1.1.1.1" port (514));

tcp("2.2.2.2" port (2001));};

log { source(my_source); filter(my_filter);

destination(my_dest);

destination(my_dest_hosts);};

2 Destinations

Apply

Final Sample Configuration

96


Syslog-ng: Getting Help

Website

http://www.balabit.com/network-security/syslog-ng/

Documentation

http://www.balabit.com/support/documentation/

Mailing list

https://lists.balabit.hu/mailman/listinfo/syslog-ng

97


Implementation



98


Installing LogZilla

VMWare images are freely available at http://www.logzilla.pro/downloads

Subversion

Cloud

Hardware

Four Ways to Obtain

99


Installing LogZilla (Cont.)

Visit http://www.logzilla.pro/downloads to grab the latest VM build

Use VMWare ESXi, Workstation, or VMWare Player and double click on the downloaded .ovf file to install the VM

VMWare Method

Read the console instructions after booting the VM

100



Make sure your OS has the correct prerequisites! ‒ https://www.assembla.com/spaces/LogZillaWiki/wiki/Prerequisites

Check out the code from the repository root# cd /var/www

root# svn co https://subversion.assembla.com/svn/logzilla.2/trunk/ logzilla

Optional but recommended: create an “answer” file (see next slide)

Install root# cd /var/www/logzilla/scripts && ./install.pl

Subversion Method

101



You can create an answer file at /var/www/logzilla/scripts/.lzrc so that the install won’t need to prompt you for answers during installation

Please visit https://www.assembla.com/spaces/LogZillaWiki/wiki/RC_File for the contents of this file as it may be updated from time to time.

Answer File

102


Installing LogZilla

Login to http://<your_url> and check for data

After Install Completes

103


Troubleshooting

Try these in order. 99% of the time, it will be one of these 3 things

1. Is LogZilla able to receive events?

2. Is syslog-ng sending events to LogZilla?

3. Is syslog-ng getting events from the network?

Three “Parts” of Troubleshooting

104


Troubleshooting

Bypass syslog-ng by piping a message directly into LogZilla ‒ If this passes, then the problem is most likely not LogZilla

(note: all of this on a single line)

printf "`date \"+%Y-%m-%d %H:%M:%S\"`\ttest\t190\tCRON\tTest\n" | /var/www/logzilla/scripts/db_insert.pl –d5 -v

Is LogZilla Able to Receive Events?

105


Troubleshooting

Check for errors, for example: ‒ Error 13 from MySQL (unable to write to disk due to permissions)

Starting /var/log/logzilla/db_insert.log for /var/www/logzilla/scripts/db_insert.pl at pid 23614 Using Database: syslog Debug level: 5 Table: logs Adminuser: PW: DB: syslog DB Host: DB Port: Deduplication Feature = 0 Logging results to /var/log/logzilla/db_insert.log Printing results to screen (STDOUT) Snare Enabled …snip… (bunch of debug output – note that it’s not all just errors)

Pipe Output

106


Troubleshooting

Enable the DEBUG.log in syslog-ng.conf for LogZilla

By default, LogZilla installs the DEBUG.log output in syslog-ng.conf, just enable it and restart syslog-ng

Also useful for “replaying” events into LogZilla by piping them into db_insert.pl

Verify that syslog-ng is Sending Events

107


Troubleshooting

#/etc/syslog-ng/syslog-ng.conf

log {

source(s_net);

destination(d_logzilla);

# Uncomment below and restart syslog-ng for debugging

destination(df_logzilla);

};

root@myserver# /etc/init.d/syslog-ng restart root@myserver# tail -f /var/log/logzilla/DEBUG.log

(make sure you see events coming into the file)

Verify that syslog-ng is Sending Events

108


Troubleshooting

Use tcpdump to sniff the udp packets root@myserver# tcpdump udp port 514

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

16:32:58.309204 IP 12.5.92.4.51220 > 12.5.92.2.syslog: SYSLOG daemon.error, length: 113

Is syslog-ng Getting Events from the Network?

109


LogZilla - Getting Help

Main Website

http://www.logzilla.pro

Forum

http://forum.logzilla.pro

Support

http://support.logzilla.pro (or email [email protected])

Ideas (submit ideas for future enhancements!)

http://ideas.logzilla.pro

Documentation and Installation Guide

http://docs.logzilla.pro

110


General Useful Links

Clayton’s NMS Wiki

http://nms.gdd.net

111


Personal Goals (For LogZilla)

Track devices’ individual EPS rates using RRD

Implement Known Event/Actionable Event interface with northbound forwarding to incident management systems

Build an AI syslog deterministic tool

What I Want to Do in my Copious Amounts of Spare Time

112


Personal Goals

Current method (DB entries) do not scale well when the server has 50k devices in it

The answer is to use RRD files to store metrics on individual hosts

The problem with traditional RRD usage is the charts it creates, way too ugly. So I am working on a backend to pull the data into a php array and make much prettier charts using jQuery

Track Devices’ Individual EPS Rates

113


Personal Goals

Congruent with my whitepaper and the slides in this session.

Plan to add a right-click menu from the LogZilla search results that can mark as known and/or non-actionable.

Ability to trigger a northbound event to a problem or incident manager.

Known Event/Actionable Event Interface

Ability to tag and anonymously share with a community of LogZilla users to help you determine if an event *should* be actionable ‒ e.g.: 3000 other users marked this event as

“requires immediate action”

114


Personal Goals

Something I’ve been working on for about 8 or 9 years now. The concept is to build an AI that can predetermine certain pattern types and possible

actionable events in order to mitigate network downtime and/or security breaches. ‒ The key is that new messages may never have been seen before. How do I know what I don’t

know? ‒ How can I determine that my network is being attacked AS it is happening versus waiting until

after it has happened? (Anyone want a free credit card?)

Think of it as a spam bot, but with multidimensional probabilities (and quite a bit more complex). ‒ Spam determination is a yes/no (spam/not spam) calculation. ‒ The syslog AI is a determination of a multitude of variables (fcaps, vendor, severity, technology,

etc.)

Prototype is now working and I plan to launch http://sysloggenome.com when time permits. The concept being somewhat of a game where I guess what log you give it and, if I am wrong, you help me “learn” the correct one(s).

Build a Syslog Artificial Intelligence Program

115


Key Takeaways

People tend to be a bit overwhelmed by the amount of data they have to parse through ‒ Proper implementation of tools, metrics and processes will

solve that problem

Configure ALL devices consistently and properly; Make sure any new device deployment ALSO has the correct configuration

Properly designed, syslog can be one of the BEST sources for proactive network management available

116


Other “NMS” Sessions If you like this one, you might like some of these:

Source: Placeholder for Notes is 18 points

117

Monday

Session Title

Mo

nd

ay

BRKNMS-2515 A “Difference-Maker” for Operating Cisco Collaboration Services – Proactive Problem Management

BRKNMS-2658 Securely Managing Your Networks with SNMPv3

BRKNMS-3043 Advanced Performance Measurement for Critical IP Traffic with Cisco IOS IP Service Level Agreements

BRKNMS-1037 Cisco Unified Communications Management Suite Methodology and Best Practices

BRKNMS-3021 Advanced Cisco IOS Device Instrumentation

CCSNMS-1903 Customer Presentation: Learn How Human Kinetics Uses a New Cisco Solution to Dramatically Improve Network and Security Operations

PSONMS-1038 Cisco Prime for Enterprise – Innovative Network Management

Tues

day

BRKNMS-1032 Network Management KPI's

BRKNMS-3132 Advanced NetFlow

BRKNMS-2035 Ten Cool LMS Tricks to Better Manage Your Network

BRKNMS-3021 Advanced Cisco IOS Device Instrumentation

BRKNMS-2640 Advanced DHCP and DNS Deployments


Other “NMS” Sessions If you like this one, you might like some of these:

118

Monday

Session Title W

edn

esd

ay

BRKNMS-2512 Moving Cloud from hype to realty: Delivering Cloud using management architecture

BRKNMS-2031 SYSLOG Design, Methodology and Best Practices

PSONMS-1038 Cisco Prime for Enterprise – Innovative Network Management

BRKNMS-2514 Performance Assurance: An integrated Approach to Managing Service Delivery in Enterprise Networks

BRKNMS-2513 Best practices for cloud automation with Cisco Intelligent Automation software

BRKNMS-2202 Ethernet OAM – Technical Overview and Deployment Scenarios

BRKNMS-2030 Onboard Automation with Cisco IOS Embedded Event Manager

Th

urs

day

BRKNMS-1036 Simplify Network Management and Reduce Operational Costs with Cisco Prime

BRKNMS-2511 Cisco Videoscape Management Solution

BRKNMS-2031 SYSLOG Design, Methodology and Best Practices

BRKNMS-1035 The NOC at CiscoLive


Complete Your Online Session Evaluation Give us your feedback and you

could win fabulous prizes. Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.

119


Final Thoughts

Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042

Come see demos of many key solutions and products in the main Cisco booth 2924

Visit www.ciscoLive365.com after the event for updated PDFs, on-demand session videos, networking, and more!

Follow Cisco Live! using social media: ‒ Facebook: https://www.facebook.com/ciscoliveus

‒ Twitter: https://twitter.com/#!/CiscoLive

‒ LinkedIn Group: http://linkd.in/CiscoLI

120


Extra Credit Event Correlation using Simple Event Correlator (SEC)

122


Installing Simple Event Correlator (SEC)

SEC is available in the apt repositories:

aptitude install sec Reading package lists... Done Building dependency tree Reading state information... Done Reading extended state information Initializing package states... Done Writing extended state information... Done The following NEW packages will be installed: sec

root@log#

Setting up sec (2.4.2-1) ... SEC disabled in /etc/default/sec

Use sudo If You’re Not Logged in as Root (“sudo aptitude…)”

123


#Defaults for sec RUN_DAEMON="no" DAEMON_ARGS="-conf=/etc/sec.conf -input=/var/log/syslog -pid=/var/run/sec.pid -detach -syslog=daemon"

Change to “yes”


Edit the SEC config to allow it to start

vi /etc/default/sec root@log#

124



SEC uses a configuration file and takes input from a file or a named pipe

vi /etc/sec.conf # Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly

root@log#

First step is to create a config file:

Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 125


SEC Rules

SEC includes several different types of rules that are useful in event correlation.

This rule is of type Single. # Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly



SEC Rules

RegExp is the pattern type, select RegExp for (“Regular Expression”) matching or SubStr, for simpler string matching

# Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly



SEC Rules

foo\s+(\S+) is the actual pattern - in this case a perl regular expression pattern




SEC Rules

desc is a variable definition for the pattern description (captured from the foo pattern using parentheses).

In this case a perl numbered variable, $0, is set to the entire matched pattern




SEC Rules

The action statement describes the action taken when the pattern is recognized.

In this case, the logonly action simply writes the pattern to the logfile if one is indicated on the command line, or to standard output if not




SEC Rules

Save the file and execute the following command:

sec -conf=/etc/sec.conf -input=- root@log#

SEC (Simple Event Correlator) 2.4.2 Reading configuration from /etc/sec.conf 1 rules loaded from /etc/sec.conf



SEC Rules

This example will take input from directly from the terminal. Type the following lines of input:

foo

foo bar foo bar

baz

bar foo baz bar foo

Notice that SEC Responds by Replying Every Time a Pattern Is Matched

bar foo baz

Response

Response

No Response

No Response



SEC Actions

SEC has over a dozen different actions it can perform once it matches a pattern in the input stream

Some of the actions depend on their context



SEC Actions

write

Writes the specified text to the named filename.

E.g.: action=write - Hello from SEC. Matched text was $0

shellcmd

Causes SEC to execute a shell command.

E.g.: action=shellcmd mycommand.sh

spawn

Identical to the shellcmd action, but output (e.g. from an exit status in that shell script) from the command is fed back into SEC for pattern matching

E.g.: action=shellcmd mycommand.sh

Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html

134


SEC Actions

assign and eval

Both assign and eval deal with ``% <letter>'' variables. They are internal SEC variables that can be used in rules

E.g.: action=assign %f Joe bob likes,

action=eval %h ($t = “fishing and nascar")

action=write - %f %h at %t

Note: These are all parts of separate rules, but kept out for brevity



SEC Actions

event

event allows the insertion of input to SEC from inside SEC itself

event is feedback mechanism - one controlled by SEC's own rules.

The time parameter is the number of seconds to wait before inserting the event text into SEC's input stream.

E.g.: action=event 5 baz is now matched. ; write - foo matched at %t. baz event in 5 seconds...



Sample Rules—Suppress

# System configuration events # suppressed because we don't care about it type=suppress ptype=substr pattern=%SYS-5-CONFIG_I: desc=device configuration

What’s Wrong with This Rule? (Should We Suppress Configuration Changes?)

Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec 137


Sample Rules—Time Based

# Looks for a reload followed by a restart event # type=pairWithWindow ptype=regexp pattern=\d+:\d+:\d+.*?(\S+)\s+\d+:.*?%SYS-5-RELOAD: desc=(CRITICAL) $1 RELOAD_PROBLEM action=pipe '%s' mail -s 'cisco event' [email protected] ptype2=regexp pattern2=($1).*?%SYS-5-RESTART: desc2=(NOTICE) $1 RELOAD_OK action2=pipe '%s' mail -s 'cisco event' [email protected] window=300



Sample Rules—Escalation

# This rule escalates to CRITICAL if there are more than 5 # neighbor changes in 5 seconds # type=SingleWithThreshold ptype=substr pattern=(MINOR) OSPF adjacency change desc=(CRITICAL) More than 5 OSPF neighbor changes in 5 seconds action=pipe '%s' mail -s 'cisco event' [email protected] thresh=5 window=5



Sample Rules—Link Up/Down Pairs

# This rule deals with link down events # type=PairWithWindow ptype=RegExp pattern=\d+:\d+:\d+.*?(\S+)\s+\d+:.*?%LINK-3-UPDOWN: Interface (\S+), changed state to down desc=(MINOR) $1 INTERFACE $2 DOWN and not up in one minute action=pipe '%s' mail -s 'cisco event' [email protected] ptype2=RegExp pattern2=($1)\s+\d+:.*?%LINK-3-UPDOWN: Interface ($2), changed state to up desc2=(WARNING) %1 INTERFACE %2 BOUNCE action2=event %s window=60




# when the first bounce event is seen, create a reporting trigger # type=Single continue=TakeNext ptype=regexp pattern=(\S+) INTERFACE \S+ BOUNCE context=!INTERFACE_BOUNCE_WAIT_$1 desc=interface bounce summary event for router $1 action=create INTERFACE_BOUNCE_WAIT_$1 10 (report INTERFACE_BOUNCE_$1 mail -s 'cisco events' [email protected]; delete INTERFACE_BOUNCE_$1)




# accumulate all interface bounce events into a context # type=Single ptype=regexp pattern=(\S+) INTERFACE (\S+) BOUNCE desc=interface bounce for router $1 interface $2 detected action=add INTERFACE_BOUNCE_$1 %t: %s



SEC—Getting Help

SEC Main Page

http://simple-evcorr.sourceforge.net/

Email list

http://simple-evcorr.sourceforge.net/#mailinglist

Good install and explanation guide

http://sixshooter.v6.thrupoint.net/SEC-examples/article.html

143

BRKNMS-2031

Documents

Transcript of BRKNMS-2031