Security event processing with SEC (Simple Event Correlator)

Risto Vaarandi

Estonian Defence Forces

2

Outline

• Event correlation – what it is and why do we need it?

• Simple Event Correlator (SEC) – motivation, features and performance.

• Sample event correlation scenarios.

• SEC application experience and further reading.

3

Event correlation – what it is?

• Commonly used formal definition (Jakobson and Weissman, 1995) – event correlation is an interpretation procedure where a new meaning is assigned to a set of events that occur within a predefined time interval.

• A simple example – if a network link down event is not followed by network link up within 5 seconds, emit an output event network link broken; otherwise create an internal event network link short outage. If more than 10 network link short outage events have been observed within 3 hours, emit an output event network link quality degradation.

• Please note that event correlation is a real-time (or near-real-time) event processing technique!

4

Event correlation – why do we need it?

• Event correlation produces events that are more meaningful to a human.

• Event correlation can significantly reduce the number of events that are presented to the human.

These considerations are important, because in today’s IT systems millions of events can occur in every hour – no hope for a human to follow the events and react to them in a timely manner!

5

Example - centralized event management infrastructure

Application

Server

Router

event receiver

eventcorrelator

Event logs

Notifications tothe monitoringconsole

events

network

Central log server

Applications, servers, and network devices send their events to the central log server.Events are stored and correlated on the central log server, and output alerts from the event correlator are sent to the central monitoring console.

6

Simple Event Correlator (SEC) – motivation

• Existing event correlation products are mostly heavyweight and expensive solutions:

many products can only be deployed as a client-server system with a requirement for a dedicated high-end event correlation server,

some products have been designed for specific tasks only (like network fault management),

several products depend on other complex and expensive software packages (e.g., Oracle),

there are products that can’t be deployed without the assistance from vendor consultants!

there exist only a few free and open-source event monitoring tools that have some support for event correlation.

• SEC implements a completely different event correlation paradigm – lightweight and platform independent event correlation with an open source tool

7

SEC – design and features• Employs rule-based approach for event correlation – events are

correlated according to rules written by a human analyst.• Lightweight design – the size of SEC is < 300KB and < 9000 lines,

runs as a single process.• Modest resource requirements – e.g., with 10,000+ event correlation

operations and contexts active, < 20MB memory is consumed.• Platform independence – SEC is written in Perl, thus runs on almost

any UNIX flavor and Windows (with CygWin Perl).• No dependencies on other software (Perl only).• Reads input events from one or more textual event logs (plugins can

be written for binary formats).• Input events are matched with regular expressions, Perl

subroutines, string patterns, and truth values.• Produces output by executing custom command lines (e.g., sends

an SNMP trap or an e-mail), writing to files and FIFOs, etc.• Licensing – GNU GPLv2 (free and open-source software).

8

Snort IDS alert consolidation I# The following ruleset processes Snort IDS syslog messages:# Apr 4 10:10:55 snorthost.mydomain [auth.alert] snort[18800]: # [1:2528:14] SMTP PCT Client_Hello overflow attempt [Classification: # Attempted Administrator Privilege Gain] [Priority: 1]: # {TCP} 192.168.5.43:28813 -> 192.168.250.44:25# # If a Snort priority 1 alert has been seen for a certain source IP address, create # a context for this IP, in order to consolidate all events for the same source IP;# also, send a short note to the local admin that an attack has begun type=Singleptype=RegExppattern=snort\[\d+\]: \[[\d:]+\].*\[Priority: 1\]: \S+ ([\d\.]+):?\d* -> [\d\.]+:?\d*context=!ATTACK_FROM_$1continue=TakeNextdesc=Priority 1 attack started from $1action=create ATTACK_FROM_$1; \

pipe '%s' mail -s 'Snort: priority 1 attack from $1 (alert)' root

9

Snort IDS alert consolidation II# If a Snort priority 1 alert is observed for a certain source IP address, then:# 1) store it to the context for that IP (the context has been created by the previous rule) # 2) extend the context lifetime for 5 minutes. ## If the context lifetime ends (in other words, there have been no more events from# the given source IP), send all events from the context event store to the local admin# as a single mail. type=Singleptype=RegExppattern=snort\[\d+\]: \[[\d:]+\].*\[Priority: 1\]: \S+ ([\d\.]+):?\d* -> [\d\.]+:?\d*context=ATTACK_FROM_$1desc=Priority 1 incident from $1action=add ATTACK_FROM_$1 $0; \

set ATTACK_FROM_$1 300 ( report ATTACK_FROM_$1 \ mail -s 'Snort: priority 1 attack from $1 (report)' root )

10

User account probe detection I# Process sshd PAM authentication messages from Solaris system log, in order to detect repeated# probing of different user accounts:# Apr 3 14:20:19 myhost sshd[25888]: [ID 800047 auth.error] error: # PAM: Authentication failed for risto from myhost2# Apr 3 14:20:23 myhost sshd[25888]: [ID 800047 auth.info] Accepted # keyboard-interactive/pam for risto from 192.168.27.69 port 9729 ssh2## If a user fails to login with SSH and this is not followed by a successful login within 30 seconds,# generate an event LOGIN_FAILED_FOR_username

type=PairWithWindowptype=RegExppattern=sshd\[\d+\]: \[ID \d+ auth\.error\] error: PAM: Authentication failed for (\S+) from \S+desc=PAM authentication failed for $1action=event LOGIN_FAILED_FOR_$1ptype2=RegExppattern2=sshd\[\d+\]: \[ID \d+ auth\.info\] Accepted keyboard-interactive/pam for ($1)\ from \S+ port \d+ ssh2desc2=PAM authentication successful for $1action2=logonlywindow=30

11

User account probe detection II# Count LOGIN_FAILED_FOR_username events from the previous rule – if 10 distinct usernames# have been seen within 10 minutes, alert the local admin; then switch to silent mode for 1 hour

type=SingleWithThresholdptype=RegExppattern=LOGIN_FAILED_FOR_(\S+)context=!USER_$1_ALREADY_COUNTED && !COUNTING_OFFcontinue=TakeNextdesc=Ten login failures for distinct users have been observedaction=pipe '%s' mail -s 'PAM alert' root; create COUNTING_OFF 3600window=600thresh=10

# Set up a context for the username after it has been counted, in order to prevent repeated counting

type=Singleptype=RegExppattern=LOGIN_FAILED_FOR_(\S+)context=!USER_$1_ALREADY_COUNTED && !COUNTING_OFFdesc=Set up the "count once" context for user $1action=create USER_$1_ALREADY_COUNTED 600

12

User account probe detection III

Feb 18 12:15:11 PAM: authentication failed for john

Feb 18 12:10:23 PAM: authentication failed for ristoPairWithWindowFeb 18 12:10:33 Accepted keyboard-interactive/pam for risto

logonly: PAM authentication successful for risto (at 12:10:33)

SingleWithThreshold

neither USER_john_ALREADY_COUNTED nor COUNTING_OFF context exists at 12:15:41, so start the counting operation with the window of 10 minutes and set event counter to 1

Single

LOGIN_FAILED_FOR_john

create: USER_john_ALREADY_COUNTED

Feb 18 12:16:49 PAM: authentication failed for john doesn’t producea match anymore at 12:17:19

10 LOGIN_FAILEDevents for distinct users within 10m

pipe: send mail alertcreate: COUNTING_OFF(for getting 1 mail per 1 hour)

event: LOGIN_FAILED_FOR_john

Feb 18 12:15:56 ntpd: …..

13

SEC – performance

• SEC’s event processing speed depends on a number of factors – the size and nature of the rule base, the nature of input data, and the underlying hardware.

• Experiment1: Linux workstation with a 3.0GHz P4 CPU, one rule with a regular expression pattern, 1% of input lines were matching and written to a file by the rule – event processing speed was 17,000+ events per second.

• Experiment2: Linux server with a 3.2GHz P4 CPU, 100+ rules with complex regular expression patterns and non-trivial output actions (at least 2-3 actions executed per second), banking card events with a complex structure as input – event processing speed was 3,000 events per second.

14

SEC – experience

• Has been developed by Risto Vaarandi for 8+ years.• Employed by many institutions – large North-American

telecom companies, government and financial institutions, universities, medical and car industry, etc.

• Has been used for security management, network management, system monitoring, etc.

• Has been employed for correlating events from many sources – Snort IDS, HP OpenView, CiscoWorks, BMC Patrol, Nagios, syslogd/syslog-ng, various routers and firewalls, modems and other network equipment, etc.

• Has been packaged for major Linux and BSD UNIX distributions – RedHat, Fedora, Debian, Ubuntu, Gentoo, OpenBSD, FreeBSD.

• SEC mailing list has 300+ members.

15

SEC – references• SEC home page: http://simple-evcorr.sourceforge.net.• John P. Rouillard, University of Massatchusetts at Boston, “Real-

time Logfile Analysis Using the Simple Event Correlator (SEC)”, USENIX 2004 System Administration Conference (see also http://www.cs.umb.edu/~rouilj/sec/).

• Jeff Becklehimer and Cathy Willis, Cray Inc. and Josh Lothian, Don Maxwell, and David Vasil, Oak Ridge National Laboratory (ORNL), “Real Time Health Monitoring of the Cray XT Series Using the Simple Event Correlator (SEC)“, 2007 Cray User Group Conference (see also http://www.nccs.gov/wp-content/uploads/2007/08/becklehimer_paper1.pdf).

• James Turnbull, “Hardening Linux” (Chapter 5), Apress 2005 (see also http://www.apress.com/book/view/1590594444).

• Risto Vaarandi, “Simple Event Correlator for real-time security log monitoring”, Hakin9 Magazine 1/2006 (6) (see also http://en.hakin9.org/attachments/pdf/hakin9_05_2006_10_EN_str28-39.pdf).