SN08 Javier-Inclan Writing Rules

www.arcsight.com 1© 2010 ArcSight Confidential

© 2010 ArcSight, Inc. All rights reserved. ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.

SN08: Primer: Writing Rules Not Meant to be Broken

Javier Inclan Worldwide Principal Instructor

September 2010


Agenda

Rules foundations Understanding rule components

– Conditions– Aggregation– Actions– Triggers

Mastering rules Additional rule features Troubleshooting rules Tuning rules


Rules Foundations


First Things First: Rule Definition

What is a rule? Process who is running in the ArcSight ESM manager Evaluates incoming events looking for specific conditions and patterns.

Based on these results it infers meaning about their significance and can initiate actions in response

Applied to events during the correlation evaluation phase of the event lifecycle

Rules are loaded by the ArcSight ESM correlation engine when ArcSight ESM starts up

Rules Foundations


and or not

Event Definition

JoinCondition

Two additional editor conditions

plus same regular conditions

Rules Foundations

Concepts for Configuring Rules

Constructed using aggregation and Boolean pattern matching within the CCE (AND, OR, NOT)

Rules operate in the real time event stream

Must be activated (save or linked into the real time rules folder)

Moving rules in and out of the “real-time rules” folder triggers the correlation engine to reload the rules

Rules can be scheduled to run at predefined intervals

Scheduled rules do not have to be in the “real-time rules” group


What Rules Do

Incoming events are compared to conditions and aggregation settings of each enabled rule

Event matches trigger pre-configured actions and a correlation event is generated by default

Rules Foundations


What is a Correlation Event?

Correlation events become new events to be evaluated by the correlation engine

Rules Foundations


Identifying Rules Types

Types of Rules Simple Rules

– Match one or more events against one set of conditions

Join Rules– Match more than one event against two or more sets of conditions

Rules Foundations


Understanding Rule Components



A rule definition is based on four components – Conditions– Aggregation– Actions– Triggers

These components are distributed within three different tabs in the rule’s editor


What Events am I Looking For?

Conditions will define the set of events that I am looking for

Drivers for define this set of conditions could be use case definitions, compliance or computer/network/device security business requirements

Conditions are created using the CCE

Conditions rely on Boolean Logic principles



Defining Precise Conditions

What events am I looking for? Events who have met these conditions are going to be named

“matches” Conditions can be loose

attacker address inSubnet 192.168.1.0/24 or target address HasVulnerability xxxx

Conditions can be well defined/precise attacker address=192.168.1.10

Which one is best for performance?

The devil is in the details (i.e… put a = instead of a > and your rule conditions change, whether intentional or !=)



Use Categorization Fields in Rules

Develop logical framework for grouping resources Leverage ArcSight event categorization

– Since devices do not utilize a common naming schema for events, ArcSight Connectors map individual signatures to a common taxonomy so that ArcSight ESM can later reason over those events

– Without categories: [ID contains 529 or 621] OR [login and failure and SSH] OR [login and failure and target port 23]

– With categories:



Benefits of Network Modeling in Rule’s Conditions

Effective Rules DEPEND on ArcSight ESM product intelligence Enables content that can make informed decisions based on more

detailed information– Asset model describes attributes of the assets

• Vulnerabilities locations active lists asset categories• Increases accuracy of ArcSight priority formula• Identifies assets subject to compliance




Benefits of Network Modeling in Rule’s Conditions

Enables you to build a business-oriented view of dataassets/ranges zones networks customers

ArcSight WITHOUT network, zone, asset modeling, categorization and vulnerability information will produce more false positives and “background chatter” than a mis-configured IDS(aka OPEN THE FLOOD GATES)

How is a vulnerability scanner throwing out traffic on port 23 to 100 servers and analyzing a response differ from a CiscoWorks server using port 23 to push IOS upgrades to 100 switches?


Aggregation or Aggravation?

Not a mis-spelling; we did not say aggravation

Aggravation might be a symptom if timing parameters and number of events within a specified time frame aren’t well understood

Do you want to aggregate on unique or identical fields?

Before rolling your rule out to production – TEST IT in development, QA or with simulated events fed from a test connector



Defining Aggregation

Rule Aggregation Sets required number of event matches within a specified timeframe

– Time frame set here is known as time window expiration Matches only if specified field or fields is unique amongst evaluated events Matches only if specified field or fields is identical amongst evaluated events Values from fields listed in aggregation settings will be carried from base events

to correlation events



Rule: Defining Aggregation

What fields to aggregate on?– Generally: event name, attacker/target

Hostname/Address/FQDN/Domain Name/User Name/Zone Resource– Non-aggregated fields can’t be used in dashboards and reports

– Aggregation impacts memory, as aggregation matchesare counted and tracked• Do not aggregate over long periods of time; instead use an active list• Limit the set of aggregated values

!Tip For MSSPs:Aggregate on CustomerResource to ensure eventsfrom the same IP addressare really from thesame machine



Defining Aggregation

Use to limit the amount of rule firing for repeat events, or to set thresholds that define certain scenarios This specifies number of matches (threshold) in specified

amount of time by the rule Example – five failed login attempts in two minutes may signify

a brute force



Advanced Aggregation

There are four time-evaluation criteria that can affect event-occurrence aggregation and rule-triggering

You can apply these to rules through the aggregation tab and the statement panel of the conditions tab1. Time Frame – establishes the time span for occurrence aggregation

• Event-occurrence aggregation is always controlled by time frame 2. Global Expiration – global expiration applies to an entire rule

• This is the amount of time that qualifying events for all aliases will be retained in memory for evaluation and is based on manager receipt-time

3. Alias Expiration – an alias expiration applies to a single alias within a rule

• This is the amount of time that a qualifying event for this alias will be retained in memory for evaluation and is based on manager receipt-time

4. Matching Time – matching time creates a time-proximity comparison for multiple-alias rules and is based on events' actual creation times



Now, We Are Ready for the Action!

Once the rule conditions are met and we meet the threshold requirements set in aggregation, it’s time to take action!

When a rule fires, an action will be taken based on the trigger that you set

You can select single or multiple triggers

Why is my rule firing at weird times?

Why is my rule not firing? (lets look a little closer at timing and triggers)



Types of Available Rule Actions

A rule can trigger any combination of the following actions Set event field Send to open view operations Send notification Execute command Execute connecter command Export to external system Create new case Add to existing case Add to active list Remove from active list Add to session list Remove from session list



Defining Triggers – Rule Action Triggers

Three types of rule action triggers are available1. Event triggers – act on individual events

– On first event– On every event– On subsequent events

2. Threshold triggers – act on groups of events that satisfy the time frame requirements– On first threshold– On every threshold– On subsequent thresholds

3. Timing triggers – act on timing of events– On Time Unit – triggers on a specified unit of time after a threshold is

met– On Time Window Expiration (TWE) – triggers after the time frame

expires without meeting the number of matches requirement



60 sec

1 2 3 4 5 Matches 1 2 3 4 5

1st Threshold

Time

First Threshold

Threshold condition: five matches within two minutes Threshold condition reached Action takes place Threshold time window resets

(2 Minutes)

180 sec

1st Threshold

240 sec

Time Reset



60 sec

1 2 3 4 5 6 7 8 9 10 11 12 13 14

100 sec

1st Threshold 2nd Threshold

Time

Every Threshold

Threshold condition: five matches within two minutes Every time the threshold is met Action takes place Continues until TWE

TWE

(2 Minutes)



60 sec

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

100 sec

1st Threshold (No Action)

Subsequent Threshold

Time


Threshold condition: five matches within two minutes After the first threshold is met Waits for second threshold to be met Action takes place at subsequent thresholds Continues until TWE

TWE

(2 Minutes)160 sec




60 sec

1 2 3 4 5 6 7 8

120 sec

1st Threshold

TWE

Time

On Time Unit

Threshold condition: five matches within two minutes with a 30 seconds time unit

Initial threshold is met Action takes place every time the time unit elapses Continues to take action until TWE

30 sec Time Unit

30 sec Time Unit

90 sec



60 sec

1 2 3 4 5 6 7 8 9

180 sec

1st Threshold

TWETime

(2 Minutes)

Time Window Expiration

Threshold condition: five matches within two minutes Initial threshold is met Waits until TWE Action takes place



Correlation Events Created by Rules

What fields to set in correlation events? These are the fields that will be set in the correlation event Make sure you don’t create a feedback loop

(rules firing on themselves)

!Tip: Agent SeverityUse low for informational rulesthat have indirect consequenceUse medium to very high forrules of direct consequence



Don’t Break Your Rule with Excessive Actions

What to do when conditions and thresholdshave been met?– Create a new event, create a case, etc

Use “on first event” or “on first threshold” to avoid excessive rule firing due to heavy attack traffic



Don’t Break Your Rule with Excessive Actions

To add all rule firings to a single case, use “on subsequent events” A solution to handle long running continuous attacks would be to

define following triggers– On first threshold – will notify start of attack– On time unit – will periodically notify that the attack is still going on– On time window expiration – will notify end of attack



Mastering Rules


Mastering Rules

Know the business conditions, requirements or use case– That’s how you start to build a rule!

Rule development– ½ science (boolean, timing, action definitions, etc)– ½ art, so keep rule’s conditions as simple and precise as possible

Know our ArcSight event SCHEMA– that’s what you have to work with!

(the fields and the output of those fields)


Create Multiple Simple Rules Instead of One Complex

Break down the use case requirements by listening for key words Define organizations ArcSight network topology

– Network modeling

Track all user logins, from where and to what device – Rule / session List

Track all user logouts, from where and from what device– Rule / session List

Mastering Rules


Document Your Rules

Long after you’ve forgotten (maybe six months to a year down the road) when you need to review what you were thinking….

A best business practice when developing rules is to DOCUMENT the use case, business requirements and details of how the rule was developed on the NOTES tab

Possible topics to note: Who requested the rule, who are the stakeholders, original date and time of testing vs. deployment, etc

Mastering Rules


Use Stock Content and Solutions Foundations

ArcSight provides MANY solutions foundation and stock content rules to facilitate out of box functionality upon installation

If you need to get your bearings this is a good place to start

Remember, any rules enabled in the real time rules folder are LIVE

Mastering Rules


Additional Rule Features



Automatic rule disabling– ArcSight automatically disables improperly written rules that would

produce excessive or meaningless events

Clearing rule actions– In a grid view, select a correlation event– Right-click and choose “correlation options”– “Clear rule actions” to clear all actions associated with this rule

Showing rule errors– If rules have errors, the rule icon ( ) changes to indicate it– In the rules resource tree, right-click the rule-error icon and choose

“show error”– The error appears in a dialog box


Automatic Rule Disabling

Rule disabling factor operation– Alias matches – if an alias is defined, this is the number of events

matching that alias and is independent of other defined aliases in the same rule

– Partial matches – if more than one alias is defined, the number of events matching the aliases defined before the current one, and for the current one, and for their join condition (if present)

– Generated events counts – the number of correlation events generated

– Base event counts – number of base events used to generate correlation events

– Time unit counts – number of time units (minutes) that passed since the rule activated

Above values for rule disabling may be adjusted for your enterprise– ArcSight ESM will disable a rule if the rule exceeds the configured

limits on number of rules triggered per minute or the ratio of base events to triggered rules and is defined in server.defaults.properties file on the manager



Troubleshooting and Tuning



Rules: Troubleshooting

What do you do when the “check engine light” comes on in your car?– Apply the same methodology– Break components down into their most basic form

(don’t digest the entire conditions tab, take it one line or maybe one statement at a time)

Is the data your looking for actually available?– Start back at the basics

(RAW logs from the device prior to hitting our connector and being normalized)

Was the rule imported via an ARB?– If so, was it done on the same revision of ArcSight ESM? – Were resource ID’s exported into the ARB?

Has the rule completed? (partial matching rule?)


My Rule is Broken!

How would I know? What Clues Do I look for?



Check your Condition Logic First

What are your rules dependant on?– Active/session lists, asset/network modeling, variables, etc?

How do you know? Check out https://localhost:8443Resource management and rules to look at details:


https://localhost:8443/�


Identifying Attacks

If a rule is defined to identify the following attacks, it will excessively fire: Denial of Service or Distributed Denial of Service AttackIDS / SIM / SIEM / ESM “Smoke Screening” aka…Copperfield/Angel magic

If rule trigger is activated on EVERY EVENT or EVERY THRESHOLD, it may lead to excessive firing

What would this look like?



Potential Issues Related with Timing

Timing is very sensitive in rules firing End time field is a key player during correlation phase. Network latency could lead to potential issues during correlation:

Verify start time, end time, agent receipt time and manager receipt time values – 1-2 min off could be an indicator of network latency

Poor bandwidth or high EPS could produce same results Did something “recently” change that could effect the arrival of events

into the connector?Anything more – could trip an exception error "DCERPC pipe is no longer open" reported in server.log – check the following:– Changed behavior of A/V or HIPS which now blocks remote pipes– Changed network behavior after a Patch (those do get tested first, right? ;> )– Has your OS stopped allowing remote pipe comms? (ie…Windows Firewall or

IPTABLES)– Domain Admin recently tightened access policy or net admin threw a new

ACL/rule




Rules: Troubleshooting

Is your rule recursive ? Starting in Arcsight ESM 4.5.1, rules that trigger themselves recursively will Automatically be disabled temporarily, then re-enabled

(aka – rule bouncing) Has your rule trigger exceeded the max. # of correlated alerts per min. limit?

You would see an error as seen below in your server.log file:[2009-07-30 10:21:59,750][ERROR][default.com.arcsight.rulesengine.actionengine.ActionCommandHandler][onSingleEvent] Too many pending actions 1000, not adding more ....– This is set in server.default.properties as:

• #number of correlated alerts per rule per minute• rules.max.fan-out.time-unit.ratio=1000• Remember persisted settings must be set in server.properties

To reduce excessive firing, consider using ON FIRST and TU/TWE triggers Monitor your rules engine via rcsight ESM dashboards or status monitoring web

page


Limit Partial Match Storage Using Time Constraints

This condition occurs when using join rules and an event matches one alias

Partial matches for a rule are stored in memory for the specified time window

To limit memory consumption– Limit the aggregation time frame– Use active lists to correlate information from events spaced far in time

! Tip: Partial MatchesCan be monitored usingthe “Rules Status” dash-board in ArcSightAdministration



Tuning Rules

ArcSight ESM comes with a dashboard that can enable you to view the statistics of the rules within your environmentThe following data monitors are included:– Partial matching– Top firing rules– Recent fired rules– Rules engine internal stats– Rule error logs



More Information?

Rules aren’t something we expect you to be a subject matter expert in by attending this workshop or by attending 3-5 day classes

4.5.1 user guide; chapter 13: Rules Authoring 4.5.1 system content reference guide Talk through your rules

– Engineering 101: “If you can’t explain the process, you don’t understand the process”

ArcSight Protect 724– Content sharing and ARB’s– how are your colleagues writing rules?

Review summary for “SQL look” http://en.wikipedia.org/wiki/De_

Morgan's_laws


http://en.wikipedia.org/wiki/De_Morgan's_laws�

http://en.wikipedia.org/wiki/De_Morgan's_laws�


Your Feedback Builds a Better Conference!

Download session replays after the conference:https://protect724.arcsight.com/community/protect10/sessions

Excellent Good Fair Poor

Rate the speaker a b c d

Rate the content e f g h

Please provide comments: (*) enter any comments/feedback

Text to 32075 (USA & Canada) or 447786204951 (Non-USA) Type ARCS <space> 08 and the letter to each response

SMS body example: ARCS 08ae*your comments


ArcSight, Inc.Corporate Headquarters: 1 888 415 ARST

EMEA Headquarters: +44 (0)844 745 2068Asia Pac Headquarters: +65 6248 4795

www.arcsight.com

SN08 Javier-Inclan Writing Rules

Documents

Transcript of SN08 Javier-Inclan Writing Rules