Leveraging super -indexed searchesh41382. · 2014. 9. 9. · •Master super-index is checked –...

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Leveraging super-indexed searches Jason Stoops, Software Designer #HPProtect


Super-index basics

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4

What do super-indexes do?

Track columns to rule out ranges of CORR-engine event data This can improve performance in: • Search queries

– Local and Peer search • Reports • Event export


Super-index on event ingestion

• Super-indexing occurs after Indexer • Two tiers:

– Master super-index tracks all event data

– Data-range super-index tracks 1-2 hours

Master super-index

Data-range super-index

CORR-engine storage

Indexer


Super-index on search

• Master super-index is checked – If no results, search is done.

• Data-range indexes covering search time range are checked – Data-ranges with no results can be ignored

• Finally, ROS search data-ranges with hits in CORR-engine storage

Master super-index

CORR-engine storage


What types of queries benefit most?

Queries with no results Master super-index can return in 1-2 seconds regardless of time range searched

Queries with very few results Ranges of event data are ruled out to significantly improve search speed

Needle-in-haystack style searches!


Super-index use cases

Queries expecting few or no results • Forensic searches for malicious hosts • Find a particular class of event • Determine if a device was down for some period of

time


Using super-indexes


Super-indexed fields

Host fields • destinationHostName • destinationPort • destinationAddress • sourceHostName • sourcePort • sourceAddress • deviceAddress • deviceHostName

User fields • destinationUserId • destinationUserName • sourceUserId • sourceUserName

Event source fields • deviceEventClassId • deviceProduct • deviceVendor


Super-index operators

Queries with equal to (=) are accelerated No performance impact on: • Greater than (>) • Less than (<) • Not equal to (!=) • Other operators

– (CONTAINS, STARTSWITH, BETWEEN, etc.)


Example queries


Super-indexed single field-based search

Using equal to (=) operator sourceAddress = 192.0.2.0 • Quick response if address does not exist • Reduced search time if address exists. • Only time-ranges of CORR-engine data containing

this source address are searched.


Super-indexed fields with AND / OR

AND sourceAddress = 192.0.2.0 AND destinationAddress = 192.0.2.2

• Quick response if either does not exist • Reduced search time if addresses exist

OR sourceAddress = 192.0.2.0 OR sourceAddress = 192.0.2.2

• Quick response if both do not exist • Reduced search time if addresses exist


NON-Super-indexed full text search

Full text search "arcsight" • Super-indexes not available on full-text search • No negative impact on existing search performance


Super-indexed fields AND / OR (cont’d)

AND sourceAddress = 192.0.2.0 AND "arcsight" requestMethod = GET AND sourceAddress = 192.0.2.0

• Quick response if address does not exist • Reduced search time if address exists

OR sourceAddress = 192.0.2.0 OR "arcsight" requestMethod = GET OR sourceAddress = 192.0.2.0

• No super-index speed improvement • Knowing if address exists doesn’t assist with OR

condition • Non-super-indexed field can still make “OR”

condition true even if address does not exist


Ensuring super-indexes are utilized


Ensuring super-indexes are used

Data before Logger 5.5 upgrade is not super-indexed • Ensure your searches are against data received

after Logger 5.5 upgrade


Ensure super-indexes are used

Case-sensitive searches only • Ensure search case-sensitivity is enabled

ROS searches only Super-index benefits cannot be applied to WOS searches • Keep indexing is up-to-date • Only combine super-indexed fields with indexed

fields


For more information

Attend these sessions

• TT3132, Army of loggers: Searching for that needle at billions of EPS

• TT3154, Renovation of the HP Arcsight Logger user interface

Visit these demos

• HP ArcSight Logger demo station

After the event

• Contact your sales rep

• Visit the website at: www.hp.com/go/logger

Your feedback is important to us. Please take a few minutes to complete the session survey.


Please fill out a survey. Hand it to the door monitor on your way out.

Thank you for providing your feedback, which helps us enhance content for future events.

Session TT3099 Speaker Jason Stoops

Please give me your feedback


Thank you

Leveraging super -indexed searchesh41382. · 2014. 9. 9. · •Master super-index is checked –...

Documents

Transcript of Leveraging super -indexed searchesh41382. · 2014. 9. 9. · •Master super-index is checked –...