A Novel Algorithm and Architecture for High Speed Pattern Matching in Resource-limited Silicon Solution

Authors: Nen-Fu Huang, Yen-Ming Chu, Chi-Hung Tsai, Chen- Ying Hsieh and Yih-Jou Tzang

Publisher: ICC 2007

Present: Chen-Yu Lin (林呈俞 )

Date: Oct, 8, 2007

Outline

Introduction Magic State-based Heuristic (MSH) Algorithm An Example Evaluation

Introduction

NIDS/NIPS are designed to detect and identify worms, virus, and malicious code by performing deep packet inspecting on packet payloads.

Signature-based NIDS

• Snort

• Over 2500 patterns as signatures.

• Spend more than 80% CPU time on string matching

NIDS needs fast string matching algorithm to reduce its load.

Introduction

Proposed string matching algorithms

• Boyer Moore

• Solve single-pattern matching problem

• Aho - Corasick and Wu - Manber

• Solve multi-pattern matching

Proposed hardware-based implementation

• AC-Bitmap

• Parallel bloom-filter

• Reconfigurable silicon hardware

• TCAM-based mechanism

Introduction

Budget problem

• Enterprise environments.

• It is not the major concern.

• Medium-sized enterprise (SME)

• It almost the key concern. Providing a high-speed but low-cost string matching with limited resource Consider the SME

• Limited cost and resources

• Most of the networks in SME are wire-speed of 100Mbps.

WAN

LAN

DMZ The processing speed must faster than 300Mbps

Magic State-based Heuristic

General automaton-based string matching model

State transition by state table Search the pattern ID

Magic State-based Heuristic (cont)

Index = { x : y }

• X : input symbol

• Y : current state Snort 2.4 patterns is 21584 v = 16

816

Magic State-based Heuristic (cont)

State table can be represented as state transition matrix

u bit size of a symbol v bit size of a state

state

symbol

a (x, y) = next state when the current

state is y and the input symbol is x

Magic State-based Heuristic (cont)

Magic state

• When A is a DFA, for each symbol x, most of a(x,y) have the same value for different current state y.

• Call these elements “magic state”

• ms(x) : the next state that appears most frequently with symbol x.

If we know that the next state is a magic state, then the state table lookup can be skipped.

Use another bitmap matrix (say B) to indicate whether an element in A is as magic state.

Magic State-based Heuristic (cont)

Bitmap matrix B

Magic State-based Heuristic (cont)

Matrices Construction

• Automaton Transition Matrix A

• Magic State Matrix M• Stores the corresponding magic state ms(x) in the element

• Heuristic Index Matrix H

• Stores some information about whether a(x,y) equals to

Reduce the size of bitmap matrix B (become matrix H)

• Partition into blocks

• Each block size is

m x )0,(

m x )0,(

Magic State-based Heuristic (cont)

Construct the Heuristic index matrix H

• Matrix B Matrix H

Compression ratio (CR)

• CR =

Perform AND operation to each block

Magic State-based Heuristic (cont)

Heuristic Pattern Matching with Magic State

Examining in matrix H

10

It’s a magic stateIt’s maybe a magic state

Get the magic state in matrix M directly

Get the next state from matrix A

An Example

To illustrate the proposed algorithm

• M = [178, 671, 2718, 2732, 4600] (Magic state matrix)

0x31 0x32 0x33 0x34 0x35

Correspond value

An Example

Suppose : m = n = 1

An Example

Case 1:

• State 35 receives input symbol 0x34

• Get the magic state 2732 if symbol 0x34 from matrix M

Case 2:

• State 42 receives input symbol 0x31

• Access matrix A to get the next state 178 (Actually it is a magic state).

1

0

Evaluation

Suppose

• K input symbols

• Hit rate of Heuristic Index Matrix H

95%

675KB

85%

42KB

3KB

46%

Evaluation (cont)

Magic State

• Snort 2.4 has 21584 pattern.

• With 256 symbols Total 21584*256 = 5525504 element in matrix A.

• There are 5243748 magic states (94.9%).

HitRate vs. Compression Ratio (CR)

• Value of m and n impact the HitRate

• Higher CR conducts a lower hit rate.

Evaluation (cont)

85%

68% 70.2%70.6%

70.8%

Largest gap is 85%-68% = 17%

• Interesting result

Evaluation (cont)

False Negative

• When (m,n) = (4,0) there are 15% state transition that we don’t sure the next state is a magic state.

• Need to access Automaton Transition Matrix

• Among these 15%, only 5% are non-magic states.

• Thus, 10% state transitions is false negative.

Evaluation (cont)

Total time of state transition

• If matrix M and matrix H can be accessed concurrently

• Algorithm without employing magic state

• The proposed algorithm has a throughput gain

Evaluation (cont)

Memory space for matrices

• Automaton Transition Table (ATT)

• Magic State Table (MST)

• Heuristic Index Table (HIT)

MST & HIT are tiny, and can be stored into on-chip memory. ATT is too large, it can stored in DDR2 SDRAM

Simulation with (m,n) = (4,0) Implementation model

• Baseline Model

• MSH Model

• Multiple PMEs MSH Model

Evaluation (cont)

Baseline Model

• Throughput is 133.33Mbps MSH Model

• Simulation throughput is 566Mbps

Store ATT

Evaluation (cont)

Hit rate = 85%, throughput is 571.42Mbps.

4.28 times faster than baseline model.

Evaluation (cont)

Multiple PMEs MSH Model

• The proposed MSH can be further extended to have multiple PME in a single FPGA to process multiple sessions concurrently.

Throughput is 1036.26Mbps,

7.77 times faster than baseline model

Evaluation (cont)

With two PMEs

Evaluation (cont)

MSH-1 AC-Bitmap

Memory 42KB 2MBCost of on-chip memory

•FPGA-based solution is expensive

•The solution can be implemented on off-chip high speed memory (SSRAM)

•SSRAM faces the problem of very low throughput.

•By utilizing the feature of Magic State more intelligently, the memory require

of MSH reduce to less than 2MB It can be stored into on-chip memory