A Scalable DDoS Detection Framework with Victim Pinpoint Capability

11
A Scalable DDoS Detection Framework with Victim Pinpoint Capability Haiqin Liu, Yan Sun and Min Sik Kim School of Electrical Engineering and Computer Science Washington State University, Pullman, USA Email: {hliu, ysun, msk}@eecs.wsu.edu Abstract—In recent years, various intrusion detection and prevention systems have been proposed to detect DDoS attacks and mitigate the caused damage. However, many existing IDS systems still keep per-flow state to detect anomaly, and thus do not scale with link speeds in multi- gigabit networks. In this paper, we present a two-level approach for scalable and accurate DDoS attack detection by exploiting the asymmetry in the attack traffic. In the coarse level, we use a modified count-min sketch (MCS) for fast detection, and in the fine level, we propose a bidirectional count sketch (BCS) to achieve better accuracy. At both detection levels, sketch structures are utilized to ensure the scalability of our scheme. The main advantage of our approach is that it can track the victims of attacks without recording every IP address found in the traffic. Such feature is significant for the detection in the high- speed environment. We also propose a SRAM-based parallel architecture to achieve high-speed process. Furthermore, we analyze accuracy estimation issues to provide hints for practical deployment with constraint memory. We finally demonstrate how to extend our original scheme to a col- laborative detection framework. Experimental results using the real Internet traffic show that our approach is able to quickly detect anomaly events and track those victims with a high level of accuracy while it can save over 90% key storage compared with previous sketch-based approaches. Index Terms—intrusion detection, distribute denial of service, victim pinpoint capability, two-level scheme, asym- metry traffic, count min sketch, bidirectional count sketch I. I NTRODUCTION In recent years, distributed denial of service (DDoS) attacks have posed one of the most serious security threats to the Internet [2]. DDoS attacks can do a great damage to the network service by exhausting resources of a server or an ingress network near a server. Since Internet-based services increases every day, the damage that DDoS caused is more severe than before. How to effective defend against such attacks still attract attention from both academia and industry. The very first step for approaching this goal is to effectively detect the attacks, which is difficult because distinguishing attack traffic embedded in a huge amount of background traffic from legitimate traffic is a hard work. Manuscript received August 15, 2011;accepted October 5, 2011. This paper is based on “Fine-Grained DDoS Detection Scheme Based on Bidirectional Count Sketch,” by H. Liu, Y. Sun, and M. S. Kim, which appeared in the Proceedings of IEEE International Conference on Computer Communication Networks (ICCCN), Hawaii, USA, August 2011. c 2011 IEEE. Over the past decades, many intrusion detection sys- tems (IDSs) have been proposed to fight against DDoS attacks. However, those existing schemes usually present a tradeoff between scalability and accuracy. That is to say, finer grained traffic monitoring can ensure the accuracy of detection while does not scale well. For example, two most popular open-source IDSs, Snort and Bro [3], [4], keep per-flow state to detect anomalies, which makes both of them not scale well in a high-speed network. Since the volume of the Internet traffic doubles every year, how to monitor a large amount of traffic in real-time is becoming more crucial in anomaly detection. Although dimensionality reduction proposed in [5], [6] may be effective in dealing with such a large data, it usually requires complex operations, and thus is impractical in real-time detection. Recently, a series of sketch-based approaches have been proposed for anomaly detection [7]–[10]. Sketch [11] is a data structure to store a summary of a large data set for space efficiency. Kompella et al. presented Partial Com- pletion Filters (PCF) by utilizing multiple hash tables for scalable attack detection in high-speed networks [7]. As we will see later, such scheme is essentially an simplified version of the original sketch [11]. The main drawback of their scheme is that it can only tell when an attack happens without providing any hint on where the anomaly occurs; the latter is critical in mitigating the attack at an early stage. Identifying victims is also useful in responding to attacks. For instance, an IDS can generate packet classification rules automatically based on the victim information, so as to minimize future damage. In order to provide sketches with the capability that can tell those keys with heavy change, a reversible sketch framework is proposed by [8], [9]. Such feature can be used to provide the victim pinpoint capability in DDoS detection. An improved reversible sketch is proposed by [10] in DDoS detection context. They proposed a flooding attack detec- tion method using a count-min sketch (CMS) with multi- channel nonparametric CUSUM (MNP-CUSUM) [10]. The CUSUM [12] is a change-point detection technique that can accumulate those small offsets during the process to amplify a varying statistical feature so as to improve the detection sensitivity. Although their improved scheme can detect flooding events effectively, it suffers from the following shortcomings which render it still insufficient to detect general DDoS attacks effectively. First of all, 660 JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011 © 2011 ACADEMY PUBLISHER doi:10.4304/jcm.6.9.660-670

Transcript of A Scalable DDoS Detection Framework with Victim Pinpoint Capability

A Scalable DDoS Detection Framework withVictim Pinpoint Capability

Haiqin Liu, Yan Sun and Min Sik KimSchool of Electrical Engineering and Computer Science

Washington State University, Pullman, USAEmail: {hliu, ysun, msk}@eecs.wsu.edu

Abstract—In recent years, various intrusion detection andprevention systems have been proposed to detect DDoSattacks and mitigate the caused damage. However, manyexisting IDS systems still keep per-flow state to detectanomaly, and thus do not scale with link speeds in multi-gigabit networks. In this paper, we present a two-levelapproach for scalable and accurate DDoS attack detectionby exploiting the asymmetry in the attack traffic. In thecoarse level, we use a modified count-min sketch (MCS)for fast detection, and in the fine level, we propose abidirectional count sketch (BCS) to achieve better accuracy.At both detection levels, sketch structures are utilized toensure the scalability of our scheme. The main advantageof our approach is that it can track the victims of attackswithout recording every IP address found in the traffic.Such feature is significant for the detection in the high-speed environment. We also propose a SRAM-based parallelarchitecture to achieve high-speed process. Furthermore,we analyze accuracy estimation issues to provide hints forpractical deployment with constraint memory. We finallydemonstrate how to extend our original scheme to a col-laborative detection framework. Experimental results usingthe real Internet traffic show that our approach is able toquickly detect anomaly events and track those victims witha high level of accuracy while it can save over 90% keystorage compared with previous sketch-based approaches.

Index Terms—intrusion detection, distribute denial ofservice, victim pinpoint capability, two-level scheme, asym-metry traffic, count min sketch, bidirectional count sketch

I. INTRODUCTION

In recent years, distributed denial of service (DDoS)attacks have posed one of the most serious securitythreats to the Internet [2]. DDoS attacks can do a greatdamage to the network service by exhausting resourcesof a server or an ingress network near a server. SinceInternet-based services increases every day, the damagethat DDoS caused is more severe than before. How toeffective defend against such attacks still attract attentionfrom both academia and industry. The very first step forapproaching this goal is to effectively detect the attacks,which is difficult because distinguishing attack trafficembedded in a huge amount of background traffic fromlegitimate traffic is a hard work.

Manuscript received August 15, 2011;accepted October 5, 2011.This paper is based on “Fine-Grained DDoS Detection Scheme Based

on Bidirectional Count Sketch,” by H. Liu, Y. Sun, and M. S. Kim,which appeared in the Proceedings of IEEE International Conference onComputer Communication Networks (ICCCN), Hawaii, USA, August2011. c⃝ 2011 IEEE.

Over the past decades, many intrusion detection sys-tems (IDSs) have been proposed to fight against DDoSattacks. However, those existing schemes usually presenta tradeoff between scalability and accuracy. That is to say,finer grained traffic monitoring can ensure the accuracyof detection while does not scale well. For example, twomost popular open-source IDSs, Snort and Bro [3], [4],keep per-flow state to detect anomalies, which makes bothof them not scale well in a high-speed network. Sincethe volume of the Internet traffic doubles every year,how to monitor a large amount of traffic in real-time isbecoming more crucial in anomaly detection. Althoughdimensionality reduction proposed in [5], [6] may beeffective in dealing with such a large data, it usuallyrequires complex operations, and thus is impractical inreal-time detection.

Recently, a series of sketch-based approaches have beenproposed for anomaly detection [7]–[10]. Sketch [11] isa data structure to store a summary of a large data set forspace efficiency. Kompella et al. presented Partial Com-pletion Filters (PCF) by utilizing multiple hash tables forscalable attack detection in high-speed networks [7]. Aswe will see later, such scheme is essentially an simplifiedversion of the original sketch [11]. The main drawback oftheir scheme is that it can only tell when an attack happenswithout providing any hint on where the anomaly occurs;the latter is critical in mitigating the attack at an earlystage. Identifying victims is also useful in respondingto attacks. For instance, an IDS can generate packetclassification rules automatically based on the victiminformation, so as to minimize future damage. In orderto provide sketches with the capability that can tell thosekeys with heavy change, a reversible sketch framework isproposed by [8], [9]. Such feature can be used to providethe victim pinpoint capability in DDoS detection. Animproved reversible sketch is proposed by [10] in DDoSdetection context. They proposed a flooding attack detec-tion method using a count-min sketch (CMS) with multi-channel nonparametric CUSUM (MNP-CUSUM) [10].The CUSUM [12] is a change-point detection techniquethat can accumulate those small offsets during the processto amplify a varying statistical feature so as to improvethe detection sensitivity. Although their improved schemecan detect flooding events effectively, it suffers from thefollowing shortcomings which render it still insufficientto detect general DDoS attacks effectively. First of all,

660 JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011

© 2011 ACADEMY PUBLISHERdoi:10.4304/jcm.6.9.660-670

WSU-0 WSU-1 AU-0 AU-1 CDAD-0 CDAD-1

101

102

103

104

105

# o

f p

acke

ts

(a) Average # of incoming packets per DIP

WSU-0 WSU-1 AU-0 AU-1 CDAD-0 CDAD-1

101

102

103

104

Ave

rag

e as

ymm

etry

ind

ex

(b) Average AI per DIP

WSU-0 WSU-1 AU-0 AU-1 CDAD-0 CDAD-110

0

101

102

103

104

Ave

rag

e d

isti

nct

# o

f so

urc

e IP

s

(c) Average distinct # of source IPs

Fig. 1. Measurement on Real Traces

their scheme only takes the high frequency of packets ina flow as the evidence of an anomaly event. However, thisfeature of traffic alone is not enough to detect an anomaly.For example, a Flash Crowd event, which is caused by alarge number of legitimate users simultaneously accessingthe same server during historical events, can also resultin an outburst of the traffic. Their scheme will lead toa large false positive in such a case. Moreover, theirscheme suffers from the scalability problem. Because ofthe key recovery issue in the original sketch scheme theyexploited, their method has to record every incomingdestination IP (DIP) for the key recovery later, whichmakes it unscalable to a large amount of traffic dueto the huge memory consumption. Finally, it applies amulti-channel CUSUM algorithm to every bucket in thesketch, which requires heavy computations in high speednetworks.

In this paper, we propose a two-level approach forDDoS detection. We are motivated by the fact that atypical DDoS attack traffic possesses three characteris-tics: high frequency of incoming packets, asymmetry ininteraction patterns, and high diversity of source IP (SIP)addresses. Our modified count-min sketch (MCS), bidi-rectional count sketch (BCS), and distinct IP addressesestimator are designed precisely for detecting these threecharacteristics. Although sketch is also used in our workto achieve high scalability, the differences between theprevious sketch-based detection approaches and ours liein the following ways:

a) Memory consumption: Our scheme outperformsprevious works in terms of the space requirement. Byutilizing the two-level model, most of benign trafficinformation, which may be considered as a redundancyfor IDS systems, does not need to be recorded in thesystem. Moreover, the traditional key recovery process insketch [9] requires sketches to record every input key,which will consume much space, especially when a largenumber of DIPs are involved in high-speed networks.By taking advantage of the high diversity of source IPaddresses feature, our approach can reveal the victim setwithout recording every destination IP as previous worksdo.

b) Searching time: Our scheme also can achievefaster detection than previous sketch-based approaches intwo aspects. Firstly, since most of traffic is benign, theadopted two-level scheme can greatly reduce the searchspace while the sketch adopted in [10] processes different

kinds of traffic equally. Secondly, rather than applyingCUSUM to multiple channels of each bucket in the highfrequency anomaly detection phase, we utilize a light-weight exponentially weighted moving average (EWMA)technique to achieve the same goal while introducingmuch less computing overhead.

c) Accuracy: By taking the asymmetry feature intoaccount, our approach can greatly reduce the false posi-tives by distinguishing between Flash Crowds and DDoSattacks, and thus can improve the overall accuracy.

The remainder of this paper is organized as follows.Based on the real Internet measurements, Section IIdescribes the data structures in our scheme and overviewsthe architecture. Section III presents the design and im-plementation in detail. Section IV analyzes the accuracyestimation and demonstrates a collaborative detectionframework. Experimental results are presented in Sec-tion V, and we conclude in Section VI.

II. SYSTEM DESCRIPTION

A. Measurement on Real Traces

The effectiveness of our scheme is based on the as-sumption that a typical DDoS attack traffic possessesthree characteristics: high frequency of incoming packets,asymmetry in the interaction patterns, and high diversityof source IP addresses.

Fig. 2 demonstrates the “asymmetry” in typical DDoSattacks for web services. We pay attention to the funda-mental difference in flow patterns between DDoS attacksand normal traffic. In order to exhaust resources at theserver side, an attacker (or more likely a large numberof “puppets” in his or her botnet) tries to generate asmany requests as possible. When the number of requestsexceeds the capacity of the server, we observe fewerresponses from the server than requests it receives. Wenotice that it is not always true that an IP address willserve as both source and destination of traffic, especiallywhen the UDP or ICMP protocol is involved. Thus, in thispaper we only refer to the TCP flooding attack by default.Let Nforward(i) denote the number of flows from othernodes to a server i, and Nbackward(i) the number of thoseflows that originate from the server i. By “flow,” we meana group of packets with the same pair of source IP (SIP)and destination IP (DIP) addresses. We do not considerthe port information in the flow because we only considerthe node-level interactions in our scheme. We define theasymmetry index AI (i) = |Nforward(i) − Nbackward(i)|

JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011 661

© 2011 ACADEMY PUBLISHER

a

b

c

d

e

f

Source Destination

(a) Normal

a

b

c

d

e

f

Source Destination

(b) Attack

Fig. 2. Flow patterns of normal traffic and DDoS attacks

for the server i. We expect that the server under aDDoS attack will exhibit a higher AI value than theone experiencing no attack. For example, Fig. 2(a) showsnormal interactions between clients and a server, andAI (f) is 0. On the other hand, in an attack scenariodepicted in Fig. 2(b), AI (f) is 4.

In order to support the above assumption, we sought toevaluate a set of public traces. We derived Internet trafficfrom Auckland University (AU) [13] and WashingtonState University (WSU) as the normal traffic and theattack traffic from “The CAIDA DDoS Attack 2007Dataset” (CDAD) [14]. The traces are labeled AU-0,AU-1, WSU-0, WSU-1, CDAD-0, CDAD-1, where thenumbers with each label are corresponding to differenttime periods from each trace. The default time intervalfor the measurements is 10 minutes.

We firstly measure the average number of the incomingpackets per DIP in each trace for comparison. Fig. 1(a)shows the results. We notice that in Fig. 1(a) the averagenumbers of the incoming packets per DIP of CDAD-0and CDAD-1 are both much larger than that of othertraces (nearly 100 times larger). We also measure theaverage number of flows per DIP and the average trafficvolume per DIP over every trace, and we get the similarresults that the numbers measured from CDAD are muchhigher than that of other traces. Thus, rather than using allthe three metrics (packet, flow, volume), we only utilizethe high frequency feature of incoming packets as theanomaly indicator during the coarse-level detection sincethese three metrics are highly correlated with each other inour measurement. We notice that some attacks might notpossess such correlations. For example, alpha flows [15]usually result in high volume of traffic while only havea small number of flows per DIP or even low-rate DDoSattacks [16] can have low value of all the three metrics.In this paper, we do not aim at the development of apanacea, which is very hard if not impossible, for allkinds of attacks. Instead, we only focus on the detectionof the general flooding attack in high speed networks.Other types of attacks need to be further filtered by extraworks. Regarding the asymmetry features, we measurethe asymmetry index of DIPs of each traces, the result ofwhich is shown in Fig. 1(b). We can see that the AI valueobtained from CDAD traces is greatly larger than that ofother traces. Similar results can be seen in Fig. 1(c) afterwe measure the number of distinct IPs that are associatedwith each DIP in all the traces.

1h

2h

Hh

1 ( )h k

2 ( )h k

( )Hh k

·

·

·

1 2 K· · ·

( , )k v

h

h

h

1 2 KFig. 3. Illustration of sketch data structure

B. Data Structure

K-ary sketch is a data structure to efficiently andaccurately estimate the original signals by aggregatinghigh dimensional data streams into fewer dimensions.As shown in Fig. 3, it consists of H hash tables ofsize K. A hash function for each row is selected inde-pendently and randomly from a set of hash functions.Each data item contains a key ki and an associatedvalue vi. When a new item si = (ki, vi) arrives, itsvalue vi is added to those buckets corresponding to thekey ki. The CMS Query(key) function can return theminimum value among all the buckets corresponding toa specific key. In case of hash collisions, the collidingkeys will be listed in the bucket for the key recoverypurpose later. The key recovery process [9] can revealthose keys with high frequency in the sketch by lookinginto the intersection set of high value buckets acrossthe whole sketch. The recovery process is crucial intracking victims, which will greatly benefit in respondingto attacks. Our proposed approach makes two importantchanges to this original sketch structure: modified count-min sketch (MCS) and bidirectional count sketch (BCS),which will be introduced in the Section II-C.

C. System Architecture

Coarse level

detection

Fine level

detection

Anomaly

with key?Anomaly?

Y

N

N

Y

Generate alarms and the

victim set

Input key

streaming

Output

streaming

Fig. 4. High-level view of detection process

The overall framework of our detection system isshown in Fig. 4. During each detection period, the relatedinformation of every incoming packet is inserted intoMCS for the coarse-level detection. We use DIP as thekey, and the number of packets that are destined forthat IP address as the associated value. MCS only main-tains counters for input IP addresses. Compared with theoriginal CMS structure, our MCS structure utilizes spacemore efficiently because no information on IP addressesthemselves is stored in this structure. Besides, unlikeCMS, MCS does not rely on CUSUM; MCS is used onlyfor coarse-grained filtering and a light-weight EWMAtechnique is applied to each bucket to determine whetherto generate an alarm for this bucket or not. Whenever anincoming packet satisfies the condition that every bucket ithashed into has an alarm signal, it will trigger the second

662 JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011

© 2011 ACADEMY PUBLISHER

stage for finer-grained detection, where a new structurecalled BCS is used.

In BCS, those suspicious flows detected in the coarse-level detection in both directions are mapped into buckets.While the general sketch structure in which a value ineach bucket can increase only, a bucket value in BCSmay increase or decrease. We demonstrate how we applyBCS to exploiting the asymmetry of the attack traffic inSection III-B. Once an attack is detected, we use a light-weight distinct IP addresses estimator to pinpoint victimsthat have the most number of distinct sources, which is astrong indication of DDoS attacks.

III. SCHEME DESIGN AND IMPLEMENTATION

In this section, we describe the detection processes ofboth coarse and fine level in details and then explain howthe proposed distinct sources estimator works and whyit can help to indicate DDoS attacks. Finally, a SRAM-based parallel architecture is proposed to achieve high-speed process.

A. Coarse-level Detection

In our scheme, each bucket in the MCS contains fourvalues (vt, vt−∆t, vbackup,Flag). vt is the number ofpackets that are accumulated from t − ∆t to t, vt−∆t

is the previous value of vt, and vbackup is the value ofvt right before the alarm occurs (or null if there hasbeen no alarm). Flag is set to 1 whenever the alarmcondition is satisfied; otherwise it is set to 0. Here, thedefinition of alarm conditions depends on the practicaldeployment, and we will further explain it when wedescribe Algorithm 1 below. For each incoming record,we update the sketch with (ki, 1) where ki is the DIPand 1 represents the number of this incoming record. Inour MCS, rather than returning the minimum value ofvt as the original sketch does, the CMS Query functionin MCS returns the minimum value of Flag among allthe buckets corresponding to a specific DIP to indicatewhether it is under flood attacks. As we can see, thesketch adopted here only requires O(H×K) cells, whichis constant.

The main purpose of MCS is to detect items withabnormal frequency at the coarse level. It is the firststage in the system, which every packet must go through.When a new packet arrives, hash values of H hashfunctions are computed, and the corresponding bucketsare updated; the value in each bucket is incremented by1. This accumulation process repeats every ∆t seconds.The alarm condition is tested for all H × K bucketsperiodically. If the alarm condition is satisfied, then thealarm flag associated with the bucket is set to 1. Wheneverthere is an alarm, the previous v value of the bucket isrecorded in the vbackup for determining whether the raisedalarm is terminated or not.

We use an EWMA technique to decide whether thereis an anomaly in each bucket, as shown in Algorithm 1.For each bucket, if the bucket status is normal, then weestimate vt with an EWMA parameter α. Whenever vt ≥

Algorithm 1: Adjustment procedure of Flag1 for k = 1, h = 1 to K,H do2 if Flag = 0 then3 vt ← (1− α)vt−∆t + αvt ;4 if vt ≥ (1 + θ)vt−∆t then5 Flag ← 1 ;6 vbackup ← vt−∆t ;7 end8 else9 vt ← (1− α)vbackup + αvt ;

10 if vt < (1 + θ)vbackup then11 Flag ← 0 ;12 end13 end14 end

1h

2h

Hh

·

·

·

1 2 K· · ·

Bloom filter 1

Bloom filter 2

Bloom filter H

·

·

·

DIP 1

Counter

DIP 2 DIP 3 DIP 4

BucketDIP

max

min

DistNum

Fig. 5. Illustration of BCS data structure

(1 + θ)vt−∆t, which is considered as the satisfaction ofthe alarm condition, an alarm is raised. θ is the parameterthat represents the percentage above the estimated valuethat can be considered to be an indication of anomalouspattern. The procedure is different after an alarm wasraised. In order to estimate when the generated alarmshould be terminated, we need to compare the currentvalue with the specific value right before the time thatthe alarm happened. Such specific value is recorded invbackup before the alarm is generated. Also, rather thanusing the previous value vt−∆t, we estimate the vt byvbackup in order to eliminate the impact of the anomalyon the next following vt series.

We can do the coarse-level detection by queryingthe minimal value of alarm flag for a specific key. IfCMS Query(key) = 1, then there may be an anomalyassociated with the key. However, the coarse-level de-tection would yield a certain number of false positives.There are two possible reasons for false positives. Thefirst possibility is hash collisions, which can be reducedby carefully selecting hash functions or enlarging the sizeof the sketch. The second possibility is flash crowds.They can also yield many items with high frequenciesin the sketch. Thus, we need to examine traffic further toseparate these possibilities from true attacks, which is thegoal of our next technique, BCS, which detects anomaliesat the finer level.

B. Fine-level Detection

The objective of the fine-level detection is to find outthose DIPs exhibiting high asymmetric communication

JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011 663

© 2011 ACADEMY PUBLISHER

Algorithm 2: BCS update procedure in the forwarddirection

1 for h = 1 to H do2 k ← BCS[h].hash(DIP ) ;3 if DIP is not in BCS[h][k].list then4 insert DIP into BCS[h][k].list ;5 update BCS[h].BF by DIP |SIP ;6 BCS[h][k].counter ++ ;7 else8 if DIP |SIP is not in BCS[h].BF then9 update BCS[h].BF by DIP |SIP ;

10 BCS[h][k].counter ++ ;11 end12 end13 DistinctSourcesEstimator(SIP ) ;14 end

patterns. A successful DDoS attack employs a largenumber of zombies to exhaust resources of the target side.However, they are usually unaware of the exact capacityof the server. Therefore, to guarantee to overwhelm theserver, an attacker sends as much traffic as allowed,exceeding the server’s capacity. This results in highlyasymmetric communication patterns between clients andthe server, as shown in Fig. 2(b). There are two reasonsfor such an asymmetry pattern. First, the capacity of theserver, namely the victim, to respond is limited while theattacker can keep launching new connections. Second,the SIP addresses of attack traffic are forged, and thusthe server has to abort communications. During sometime interval ∆T , for a specific IP address, we candetect whether this address serves as both source anddestination or not. If yes, then the corresponding flow (thesource and destination IP pair containing this IP address)can be considered as a normal flow. Motivated by thisobservation, we propose the BCS structure to monitorsuch kind of anomaly with fine granularity.

During one time interval, we use DIP as the key inupdating the BCS structure. An illustration of BCS sketchis shown in Fig. 5. All the keys with hash collisions willbe stored as a list in the corresponding bucket. Ratherthan incrementing corresponding H counters by 1 everytime a new packet arrives, the counters increase onlywhen the DIP belongs to a new flow. For example, aflow (si, di), where si denotes the SIP address of nodei and di is the DIP address of node i, will contribute tothe corresponding H buckets only once during a singleperiod. On the other hand, (sj , di), which is another flowwith the same destination di, will contribute another 1 tothe buckets that the key di is hashed into. Since we do notneed to record SIP addresses in the sketch, we employ Hbloom filters (BF) with m bits and kbf hash functions asan ancillary structure to estimate whether a specific flowthat new packets belong to has been inserted to the BCSstructure or not. Algorithm 2 presents the details of howBCS works on the forward direction of traffic. We use |as the string concatenation operator.

The procedure for the backward traffic is shown inAlgorithm 3. Whenever we find a backward flow that

Algorithm 3: BCS update procedure of the backwarddirection

1 for h = 1 to H do2 k ← BCS[h].hash(SIP ) ;3 if SIP is in BCS[h][k].list then4 if SIP |DIP is in BCS[h].BF then5 BCS[h][k].counter −− ;6 end7 end8 end

can be paired with an existing forward flow in the BCSstructure, the corresponding counter decreases. In thisway, the counters with anomalous high values indicatean anomaly event caused by asymmetric communicationpatterns for a specific victim.

C. Distinct Sources Estimator

In order to avoid being detected, attackers may employa large number of SIP addresses. In such cases, thoseDIPs that are associated with the largest distinct SIPaddresses should be a good candidate for a victim underattack. Thus, how to find the number of distinct SIPsfor a victim is crucial in the DDoS defense. Withoutrecording the SIP addresses in the system, which requirestoo much memory, we need to find a way to estimatethis number. For each DIP that is hashed into BCS, wepick a hash function h : N → [0, 1] which maps everynumber into [0, 1], and then we apply h(·) to all the SIPaddresses that are associated with this DIP, and maintainsthe maximal value max and minimal value min , and thenthe number of distinct IP addresses, DistNum , can beestimated as 1

2 ·(

1min + 1

1−max

). If the hash function that

we choose is sufficiently random, then the above formulais a sufficiently good estimator for our purpose. In thisway, each DIP which has been hashed into the BCS willbe associated with a number: DistNum . For a specificDIP, this number DistNum can be used as an indicatoron how diverse the corresponding SIPs are.

D. Victims Identification

At the end of each time interval, for each row in theBCS, we compute the average counter value C[h] and thecorresponding mean square deviation D[h]. For a specificbucket, whenever its counter value BCS[h][k].countersatisfies the following condition, then it raises an alarmfor an anomaly:

BCS[h][k].counter − C[h] ≥ β ·D[h] (1)

where β is an adjustment factor that should be empiricallydetermined. Then, we merge those DIPs that correspondto those anomalous buckets together, and sort them bytheir DistNum . In addition, we eliminate those DIPs thatsatisfy the condition BCS Query(DIP) < THcounter

in the merged set, where BCS Query is similar tothe original CMS Query , which returns the minimumcounter value through all the hashed buckets in the sketch

664 JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011

© 2011 ACADEMY PUBLISHER

Fig. 6. Hardware architecture for the proposed scheme

BCS. THcounter is a threshold which can be empiricallydetermined. Finally, those victims can be chosen fromthe merged DIP set by picking the top few DIPs withthe largest DistNum value. Or, we can set a thresh-old THDistNum to select those victims that can satisfyDistNum ≥ THDistNum to process the selection ofvictims.

E. Hardware Architecture

Our proposed scheme can be implemented by hard-ware to achieve high speed process. Since the fieldprogrammable gate array (FPGA) technology has widelybeen utilized for real-time packet processing due to itscapability of reconfigure and parallelism, we propose aSRAM-based parallel architecture as shown in Fig. 6. Foreach input DIP of incoming packets, we perform the hashcomputations over the HMCS branches in parallel. TheCarter-Wegman H3 hash function [17] can be utilized inour hardware-based scheme, since the H3 hash functionmostly consists of XOR gates proportional to the numberof output bits, which can make it easily implementedin hardware. The bit values in the vector are initiallyset to 0 and they will be periodically reset to 0 atthe end of the detection interval. For each row at thefirst stage, whenever an alarm signal is generated, thecorresponding bit in the vector will be set to 1. Afterdoing the AND operation among all the bit values, it candecide whether to trigger the second stage detection ornot. Similarly, the HBCS branches during the finer leveldetection phase can also be executed in parallel. The BCSanomaly detector and ranking module can be implementedby FPGAs according to the flow logic discussed in theprevious sections. The overall search process can bedivided into several independent parts and it can bepipelined by assigning each part to a separate memoryblock to accelerate the overall processing speed. Forexample, the hash computation of the current incomingDIP and the anomaly detection of the previous DIP areindependent with each other and thus can be mapped intotwo different stages.

IV. ANALYSIS AND DISCUSSION

In this section, we firstly analyze the space requirementand then estimate the accuracy of our scheme. We furtherdemonstrate how to extend the current scheme to acollaborative detection framework.

A. Space Requirements

The primary space consumption of the system is due tothe two sketches (MCS and BCS) and H bloom filters thatare employed at the finer detection stage. Let Lmc denotethe length (in bytes) of each bucket in the sketch MCSand Lbc represent the same in the sketch BCS. Moreover,each bloom filter will occupy Lbf space. Suppose theMCS and BCS have the size Hmc ·Kmc and Hbc ·Kbc,respectively, the total memory requirement will be:

Hmc ·Kmc · Lmc +Hbc ·Kbc · Lbc +Hbc · Lbf (2)

According to our method proposed above, the length Lbc

will not be a constant value because the length of thelinked list varies. In the practical deployment, we are ableto limit the maximal number of the nodes in the link list.For example, for a specific link list, we can only keepthose top few DIPs that are associated with the highestDistNum value in the list.

B. Accuracy Estimation Analysis

We further sought to quantify the impact of the sizeof sketches to the overall accuracy of our framework. Weestimate the accuracy of our system in terms of “falsepositive rate” (FPR) and “false negative rate” (FNR). Inorder to simplify the problem, we conduct our analysis onan assumption that the “false negative rate” of both MCSFNRmc and BCS FNRbc are negligible and we willdemonstrate why this assumption holds in our schemebelow. At the presence of certain number of maliciousflows, the overall “FPR” depend on the accuracy perfor-mance of individual modules (MCS and BCS). Thus, wefirstly define false positive rate of MCS FPRmc and BCSFPRbc, and then demonstrate how they contribute to theoverall false positive rate.

JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011 665

© 2011 ACADEMY PUBLISHER

Since MCS and BCS are both proposed based onsketch, we firstly conduct a general analysis of the sketchstructure. Let us assume there are m different maliciouskeys and n buckets in each row of a sketch. Since theprobability that a specific bucket is not hashed by amalicious key is 1 − 1

n , the probability that a specificbucket is not hashed by every malicious key is

(1− 1

n

)m.

Therefore, the probability that a bucket in a row is hashedby at least one malicious key is 1 −

(1− 1

n

)mand the

expectation of the number of buckets to which these mmalicious keys hash is n

(1−

(1− 1

n

)m). When m is

much less than n, we have:

n

(1−

(1− 1

n

)m)= n

(1−

(1− 1

n

)−n·−mn

)≈ n

(1− e−

mn

)when n >> 1

= m · 1− e−mn

mn

= m ·1− [1 +

(−m

n

)+

(−mn )

2

2! + · · · ]mn

≈ m (when n >> m ≥ 1)(3)

We define those buckets that are hashed by maliciouskeys as malicious buckets. Therefore, when n >> m,the number of the malicious keys can be used to estimatethe expected number of malicious buckets in a row. Fora key that is hashed into a malicious bucket in eachrow of sketch, whether it is benign or not, our schemewill judge it as a malicious key, which is the maincause of false positives of sketch scheme. We assumethat there are totally N distinct incoming keys, whichcontains N ·Pnormal normal keys, N ·PflashCrowd keysassociated with “Flash Crowd” events and N ·PDDoS keyswith DDoS events. Pnormal, PflashCrowd and PDDoS arethe proportion of normal keys, keys with “Flash Crowd”and keys with DDoS to the total number of differentkeys, respectively. Based on the above definition, we havePnormal+PflashCrowd+PDDoS = 1. For the analysis, wecall those keys that are related to “Flash Crowd” events as“Flashcrowd” keys and keys associated with DDoS eventsas DDoS keys.

In the coarse level detection, both “Flashcrowd” andDDoS keys are considered to be positive (malicious)instances. Therefore, based on the Eq. 3, the probabilitythat a key is hashed into one of these malicious bucketsin one row is given by N ·(1−Pnormal)

Kmc. For Hmc rows, the

probability is:

FPRmc =

(N · (1− Pnormal)

Kmc

)Hmc

(4)

For the fine level detection, only DDoS keys are consid-ered to be positive (malicious) instances. Similarly, theprobability that a key is hashed into one of these maliciousbuckets for each row in BCS is given by:

FPRbc =

(N · PDDoS

Kbc

)Hbc

(5)

We define the overall false positive rate by the definition:

FPRoverall =Total # of false positive instances

Total # of negative instances(6)

For the overall scheme, negative instances contains nor-mal keys and “Flashcrowd” keys and false positives meanthose negative instances that are wrongly judged as DDoSkeys. Thus, we have:

FPRoverall

=Total # of false positive instances

Total # of negative instances

=NPnormalFPRmcFPRbc +NPflashCrowdFPRbc

N (PflashCrowd + Pnormal)

=PnormalFPRmcFPRbc + PflashCrowdFPRbc

PflashCrowd + Pnormal

(7)

From the Eq. 7, we can see that the overall false positiverate depends on the distribution of traffic and false positiverate of each individual module. We can estimate the over-all false positive rate FPRoverall by Eq. 7. For example,suppose the total number of distinct DIPs is 1000, andthere are 180 “Flashcrowd” keys and 20 “DDoS” keys.Let us assume that Kmc = 1024, Hmc = 10 for MCS andKbc = 128, Hbc = 5 for BCS. According to the Eq. 4and Eq. 5, we have FPRmc = ((180 + 20) /1024)

10 ≈8.08 × 10−8 and FPRbc = (20/128)

5 ≈ 9.31 × 10−5.Therefore, FPRoverall can be estimated as:(0.8×8.08×10−8×9.31×10−5+0.18×9.31×10−5)/(0.18+0.8) ≈1.72× 10−5.

In order to further demonstrate the impact of variousfactors on FPRoverall, we vary different parameters anddraw figures according to Eq. 7 as shown in Fig. 7. Thedefault settings for the parameters we adopted are Hmc =10, Kmc = 1024 for the sketch size of MCS, Hbc = 5,Kbc = 128 for the size of BCS and Pnormal = 0.8,PflashCrowd = 0.18 for the traffic distribution.

Fig. 7(a) and Fig. 7(b) show the impact of sizesof MCS and BCS on the overall false positive rate,respectively. From both of these two figures, we cansee that by enlarging the size of sketches (either H orK), we can greatly reduce the overall false positives.Although keeping the size of sketches large will benefitthe accuracy performance, it will also consume muchmemory space, which will be unaffordable for practicaldeployment. In practice, we should carefully design thesize of sketches employed in the scheme based on spacerequirements. In Fig. 7(c), we show the impact of trafficdistribution on FPRoverall, where F2M is defined as theratio of the number of “Flashcrowd” keys to maliciouskeys (including “Flashcrowd” keys and DDoS keys). Wecan see that the FPRoverall decreases as the proportion of“Flashcrowd” keys increases when we fix Pnormal. Thisis because the probability that a normal key is hashed intoa malicious bucket decreases as the number of maliciousbuckets diminishes.

Regarding the false negative rate, we first consider thefalse negative rate of MCS FNRmc and BCS FNRbc.

666 JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011

© 2011 ACADEMY PUBLISHER

(a) Size of MCS VS. FPRoverall (b) Size of BCS VS. FPRoverall (c) Traffic distribution VS. FPRoverall

Fig. 7. The relation between various factors and the estimated overall false positive rate

Since we define those buckets that are hashed by themalicious keys as malicious buckets and we considerthose keys that are hashed to malicious buckets in everyrow of sketches as malicious keys, there will be no falsenegative ideally. In our MCS stage, it is possible that somefalse negatives can be caused by the improved EWMAtechnique. For example, some of buckets, which shouldbe considered as malicious buckets, still do not generatethe corresponding alarm signal for detection. However,this case is much less often, because the malicious keyswill always have much higher incoming frequency thanthe normal keys. Similar things happen at the BCSstage. Although there might be some false negatives inBCS due to the inherent false positive issue of bloomfilters employed, such case rarely happens for the muchlower false positive rate of bloom filters compared withsketches. From the perspective of the overall framework,positive instances only consist of DDoS keys and falsenegatives are those DDoS keys which are classified asnormal or “Flashcrowd” keys by mistake. Thus, accordingto the definition of false negative rate, we have:

FNRoverall

=Total # of false negative instances

Total # of positive instances

=NPDDoSFNRmc +NPDDoSTPRmcFNRbc

NPDDoS

= FNRmc + TPRmcFNRbc

= FNRmc + FNRbc (Since TPRmc = 1)

(8)

Where TPRmc is the true positive rate of MCS. The truepositive rate for MCS is approximately equal to 1 basedon the definition of the positive instances in MCS. Sinceboth FNRmc and FNRbc are negligible, the overall falsenegative rate also can be neglected. Moreover, from Eq. 8,we can see that the false negative rate is irrelevant to thedistribution of the incoming traffic.

C. Collaborative Detection Scheme

Till now, our proposed two-level framework can becategorized as a host-based system, which can be de-ployed at an ingress router near the victim side. Thenearer the detection module from victims is, the largeramount of attack traffic we can observe. Thus, in order toreduce the difficulty of detection, one possible solution isto deploy the proposed detection module at the targetedserver. However, this preliminary solution is a bad ideafor two reasons. First, one deployment can only protect

Global detector

Local detector

Feedback

loopEdge

router

Edge

router

Edge

router

Local detector

Local detector

Core

network

AS

Subnet

Subnet

Subnet Subnet

Fig. 8. Illustration of a collaborative framework

one victim which render it not scale well. Secondly, itcannot even well protect the victim it supposed to protect.Because the ingress bandwidth resources near the victimserver can be exhausted as well by the attack traffic, whichwill result in the same effect to the legitimate users sincethey cannot visit the victim server. Thus, a deploymentthat is a little far from a victim server might be a goodchoice.

However, a single host-based system is inherently notrobust enough no matter where it is deployed. It isentirely possible that some unaware or intentional internetbehaviors can damage its effectiveness. For instance, dueto network device failure problems or a specific routingprotocol designed for congestion avoidance, a backwardtraffic associated with an original forward flow might berouted by a totally different path. As a result, the trafficasymmetry feature no longer can be observed by a singlerouter. Furthermore, such scheme can be easily fooled bya sophisticate attacker, which can be considered as anintentional internet event. Since attackers always employa large number of zombie machines around the worldto launch attacks, traffic that comes from every cornerof the world can be routed by different edge routersinside an AS. Thus, if we only take a single routerinto accounts, the volume of attack traffic might not beaggregated at a detectable level for a detection modulewhile the final gather of attack traffic will still cause se-vere damage to victim servers. Therefore, a collaborativedetection approach which can comprehensively considerthe global circumstance will be an attractive solution.Fortunately, our proposed approach can be easily extendedto a collaborative detection scheme, which will greatlyreinforce our original work. Fig. 8 illustrates the overallcollaborative framework. The edge routers are responsiblefor connecting subnets (it can be customer networks orother ASes) with the core network. Our collaborativedetection framework contains multiple local detectors, one

JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011 667

© 2011 ACADEMY PUBLISHER

global detector and a feedback loop between them. Thefunctionality of each component is described as below.

a) Local Detector: A local detector can be deployedat an edge router, and it is responsible for:

• Summarizing traffic statistics from partial or allpackets from both of two directional links

• Report the summarized traffic statistics to the globaldetector periodically

• Receive feedback instructions from the global de-tector and adjust the local information collectionmanner based on the feedback instructions

• Timely react to those DDoS events that can bedetected at the local side

To be specific, a local detector maintains two mainthreads. The first thread is called “update thread”, whichkeeps scanning every incoming packet and updates thetraffic profile. The second thread, which we called as“report thread”, periodically sends the built traffic pro-file to the global detector and finally refreshes theprofile after reporting. A hash table with linked listscan be utilized for the traffic profile building. Eachentry in the profile hash table contains six values(DIP, num, suspF lag,min,max, sipP tr) with DIP askey value. num accumulates the number of those in-coming packets associated with DIP during one period.suspF lag, which is set based on the feedback instructionsfrom the global detector, can be used to decide whether toupdate the remaining values in one entry or not. Whenevera DIP is suspected by the global detector due to its highpacket frequency, the suspF lag will be set to 1. WhensuspF lag = 1, the “update thread” will keep updating thefollowing values (min,max, sipP tr) in an entry. minand max maintain the minimal and maximal of hash valueby mapping all SIPs associated with the DIP into range(0, 1) as we did in the distinct sources estimator. At thesame time, sipP tr, which is a head pointer of a linkedlist, will be updated by inserting those SIPs into the list.As we can see, by reporting the built traffic profile, theglobal detector can obtain all the necessary informationfor further anomaly analysis.

b) Global Detector: The responsibility of a globaldetector contains:

• Receive those statistics reports from local detectors• Perform anomaly detection based on packet fre-

quency at coarse-level detection phase• Perform anomaly detection based on both the distinct

number of SIPs and asymmetry feature associatewith each DIP at fine-level detection phase

• Send feedback instructions to local detectors basedon anomaly detection results

The global detector also maintains two threads. The firstthread, which we called as “MCS thread”, is respon-sible for updating MCS and sending feedback to localdetectors. The MCS update process is similar as wedescribed in Section III. The total incoming frequencyassociated with a DIP can be obtained by:

∑Mk=1 numk,

(1 ≤ k ≤ M), where M is the total number of localdetectors that report their local frequency num of this

DIP to the global detector. When a key is detectedas suspicious key with high packet frequency duringMCS detection phase in the global detector, those hashentries associated with this key at local detectors willbe marked as suspicious by setting suspF lag to 1. Wecalled the second thread as “BCS thread”. The “BCSthread” also does similar works as we have demon-strated in Section III. The min and max value asso-ciated with certain DIP in BCS can be obtained by:min = MIN (min1,min2, · · · ,minM ) and max =MAX (max1,max2, · · · ,maxM ). As we can see, ouroriginal scheme can be extended in a distributed-executingway quite smoothly. Besides those advantages we pointedout before, one great benefit by running in a distributedway is that the workload of the central global detectorcan be largely reduced. As a result, the scalability perfor-mance can be further improved.

We notice that the number of one packet will becounted twice in a typical AS infrastructure. One countis at the ingress router and the other one is at the egressrouter. Similar thing happens when we measure the countfor traffic asymmetry. However, it will not impact theoverall performance, because both malicious and benigntraffic will be amplified by the same proportion when wemeasure the frequency feature. Regarding the asymmetryfeature, both forward and backward traffic will be countedtwice, the effect of which will be offset to each other whenwe measure the asymmetry feature in BCS.

V. EVALUATION

We evaluate the performance of the proposed schemevia simulations. We use the trace data from AU [13] asthe background traffic. It contains packet traces capturedfrom the link connecting Auckland University and theInternet. This background traffic, which contains bothforward and reverse directions, has an average rate of 523packets per second. We consider the accuracy of victimidentification and the amount of memory consumption astwo main performance metrics. Unless otherwise noted,the default settings for the parameters we adopted inour experiment are ∆t = 5s for the periodical sketchconstruction, Hmc = 32, Kmc = 1024 for the sketchsize of MCS, Hbc = 5, Kbc = 128 for the size of BCS,Lbf = 10000Bytes for bloom filters, α = 0.4, θ = 0.5for the coarse level detection, ∆T = 5s, β = 2 andTHcounter = 10 for the fine-level detection.

A. Detection Accuracy Evaluation

We generate the flooding traffic using attack tools wedeveloped. The attack rates vary from 25 to 500 packetsper second (25, 50, 75, 100, 200, and 500) and theduration of each the attack is 20 seconds. Those attacksare injected at the offset of every 100 seconds. Our goalis to try to gauge the detection sensitivity of our schemeunder a large range of attack rates. Fig. 9(a) shows themaximal DistNum series among all the detected victimsin the sketch BCS as the time goes. The six spikes

668 JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011

© 2011 ACADEMY PUBLISHER

0 100 200 300 400 500 600 70010

0

101

102

103

104

Time (s)

Max

imal

dis

tNu

m in

BC

S

(a) Maximal DistNum value in BCS

0 100 200 300 400 500 600 7000

10

20

30

40

50

Time (s)

# of

vic

tims

Fine-level detection

Coarse-level detection

MNP-CUSUM

(b) # of detected victims

0 100 200 300 400 5000

20

40

60

80

100

Attack rate (Packets/s)

Rec

all r

atio

(%)

Fine-level detection

Coarse-level detection

MNP-CUSUM

(c) Recall ratio

Fig. 9. Evaluation of accuracy

(excluding the smallest one) indicate all six DDoS attackswe injected. Even when the attack rate is as low as 25packets per second, which happens at the offset of 100seconds, our scheme is still able to identify such low rateattacks while maintaining high accuracy. The maximalDistNum values well reflect the rates of correspondingattacks. After we manually inspected the backgroundtraffic, we found that the remaining spike with the lowestvalue in Fig. 9(a) represents a low rate flooding attack inthe original trace. Fig. 9(b) demonstrates the number ofvictims that are identified by the coarse-level and fine-level detection. On average, the coarse-level detectionidentify 12 victims per interval. All of those victimsexperience high rates of requests, which may be causedby flash crowds or DDoS attacks. However, after wefurther filter those potential victims using the fine-leveldetection, at most one victim per interval remains, whichis the actual attack contained in the traffic. Moreover, theaverage victim number detected by the MNP-CUSUMapproach [10] is around 21, which is even higher thanthe coarse-level detection of our approach. This is becausethe original CUSUM technique does not take care of thequick termination after the alarm happens, which resultsin that too many buckets in the sketch remains highvalue for a long time. Therefore, it usually causes manyfalse positives. After we modified the original CUSUMtechniques by a method for quickly terminating alarmas proposed in [19], the average number is significantlyreduced to around 12, which can be due to flash crowds.

We also measure the recall ratio under different attackrates. A recall ratio is the fraction of the true victims in theestimated victims returned by our scheme. The estimatedvictims identified by the coarse-level detection is the setof all DIPs which satisfy CMS Query(DIP ) = 1. InFig. 9(c), we can see that the recall ratio of the fine-level detection is very stable; nearly 100% of victims areaccurately identified. Even when the attack rate is as lowas 25 packets per second, the recall ratio is still over 95%.However, with the coarse-level detection only, the ratio ismuch lower. It requires more than 350 packets per second(about 66% of the background traffic rate) to achievethe ratio over 95%. Again, due to the alarm terminationproblem, the MNP-CUSUM technique performs poorlyhere. Its recall ratio is around 23% on average.

0 100 200 300 400 500 600 7000

100

200

300

400

500

600

700

800

Time (s)

Tota

l # o

f key

s

Our approach

MNP-CUSUM

(a) Space consumption

1 2 3 4 5 6 7 80

500

1000

1500

2000

2500

3000

3500

Merging factor

Tota

l # o

f key

sOur approach

MNP-CUSUM

(b) Storage scalability

Fig. 10. Evaluation of space consumption

B. Space Consumption

We also sought to measure the memory consumption.Basically, the overall space consumption of sketch-basedapproaches consists of two different parts. The first part,which can be attributable to the sketch structure itself,takes constant size of small space while the other part,which serves for assisting functions such as the keystorage, occupies dynamic size. Since the scalability per-formance of sketch-based approaches greatly depends onthe dynamic part, we compare our approach against [10]by measuring the number of keys that should be stored.The results are shown in Fig. 10(a). During one interval,there are 47 keys that are needed to be stored in ourscheme on average while the average number of the keysof MNP-CUSUM approach is around 519. Our approachcan save up to 90% keys, which translates to less memoryconsumption and searching space, when comparing withthe previous approach. In order to evaluate the storagescalability, we shift time stamps of different periodsof traces from AU and then merge them together inorder to enhance the traffic intensity. We define “Mergingfactor” as the number of different periods, which can alsoreflect the intensity of the traffic. Then, we measure therequired key storage over various approaches as shown inFig. 10(b). Our method nearly keeps constant number ofkeys when the merging factor increases, while the MNP-CUSUM holds a linear-like trend in the same case. Thatis because our method only record those suspicious DIPsrather than storing every DIP.

JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011 669

© 2011 ACADEMY PUBLISHER

From the total number of keys in the sketches and thedefault parameter settings, the total memory consumptionof our scheme can be estimated using the Eq. 2. Theaverage memory cost is around 563.6 KB, which weconsider can be easily accommodated in modern routers.

VI. CONCLUSION

In this paper, we present a fine-grained DDoS detectionscheme based on the BCS structure to counter the threat ofDDoS attacks. Our approach employs a two-level modelto reduce both the size of the search space and time, andfurther make identification of specific victims possiblein the high-speed network environment. We adopt theMCS structure in coarse-level detection to achieve fastdetection, and the BCS structure in the fine-level to furtherguarantee the accuracy. We believe that this approachcan accurately identify victims of DDoS attacks witha low memory footprint and give a timely response.We also propose a SRAM-based parallel architecture toachieve high-speed process. We finally analyze accuracyestimation issue and demonstrate a collaborative detec-tion scheme based on the original single-host detectionscheme. Experimental results show that our scheme out-performs previous sketch-based methods with respect toboth storage scalability and detection accuracy.

Our future work will focus on designing a collaborativedefense framework against DDoS attacks. Our proposeddetection scheme can be used to facilitate defense againstDDoS attacks in the following way. Since all the victimscan be accurately detected by our collaborative scheme, anautomatical rules generator can be developed to reinforcefirewall and IDS systems in real-time.

ACKNOWLEDGMENT

We thank the reviewers for their detailed comments.

REFERENCES

[1] H. Liu, Y. Sun, and M. S. Kim, “Fine-grained DDoS detectionscheme based on bidirectional count sketch,” in Proceedingsof IEEE International Conference on Computer CommunicationNetworks, Hawaii, Aug. 2011.

[2] J. Mirkovic and P. Reiher, “A taxonomy of DDoS attack and DDoSdefense mechanisms,” ACM SIGCOMM Computer CommunicationReview, vol. 34, no. 2, pp. 39–53, Apr. 2004.

[3] M. Roesch, “Snort - lightweight intrusion detection for networks,”in Proceedings of the 13th USENIX Conference on System Admin-istration, Nov. 1999.

[4] V. Paxson, “Bro: A system for detecting network intruders in real-time,” Computer Networks, vol. 31, no. 23–24, pp. 2435–2463,Dec. 1999.

[5] A. Lakhina, M. Crovella, and C. Diot, “Diagnosing network-widetraffic anomalies,” in Proceedings of ACM SIGCOMM, Aug. 2004.

[6] ——, “Mining anomalies using traffic feature distributions,” inProceedings of ACM SIGCOMM, Aug. 2005.

[7] R. R. Kompella, S. Singh, and G. Varghese, “On scalable attackdetection in the network,” IEEE/ACM Transactions on Networking,vol. 15, no. 1, pp. 14–25, Feb. 2007.

[8] R. Schweller, A. Gupta, E. Parsons, and Y. Chen, “Reversiblesketches for efficient and accurate change detection over networkdata streams,” in Proceedings of the ACM SIGCOMM InternetMeasurement Conference, Taormina, Sicily, Italy, Oct. 2004, pp.207–212.

[9] R. Schweller, Z. Li, Y. Chen, Y. Gao, A. Gupta, Y. Zhang, P. Dinda,M.-Y. Kao, and G. Memik, “Reverse hashing for high-speednetwork monitoring: Algorithms, evaluation, and applications,” inProceedings of IEEE INFOCOM, Apr. 2006.

[10] O. Salem, S. Vaton, and A. Gravey, “A scalable, efficient andinformative approach for anomaly-based intrusion detection sys-tems: theory and practice,” International Journal of NetworkManagement, vol. 20, pp. 271–293, Sept. 2010.

[11] A. C. Gilbert, Y. Kotidis, S. Muthukrishnan, and M. J. Strauss,“Quicksand: Quick summary and analysis of network data,” DI-MACS, Tech. Rep. 2011-43, 2001.

[12] M. Basseville and I. V. Nikiforov, Detection of Abrupt Changes:Theory and Application. Prentice Hall, 1993.

[13] “Auckland-IV trace data,” 2001, http://wand.cs.waikato.ac.nz/wand/wits/auck/4/.

[14] P. Hick, E. Aben, K. Claffy, and J. Polterock, “The CAIDADDoS Attack 2007 Dataset,” http://www.caida.org/data/passive/ddos-20070804 dataset.xml (accessed on 2010-02-28).

[15] S. Sarvotham, R. Riedi, and R. Baraniuk, “Network traffic analysisand modeling at the connection level,” in Proceedings of InternetMeasurement Workshop, San Francisco, Nov. 2001.

[16] A. Kuzmanovic and E. W. Knightly, “Low-rate TCP-targeteddenial of service attacks and counter strategies,” IEEE/ACM Trans-actions on Networking, vol. 14, pp. 683–696, Aug. 2006.

[17] J. L. Carter and M. N. Wegman, “Universal classes of hashfunctions,” Journal of Computer and System Sciences, vol. 18,no. 2, pp. 143–154, 1979.

[18] H. Liu, Y. Sun, V. C. Valgenti, and M. S. Kim, “Trustguard: Aflow-level reputation-based DDoS defense system,” in Proceedingsof the 5th IEEE International Workshop on Personalized Networks,Las Vegas, Jan. 2011.

[19] H. Liu and M. S. Kim, “Real-time detection of stealthy DDoSattacks using time-series decomposition,” in Proceedings of IEEEInternational Conference on Communications 2010, May 2010.

Haiqin Liu received the B.S. degree inelectrical engineering from Harbin Insti-tute of Technology, Harbin, China, in2005, the M.S. degree in network com-munication system and control from theUniversity of Science and Technologyof China, Hefei, China, in 2008. He iscurrently a Ph.D. candidate in computerscience at Washington State University,

Pullman. His research interests include network security, contentdistribution network and peer-to-peer network.

Yan Sun received the B.S. degree inapplied physics in 2005 from Univer-sity of Science and Technology, Beijing,China, the M.S. degree in 2008 in mi-croelectronics from the University of Sci-ence and Technology of China, Hefei,China, and is currently a Ph.D. candidatein Computer Science, Washington StateUniversity. His research interests include

network security, high-performance VLSI systems and computerarchitectures.

Min Sik Kim received the B.S. degreein computer engineering from Seoul Na-tional University, Seoul, Korea, in 1996,and the Ph.D. degree in computer sci-ence from the University of Texas atAustin in 2005. At present, he is anAssistant Professor of computer sciencewith the School of Electrical Engineeringand Computer Science, Washington State

University, Pullman. Prior to joining Washington State Univer-sity, he cofounded and served as Chief Technology Officer ofInfnis, Inc., from 2002 to 2004. His research interests includeoverlay networks, network monitoring, and network traffic anal-ysis.

670 JOURNAL OF COMMUNICATIONS, VOL. 6, NO. 9, DECEMBER 2011

© 2011 ACADEMY PUBLISHER