EFFORT: A New Host-Network Cooperated Framework forEfficient and Effective Bot Malware Detection

Seungwon Shin, Zhaoyan Xu, and Guofei Guseungwon.shin@neo.tamu.edu,{z0x0427,guofei}@cse.tamu.edu

ABSTRACTBots are still a serious threat to Internet security. Althougha lot of approaches have been proposed to detect bots athost or network level, they still have shortcomings. Host-level approaches can detect bots with high accuracy. How-ever they usually pose too much overhead on the host. Whilenetwork-level approaches can detect bots with less overhead,they have problems in detecting bots with encrypted, eva-sive communication C&C channels. In this paper, we pro-pose EFFORT, a new host-network cooperated detectionframework attempting to overcome shortcomings of bothapproaches while still keeping both advantages, i.e., effec-tiveness and efficiency. Based on intrinsic characteristics ofbots, we propose a multi-module approach to correlate in-formation from different host- and network-level aspects anddesign a multi-layered architecture to efficiently coordinatemodules to perform heavy monitoring only when necessary.We have implemented our proposed system and evaluatedon real-world benign and malicious programs running onseveral diverse real-life office and home machines for severaldays. The final results show that our system can detectall 17 real-world bots (e.g., Waledac, Storm) with low falsepositives (0.68%) and with minimal overhead. We believeEFFORT raises a higher bar and this host-network cooper-ated design represents a timely effort and a right directionin the malware battle.

1. INTRODUCTIONBotnets (networks of bot malware controlled machines)

are considered as one of the most serious threats to cur-rent Internet security[10, 32]. To eradicate threats posed bybots/botnets, a lot of research has been proposed so far, andthey fall into two main categories: (i) network-level detec-tion and (ii) host-level detection. Network-level detectionapproaches focus on network behavior of bots/botnets andthey typically concentrate on finding signatures or commoncommunication patterns between bots and their masters [11,10, 12, 32]. Host-level detection approaches investigate botruntime behavior in the host and they mainly employ sys-tem call monitoring and/or data taint analysis techniques[18, 20, 28].Both detection approaches have their own advantages and

disadvantages. Network-level approaches can detect differ-ent types of bots without imposing overhead to the hosts,because they mainly monitor network traffic. However, theirlimitations appear when they need to detect a bot commu-nicating through encrypted messages or randomized traffic[29]. Host-level approaches, on the contrary, analyze suspi-cious runtime program behavior, so that they can detect abot even if it uses an encrypted or evasive communicationchannel. However, they typically suffer from performanceoverhead because they need to monitor all invoked systemcalls [18] at real-time and/or taint memory locations touchedby the program [28].Observing their clear advantages and disadvantages mo-

tivates us to consider a new system with merits from bothapproaches: (i) effectiveness and (ii) efficiency. For the effec-tiveness, the system should detect malware with few misses.In addition, the system should not put too much burden onboth host and network to achieve the efficiency.

As a promising step toward such a system, we proposeEFFORT, a new detection framework balanced with highaccuracy and low overhead. EFFORT considers both host-and network-level features that are helpful to enhance strongpoints of each other and complement weak points of eachother, and it coordinates these features to achieve the maingoal (i.e., detecting bots effectively and efficiently).

To build EFFORT, We start with investigating severalnotable intrinsic characteristics of recent popular botnets.First, bots are usually automatic programs without requir-ing human-driven activities. Second, bots highly rely onDNS (instead of hard-coded IP address in binaries) for flex-ible and agile C&C (command and control). In addition,they use more advanced DNS tricks for robustness and eva-sion, such as fast-flux service networks or even domain flux-ing [14]. Third, bots access system resources anomalously(e.g., registry creation and file read/write) to steal systeminformation or launch themselves automatically. Finally,bots are likely to distribute their information for their mali-cious activities to the network (e.g., sending massive spam)instead of gaining information from the network which iscommon in normal networked user applications and theytend to minimize incoming C&C command communicationto reduce the exposure/detection probabilities.

Based on their characteristics, we find several useful fea-tures at host and network level. For efficiency, we per-form lightweight human-process-network correlation analy-sis. We correlate interactions between human and process,and record correlated clues between processes and outgoingDNS connections. Thus, we can filter majority benign pro-grams, and focus on very few suspicious automatic programscontacting DNS servers and start further investigation.

To detect bots effectively, we further employ three inter-esting new modules. First, we monitor system resource ex-posure patterns of the suspicious process. Second, we builda reputation engine to characterize the reputation of a pro-cess through examining the process and its social contact-ing surfaces. Our intuition is that the reputation of a pro-cess could be approximately inferred by the reputation ofits social contact surface, i.e., reputations of communicatedremote hosts. This is intuitively sound because bots arelikely to communicate with “bad” targets while good soft-ware tends to communicate with “good” ones. Althougha pre-established host reputation database (or blacklist) ishelpful, we do not require it as prior knowledge. Instead, wecan use anomaly-based features from DNS registration infor-mation. Furthermore, we want to leverage community-basedknowledge and intelligence by using public search engines tolocate information about certain communicated targets andthen infer their reputation. Third, we analyze network in-formation trading rate for any network process to infer how

1

likely the program is information gaining oriented or infor-mation leaking/outgoing oriented.It is well accepted that there is probably no any single

module/feature can detect all bots. We do not claim thatbots cannot evade any of our proposed individual module.Instead, we rely on a correlative approach to combine in-formation from multiple complementary modules for a finaldecision, because the chance that a bot evades all our mod-ules is very slim unless they sleep or behave like normalprograms.In short, our paper makes the following contributions.

• We propose a new host-network cooperated frameworkfor bot malware detection with correlative and coor-dinated analysis. This design demonstrates an impor-tant step from current state of the art toward botheffective and efficient botnet detection.

• We implement EFFORT, a first-of-its-kind real-worldprototype system containing several novel modules tocover bot invariants at both host and network levels.

• We extensively evaluate our system on real-world datacollected on many real-life machines for several days.Our results show that EFFORT can detect all 17 real-world bots and it has only 8 false positives (out of 1,165benign processes) in about a week of testing (withoutany whitelisting). We demonstrate that EFFORT hasalmost negligible overhead for host modules and itsnetwork modules such as process reputation analysisare scalable.

2. SYSTEM DESIGN

Figure 1: EFFORT Design Architecture. M1 ishuman-process-network correlation analysis module, M2 isprocess reputation analysis module, M3 is system resourceexposure analysis module, M4 is network information trad-ing analysis module, and M5 is correlation engine.

The overall architecture of EFFORT is shown in Figure1, which contains five modules:

• Human-process-network correlation analysis module. Itanalyzes the interaction and correlation between hu-man activity, process, and network connection. It tellswhether a network connection is human-driven or bot-driven.

• Process reputation analysis module. It characterizesthe reputation of a process from the process itself (whoyou are) and its social contact surface (the communi-cated targets, i.e., whom you have talked to).

• System resource exposure analysis module. It examinesthe system resource exposure patterns to a suspiciousprocess in detail.

• Network information trading analysis module. It moni-tors incoming/outgoing network traffic in order to inferthe information gain/loss in a very light-weight way.

• Correlation engine. It collects all analysis results andcorrelates them to make a final decision whether theprocess is likely a malicious bot or not.

The overall operation of EFFORT is summarized as fol-lows. First, Human-process-network correlation analysis mod-ule finds a process producing bot-driven network connec-tions and notifies other three modules - Process reputation,System resource exposure, and Network information tradinganalysis modules - about the found process. Second, thesethree modules investigate the suspicious process in detailand each module issues a detection result (but not a finaldecision) by its own analysis engine. Finally, Correlationengine collects the detection results from the above mod-ules and finally decides whether the process is malicious ornot (final decision).

2.1 Human­Process­Network Correlation Anal­ysis

Since most running processes are benign, it is relativelyinefficient to monitor all of them in fine-grained detail (e.g.,monitor system call level activities) all the time. Thus,our human-process-network correlation analysis module isdesigned to sift benign programs out.

Human-Process Interactions Monitoring: Key-board and mouse are the basic components that link humanand the computer. We monitor keyboard and mouse eventsof the host to understand which program has human activ-ity/interaction. To do this, our event sensor hooks Windowssystem calls related to keyboard and mouse events, and alsodetermines which program generates those events.

Some previous studies (e.g., [6]) also use hooking to cap-ture automated processes. We differentiate from them inseveral aspects. Previous approaches will suffer if a bot sim-ply imitating human behaviors and creating fake mouse/keyboardevents from virtual devices to confuse the sensor. To ad-dress this problem, we employ two more robust approaches.First, our sensor digs into the sources of the events. If theevents are resulted from physical devices connected via PS2or USB interface, it trusts them; otherwise it regards themas suspicious. Second, the module investigates whether aprocess generating events is running in foreground or not.We assume that if someone produces mouse or keyboardevents, a process related to the events is shown on the cur-rent screen with an activated windows (i.e., running in fore-ground). Thus, if a process producing events is running inforeground, we trust the process; otherwise we regard theprocess as suspicious.

Note that in current implementation, we trust operatingsystem and we believe it provides true information, a com-mon assumption widely used in this line of research [18, 6].Of course, some malware (e.g., rootkit) may infect operatingsystem and deliver fake information or even kill our system.This issue could be solved by employing hardware/TPM[13] or Hypervisor-based introspection and protection [9, 16](and thus is out of the scope of this paper).

Process-Network Interactions Monitoring: Fur-ther different from some previous studies mainly focusing on

2

identifying automated processes, we also trace the process-network interaction to identify automated processes thatgenerate automated network connection (particularly DNSqueries). We use a connection sensor to record outgoing net-work connections from processes in a host. In particular, itcares about one special network connection, DNS query. Asbriefly discussed before, botnets heavily rely on using DNSfor flexible, efficient, and evasive C&C rallying. They caneven use fast-flux service networks [14] to frequently changethe IP addresses associated with one domain name, or evenuse domain fluxing [27] to frequently change domain names.By monitoring these DNS queries, we can obtain valuableinformation later in detection analysis, e.g., we can deter-mine if they are human driven or not, and furthermore wecan even detect if there is fast-flux in use.Interaction Model Generation and Automatic Con-

nection Detection: Combining information from the eventsensor and the connection sensor, we create a model to de-scribe which process has which network correlations. Themodel employs three metrics: (i) time difference betweenthe time when a process issues a DNS query and the priortime when a process produces a mouse/keyboard event, (ii)the source of the events, and (iii) whether a process is run-ning foreground or not at the time. We regard an event ofa process is generated from human, if the time difference isvery small, the event is from actual physical devices, andthe process is running foreground.Practical Considerations: However, in practice, this

intuitive model may not work for all DNS queries (but workwell for IP address case). The reason is because some operat-ing systems provide helper functions of relaying a DNS queryfor other processes, e.g., Windows uses the svchost.exe pro-cess for this purpose for some sockets. Thus, DNS queriesare sent from helper processes instead of the original pro-gram. To address this issue, we maintain a list of returnedIP address(es) from a DNS query (sent by helper processes),and observe successive outgoing network connections to waitfor finding the actual program (process) to connect to thereturned IP address(es)1. If we find the process, we cancreate a model by using metrics (mentioned above) for theprocess.

2.2 Detecting Malicious ProcessesWith the help of the previous module, we can focus on

some suspicious processes. However, we need more deepanalysis to investigate whether they are really malicious ornot. To do this, we perform a set of independent and par-allel checks. We check the reputation of the process andits social contact surface (the reputation of targets it hascommunicated with). We investigate the system resourceexposure patterns to the process. Furthermore, we investi-gate network information trading of the process. We detailour design of these three modules as follows.

2.2.1 Process Reputation Analysis ModuleFirst we use a novel approach to determine the reputa-

tion of the suspicious process. A quick intuitive observa-tion is that we could determine the reputation of a processby not just looking at “who you are”, but also referringto “whom you have talked to”. Bots are likely to contactsome “bad/suspicious” servers/peers automatically in order

1At this time, we do not need to monitor all network con-nections, we only monitor first packet of each connection.

to be controlled. On the contrary, benign programs are rel-atively unlikely to connect to “bad” targets automatically.Thus the problem of determining the reputation of a processcould be roughly inferred by the contacting social surfaces ofthe process and it can be approximately reduced to the ac-cumulation of “bad” communication targets (domains). Inthe context of domain names, then we need to determinethe reputation of a domain name.

Domain Information Collection: We collect repu-tation information of the domain by employing three typesof sensors. First, we employ a whois sensor to detect someanomaly features in its registration information, such as do-main creation date. Second, we use a blacklist sensor to in-vestigate its previous records in well-known blacklists (e.g.,SpamHaus [26]), which give us relatively clear clues whetherthe domain has a history of malicious activity or not. Fi-nally, since blacklists might not be complete, we apply asearch engine sensor to get another heuristic that can lever-age community-based knowledge and intelligence, i.e., ask-ing a search engine to infer the reputation of given domainnames (IP address could work too), which is motivated bythe googling idea in [30].

Feature Extraction and Normalization: Before ap-plying collected data to a model creation, we express thefeatures numerically and normalize them. In terms of do-main registration information, we use the following features(the intuitions of the expressions are described in brackets):(i) difference between current date and domain expirationdate (most malicious domains registered very recently), (ii)difference between domain expiration date and creation date(most malicious domains have short life time), and (iv) num-ber of domain registration (malicious domains are typicallyregistered to few name servers).

An interesting and novel component in the process repu-tation module is our search engine sensor. The general intu-ition is that some of the knowledge about the domain or pro-cess is probably already discovered/reported/summarizedby other people in the world. Thus, we can leverage thewisdom of the whole Internet community by using searchengines like Google. More specifically, we consider the fol-lowing simple yet effective features: (i) whether the domainname is well-indexed (thus returning many results), (ii) inthe top returned web page results, whether the domain nameand the process name are frequently used in a malicious con-text, e.g., they are surrounded by malicious keywords suchas bot, botnet, malware, DDoS, attack, spam, identity theft,privacy leak, command and control (C&C). We also use nu-meric values to represent these features. Typically, contentsof returned search results include three different types of in-formation: (i) the title, (ii) the URL, and (iii) the relevantsnippet of the returned web page. We treat each type as dif-ferent features, and we assign “1” if returned contents (title,URL, and summary) include the queried domain name. Weinspect the returned results to see whether there are anymalicious keywords or not. If there are any, we give “0” forits value.

In the case of the blacklist, it is very obvious that if adomain can be found in blacklists, it is suspicious. We give“0” if it is in blacklists, otherwise “1”. The features relatedto returned results by a search engine and blacklist are al-ready normalized, i.e., their values are between “0” and “1”.However, features of domain registration can be varied dy-namically. To make their values range between “0” and “1”,

3

we employ a Gaussian normalization approach. It regardsa distribution of data as Gaussian function and maps everydata point into the probability of Gaussian function.Process Reputation Model Creation: We employ a

Support Vector Machine (SVM) classifier [5] for the processreputation model. The SVM classifier maps training exam-ples into feature spaces and finds (a) hyperplane(s) whichcan best separate training examples into each class (the de-tailed information of the SVM classifier will be explained inAppendix A.1).In this model, we consider that the normalized features,

which are mentioned above, are training examples. In ad-dition, we define that there are two classes - benign andmalicious - in this model, thus the normalized features willrepresent one of the two classes. Finally, we will find (a) hy-perplane(s) which can best separate training examples intoeach class. Then, we can obtain a SVM classifier for theprocess reputation model.Anomalous Process Reputation Detection: It is

very frequent that a process contacts several different do-mains during a certain period. Thus, we examine all con-tacted domains using our trained SVM model, and deter-mine whether “bad” domains (i.e. classified as maliciousdomains) exist or not. If there exists at least one, we con-sider the process reputation as bad (malicious), otherwise itis good (benign).

2.2.2 System Resource Exposure AnalysisIf a bot infects a host, it usually tries to do something

useful for its master (to make profit), e.g., stealing infor-mation, sending spam, and launching DDoS attacks [15].These operations consume system resources - memory, cpu,and network - of the host, read/modify files or registries [20].If we monitor how system resources are exposed to a process(and to what degree), we could infer its anomalous accesspatterns.System Resource Exposure Patterns Monitoring:

A system resource exposure sensor monitors resource accessactivities of a suspicious process. It monitors how criticalresources such as files, registries, and network sockets areexposed to the target process. Although this sensor sharessome similar motivation with related work [20], we use dif-ferent and more light-weight features and detection modelsas described below.System Resource Exposure Model Creation: To

build this model, we use the following heuristics: (i) typi-cally normal processes rarely access files in other user’s fold-ers and system directories, (ii) typically normal processes donot modify critical registries (with a few exceptions), and(iii) typically normal processes do not create a large num-ber of sockets in a short time period. These heuristics arenot perfect, i.e., some normal processes might have someof these patterns. Our goal of this module is not to havezero false positive, instead, we want to detect most of thesesystem-resource-consuming malware. Thus, we believe theseheuristics are reasonable.More specifically, these heuristics can be represented as

the following events Yi of a process:• Y1: access files in other user’s folders• Y2: access files in system folders• Y3: modify critical registries• Y4: create a new process• Y5: create too many network sockets within a short

time windowTo build a system resource exposure model, we employ a

one-class SVM (OCSVM) classifier [24]. We use one-classinstead of two-class SVM is because we will only use benignprograms in training. To get the ground truth informationof the system resource usages of malware is tricky, e.g., somemalware may refuse running or behave normally. Thus, evenif we obtain the information of malware, it may not representits behavior clearly. To address this issue, we only use thesystem resource access patterns of known benign processes(i.e., one side of data). More detailed information on ouruse of OCSVM is explained in Appendix A.2.

2.2.3 Network Information Trading AnalysisTypically, most user programs will act as clients rather

than servers, and clients will try to gather information ratherthan distributing information. That is, if we treat a programas a communication information processing unit, normalclient programs are more likely to be an information gain-ing process. However, a bot will behave differently. Usually,the data that a bot receives is a command from a botmaster,therefore the amount of the data may be small (to minimizethe chance of being detected). However the data sent outby the bot could be relatively large as it performs maliciousoperations in the network. Information theft, DDoS attack,and massive spam sending are good examples.

Lightweight Network Traffic Monitoring: To ob-serve network information trades, a network sensor capturesnetwork flows between a process and a target address andstores them. An important thing here is that this sensormonitors network traffic generated by the specific processnot by the host. It could give us more fine-grained obser-vations of network information trading. Our sensor is verysimple and lightweight, because it does not need to analyzepayload contents and it is robust against encryption used bybots.

In addition, we monitor the host level network connec-tions to obtain an aggregated view of network informationtrading. At this time, the sensor only measures the numberof outgoing connection trials (i.e. TCP SYN packets andfirst UDP packets). We believe that this aggregated viewgives us a good clue to find DDoS, network scan, or massivespam mail sending.

Network Information Model Creation: We use asimple method to model the network information trade rate,i.e., the ratio of incoming and outgoing packets/bytes ex-changed between a process and a remote site in a certainperiod. We define the number of incoming and outgoingpackets as θ1 and θ2, and the number of incoming and out-going bytes as δ1 and δ2. Thus, each ratio can be representedas θ1

θ2and δ2

δ2.

To observe an aggregated view, we employ a time win-dow wi for each host i. We measure how many networkconnection trials happen in the time window.

Anomalous Information Trading Detection: In thecase of the fine-grained view, if one or both of the predefinedratio values of a process is (are) smaller than some thresholdγ1 (for packet) and γ2 (for bytes), we consider the processanomalous, otherwise normal. Also, we consider the networkbehavior of the host is anomalous, if a host creates networkconnection trials larger than a threshold τ in wi.

2.2.4 Correlation Engine

4

Module Evasion Ways Evasion Costshuman-process-network correlation

compromise process which people frequently use,and initiates connections when user actually run

very hard to run bots in the host

process reputation use benign domains for C&C channels hard to control bots, thus less efficientsystem resource ex-posure

do not access system and other users’ folders, donot create processes, do not create a lot of net-work sockets

very hard to perform malicious activities

network informa-tion trading

do not distribute information to remote site orsend more information to victim host

hard to perform DDoS and send spam email mas-sively, more chances of botmaster being exposedto public

Table 1: Evasion ways and associated costs for each module.

After each module makes its own decision, the correlationengine will combine these results and make a final decisionusing a weighted voting system. It is worth noting thatas any intrusion detection system, most of our individualmodules might be evaded by very carefully designed bots.Indeed in the evaluation we will show that most moduleswill have some false negatives and/or false positives. How-ever, when combining all modules together, we can achievemuch better results, as demonstrated in Section 4. We alsonote that even individual evasion is possible, it will com-promise the utility and efficiency of bots. And to evade allour modules is extremely hard without significantly compro-mising the bots’ utility or even rendering the bots useless.We extensively discuss possible evasion attempts and theirimplications, possible solutions in Section 6.At the correlation stage, we should determine the weights

of the decision of each module. We can also employ SVMtechnique to determine which element (i.e. decision resultof the module) is more important (i.e. should have moreweight) [24]. To apply the SVM technique, we need trainingexamples of both sides - malicious and benign. However,here we have the same issue as the system resource exposuremodel creation mentioned in Section 2.2.2. It would be rela-tively difficult to collect all the information of the maliciousside. Thus, we decide to employ OCSVM to determine theweight [24]. The way how to determine the weights is sameas the method explained in Appendix A.2.

2.3 Evasion Ways and Associated Costs forEach Module

It is possible for botmasters to evade each module. Ifthey want to evade the human-process-network correlationanalysis module, their bots should not produce automaticnetwork connections. Thus, bots have to compromise a pro-gram which is very frequently used by people and contacttheir masters when people are actually using the program.Moreover, the contacts have to be done when people cre-ate real mouse or keyboard events. It might be very hardto implement bots which can meet all conditions describedbefore.Likewise, we have thought possible evasion ways of our

modules and the costs from the evasions, and we summarizethem in Table 1. As explained in Table 1, to evade ourmodule, botmasters consider a lot of different factors and itmakes them hard to implement new bots. Although theybuild a new bot, which are able to evade all our modules,the bot might be useless because it is very hard for the botto perform malicious activities in the host. We leave a moredetailed discussion of evasion in Section 6.

3. SYSTEM IMPLEMENTATION

3.1 Host­Level Modules ImplementationOur human-process-network correlation analysis module

captures the mouse and keyboard events using Windowssystem functions. Basically, Windows provides functions tocapture the events from external devices [23]. Using theseAPIs, we implement the event sensor which identifies whichprocess generates the events. In addition, it investigateswhether the events are generated from real physical devicesand the process is running foreground with the help of Win-dows system functions. We also add the function to storecaptured information (process, event time) to the SharedMemory area.

To capture the outgoing DNS queries, TCP SYN, andUDP packets, we use the WinPcap library [31]. It providesfunctions to collect raw level network packets on the Win-dows OS, with little overhead. Moreover, connection sen-sor does not monitor all network packets, but monitor onlyDNS, TCP SYN and UDP packets. It also reduces the over-head, since those packets comprises a small portion of allnetwork packets.

Whenever there are network events we should capture,our module also identifies which process produces them andverifies whether the process is related to the human actionsor not. However, if a process uses a helper process for a DNSquery, we could not directly use it. To address this problem,we check the process that produces the DNS query auto-matically and if it is a helper process (e.g., svchost.exe), themodule waits a DNS reply which contains the IP address ofthe domain. Then, if there is an automatic connection fromthe process to that IP address after the DNS query, themodule knows that the process issues the DNS query. Weuse GetExtendedTcpTable and GetExtendedUdpTable func-tions to recognize which process creates the connections. Ifwe observe the TCP or UDP connection, we will call thesefunctions to identify which process acquires the source portnumber of the connection.

We implement the system resource exposure analysis mod-ule based on EasyHook [8]. EasyHook is a successor andenhanced version of Detours [7], and it provides an interfaceletting us performWindows API hooking. The hooking abil-ity allows us to observe how a process works and which sys-tem calls are invoked by the process. We select 28 systemcalls to understand the exposure patterns of the process.The selected system calls are related to the access of thesystem resources, such as files, registries, network sockets,and creation of a process. In addition, we employ TinySVMlibrary [19] to create the system resource exposure model.

5

3.2 Network­Level Modules ImplementationTo gather network features, the process reputation anal-

ysis module should utilize multiple network services suchas whois services, blacklist identification services, and websearching services. Whenever the module receives a suspi-cious process and its contacting domains, it sends a queryto multiple network services to gather network features andcache them for a while. Also, we use TinySVM library [19]to create the process reputation model. For a network infor-mation trading analysis module, we capture network packetsusing the Pcap library.

3.3 Correlation Engine ImplementationWe implement the correlation engine as an independent

process (using TinySVM library [19]) and it will wait fora message from each detection module, and finally decidewhether the process is malicious or not.

4. EVALUATIONIn this section, we first show our real-world data collec-

tion of both benign and malicious programs. We discussour model training and followed by real-world bot detectionresults. We then discuss our false positive test. Finally, wereport the efficiency, performance overhead, and scalabilityof each module in the EFFORT system.

4.1 Data Collection and Usage

4.1.1 Benign Data CollectionWe have installed our modules into 11 different real-life

hosts to collect the information of process activities andnetwork behaviors for several days. These 11 machines areused by diverse users (including some lab members, friendsin different majors, housewives) in the office or home. Thesemachines are used in real-life operation for diverse usages,e..g, some users use network heavily for finding (or sharing)some information from (or through) networks, and someusers mainly use for scientific simulation or regular doc-ument editing. The collection has been done in workinghours on business days. We carefully examine to make surethat there are no malicious programs (especially bots) in thehosts, thus we consider that the collected data can be usedas benign examples.We explicitly collect data in two periods for different pur-

poses: training and testing. We have collected training datafrom 6 machines and the collected data is denoted as SET-1.Later we have collected testing data on 8 machines (amongthem, 3 machines are also used for training data collection,but 5 machines are newly added). The data for testing isdenoted as SET-2. Note that we intentionally test sev-eral new machines that we have no training data collected.This is to demonstrate that our training models are prettygeneric and not sensitive or limited to specific programs ormachines. Some detailed information of SET-1 and SET-2is summarized in Table 2.In terms of normal WIndows programs installed on these

machines that have generated network communications, theyare very diverse, covering application programs such as browsers,multimedia applications, Windows Office programs and P2Papplications. In training dataset SET-1, we have 61 distinctprograms. And we have 71 programs in SET-2 to evaluateour system (false positive test). Program examples are listedin Table 3.

Date Set Program ExamplesSET-1 Google update, MS Word, Tortoise SVN, Vmware, Gom

player, Bittorrent, Java, WinSCP, WeDisk (p2p pro-gram), Emule, Internet Explorer, Chrome, FireFox,iTunes, Thunderbird, EndNote, Skype, Putty, Adobe-Updater, MS Sidebar, Visual Studio 2010, GoogleTalk,QQ (Chinese chatting program)

SET-2 Google update, Gom Player, Tortoise SVN, Internet Ex-plorer, Chrome, MS Word 2010, Outlook 2010, GomAudio, McAfee update, FireFox, Skype, SohuNews, MSPowerpoint, Google Talk, Eclipse, AirVideo, QQ, Km-player, Bittorrent, Emule, Windows media player, Drop-box, Windows live toolbar, MS Clip Organizer, WindowsError Reporter, Alzip, MS windows defender, WindowsTask Manager, Vmware, MS Office Protection, Adobesynchronizer, SeaPort (MS search enhancement brokerprogram), Sophos Security update, Putty, WeDisk

Table 3: Example Benign Programs in SET-1 andSET-2

4.1.2 Bot Malware Data CollectionTo test the false negative or detection rate on real-world

bots, we build a virtual environment to run several col-lected real-world bots. The environment consists of threevirtual machines which individually served as an infectedhost, a controller, and a monitor machine. All of them in-stall Windows XP SP3 operating system with basic softwareinstalled, such as Internet Explorer browser and MicrosoftMessenger. At the infected host, we create independentsnapshot for each individual malware instance to ensure nocross-infection between different malware. Our host-basedmodules are also installed to collect the information of pro-cess activities and network behaviors for these bots. At themonitor machine, we install a fake DNS server to redirect allthe DNS queries. At the controller side, we install variousmalware controllers we could find to manipulate the infectedmachine. We intend to reconstruct realistic attack scenariosthat a botmaster sends commands to his zombie army.

We have used a total of 17 different bots (including Pea-comm/Storm, Waledac, PhatBot). Their names, C&C pro-tocols, and sample functionalities are summarized in Table4. Since we just have binary samples of most bots exceptthree (B1, B2, and B5), we install and simply run them.Among them, 3 botnets (B1, B2, and B5) use IRC protocol,2 botnets (B4 and B10) use HTTP protocol, 2 botnets (B3and B4) use P2P protocols, and other 9 botnets use cus-tomized protocols. In addition, three botnets (B3, B4, andB7) use encrypted protocols to evade network-level detec-tion. In terms of their actively spreading time in the wild(i.e. when they infect victims highly), it varied from 2003(B7) to recent (B4, B16). Since these collected botnets cancover diverse cases (e.g. from old one to currently workingone, different types of protocols including encryption, andvarious kinds of malware functionalities), we believe thatthey can fairly validate our system’s detection rate (or falsenegative rate).

We note that the separate collection of bot malware dataand benign program data in our evaluation does not affectthe accuracy of false positive/negative testing because ourmonitoring/recording/detection granularity is per-process in-stead of per-host (and obviously a bot program is a separate,different program from a normal program). Thus, we caneasily mix the bot malware data and benign testing dataSET-2 to simulate real-world scenarios of bots running on

6

SET-1 (collected Nov. 2010) SET-2 (collected Apr. 2011)Host ID Usage Programs Connection Trials Collection Time Programs Connection Trials Collection Time

A Office 7 252 < 1 day - - -B Home 8 10,927 4 days - - -C Office 19 5,740 5 days - - -D Office 16 7,859 3 days 37 113,067 12 daysE Office 9 5,098 4 days 12 20,711 6 daysF Home 27 55,586 7 days 22 48,264 6 daysG Office - - - 16 12,169 4 daysH Office - - - 11 5,134 4 daysI Office - - - 8 17,373 4 daysJ Office - - - 13 4,776 5 daysK Home - - - 10 14,455 5 days

Table 2: Benign Dataset summary. (Programs represent the number of programs producing network con-nections)

ID Name Protocol Sample Functionalities

B1 PhatBot IRC Steal Key, Spam Mail Send,Network Scan

B2 JarBot IRC Kill Process, Steal KeyB3 Storm/Peacomm P2P ∗ OtherB4 Waledac HTTP,

P2P ∗Other

B5 PhaBot.α5 IRC OtherB6 Flux Custom Operate/Modify File, Kill

Process, Capture Desk-top/Screen,

B7 nuclearRat Custom∗

Download Update

B8 BiFrost Custom Operate File, Kill Process,Capture Screen, Steal Key

B9 Cone Custom Operate fileB10 Http-Pentest HTTP Operate File, Kill Process,

Capture ScreenB11 Lizard Custom Capture Screen, DDoSB12 PanBot Custom FloodingB13 Penumbra Custom Operate File, Create ShellB14 SeedTrojan Custom Download UpdateB15 TBBot Custom Capture Screen, Create ShellB16 Sality Custom OthersB17 Polip Custom Others

Table 4: Bots for Evaluation (Custom denotes a bot-net using its own protocol. ∗ represents the protocolis encrypted. Other denotes other network/systemmalicious operations not categorized in the table.)

some normal machines.

4.2 Model TrainingBased on SET-1 data, we have created detection models

for each module.Process Reputation Model: From the collected data

in SET-1, we find that processes have contacted 7,202 dif-ferent domains. In order to create the process reputationmodel, we extracted features as described in section 2.2.1.At this time, we consider that all collected domains are be-nign, hence we will use them to represent the benign class.We also need malicious domains to represent the maliciousclass. For that purpose, we have collected recent 150 mali-cious domains from [22] and also extracted the correspond-ing features. Using these collected features, we train a SVMmodel for the classifier.System Resource Exposure Model: We analyzed

system resource exposure patterns of benign processes tocreate the one-class system resource exposure model. Here,

we will only use information of benign processes and employa OCSVM classifier to build the model. To do this, we use77 collected benign processes information in SET-1. Repre-sentative benign processes here are the instances of browsers(e.g., Chrome, Firefox, IE), mp3 playing programs (e.g.,Winamp, Gom Player), p2p client program (e.g., Emule),and other programs such as MS Word, Visual Studio, Putty,Google Talk (more program examples are listed in Table 3).We extract each feature defined in Section 2.2.2 from theprocesses and build the OCSVM classifier.

Network Information Trading Model: We analyzedthe network flow patterns and verified that most benignuser programs act as clients instead of servers. We measurethe ratio between incoming packets (or bytes) and outgoingpackets (bytes). We apply the ratio of the incoming andoutgoing packets to discriminate a malicious process from abenign process. It is obvious that when a bot delivers itsown information to a master, we could detect them easilyby observing the ratio.

We investigate the number of network connection trials ofa host. We find in our training dataset that the maximumnetwork connection trials of a host within a certain timewindow (2 seconds) is 41. Based on this result, we simplyset the threshold τ as 49 (with 20% of error margin to beconservative).

Correlation Engine Model: To calculate weights forthe correlation engine, we select 27 benign processes, whichproduce network connections frequently from SET-1. Mostof them are processes of browsers, multimedia applications,and p2p client programs. We collect their detection resultswhich are performed by our detection modules. Then, wetrain an OCSVM classifier using the collected results anddetermine the weights.

4.3 Detection Results of Real­world BotsWe begin our false negative evaluation with the test of

the human-process-network correlation analysis module, fol-lowed by the results of other modules.

4.3.1 Detection Results of Automatic ConnectionsFirst, we test whether a bot program really generates au-

tomatic connections to remote servers and whether we canuse the human-process-network correlation analysis moduleto detect them. To test this, we installed each bot in a hostand leave it without any intervention. After a while, we findthat all installed bots issue automatic connections to someremote servers (to be controlled). All of the automatic con-

7

nections are captured by our human-process-network corre-lation analysis module and the detected information is de-livered to other upper-layer modules.

4.3.2 Detection Results of the Process ReputationModel

The process reputation analysis module receives domainnames that a bot process contacts. Then, the module an-alyzes the reputation of contacted domains. Since a botcontacts multiple domains, we analyzed all contacted do-mains. If the module finds any malicious domain from thecontacted domains, it considers the process malicious.The detection results on all bot programs are shown in Ta-

ble 5. As shown in the Table, the process reputation analysismodule detects 12 bots but misses 3 bots (B2, B3, and B4).

ID ContactedDomains

DetectedDomains

ID ContactedDomains

DetectedDomains

B1 1 1 B10 1 1B2 1 - B11 1 1B3 2 - B12 1 1B4 1 - B13 2 2B5 6 2 B14 1 1B6 3 2 B15 1 1B7 2 1 B16 4 2B8 3 2 B17 5 1B9 2 2 -

Table 5: Detection Results of Automatic Connec-tions

We investigate why our module missed these three. In thecase of B2 (Peacomm) and B3 (Waledac), both bots onlycontacted the remote server using direct IP addresses in-stead of domain names. Of course, we can also apply the IPaddresses to our module. However, unfortunately, their con-tacting targets are either private IP addresses (192.168.X.X)or some hosts for which we could not get any useful infor-mation from the third parties.B4 (JarBot) contacts a regular IRC server and the server

has been operated for several years and we could not findany malicious keyword from search results.

4.3.3 Detection Results of the System Resource Ex­posure Model

Receiving the information of an automatic connection trialfrom the human-process-network correlation analysis mod-ule, the system exposure analysis module begins examiningthe target process.When we test the functionality of each malware listed in

Table 4, we find that the system exposure analysis mod-ule detects most of the malicious operations because manyof these malware programs access system resources anoma-lously. The detection results are summarized in Table 6(marked with “S”).It only misses 2 malicious operations of “B6 (Flux)”, the

first operation is to operate file which moves/creates a filein the host, and the second operation is to capture screenwhich takes a snapshot of the current screen.When we analyze their resource exposure patterns, we

find that their operations are very similar to normal pro-grams. In the case of operate file, malware just creates afile under its permission and reports its success to a remoteserver. In the capture screen case, malware captures the cur-rent screen, saves in its local folder, and delivers captured

screen information to a remote server. Both operations (inthe point of host view) are very similar to resource exposurepatterns of normal applications - creates a file and saves itin its local folder. However, we believe that these operationswill be detected by the network information trading analy-sis module, because they distribute more information to theoutside.

4.3.4 Detection Results of the Network InformationModel

After notifying an automatic connection, the network in-formation trading analysis module captures network trafficbetween the process (not a host) that issued an automaticconnection and some remote server(s). If the process sendsmore packet/bytes than receives packets/bytes, our moduleconsiders it anomalous.

As listed in Table 6 (marked with “N”), the network trad-ing information analysis module detects most malicious op-erations. It misses 8 malicious operations related to down-load updates and file modification or operation. In the caseof the download updates, the process gains more data, sothat our module can not detect an anomaly. In addition,sometimes a botmaster sends commands frequently to aninfected host, but does not require an answer. In this case,a bot also obtains more data. In terms of the aggregatedview, our module detects all massive outgoing connectiontrials, such as DDoS and flooding.

4.3.5 Correlated Detection ResultsIf any of the above modules determines its decision, the

decision result is delivered to the correlation engine. Basedon all delivered results, the correlation engine makes a finaldecision for a process.

When we test malicious operations, the correlation enginecan detect all malicious operations by bots. As we discussedbefore, even though some module misses an anomaly of anattack, other modules will complement it, thus our combinedresults can still detect all attacks and the combined resultsare shown in Table 6.

4.4 False Positive Test ResultsIn order to determine whether our modules misjudge be-

nign processes as malicious or not, we have tested 1,165benign processes in SET-2, representing 71 distinct nor-mal programs. They are general Windows applications pro-grams such as browsers (e.g., IE, Chrome, Firefox), P2Psoftware (e.g., Bittorrent, Emule, Skype), AV tools (e.g.,McAfee, MS windows defender, Sophos), IM tools (e.g.,Skype, QQ, Google Talk), office programs (e.g., Word, Pow-erpoint), multimedia programs (e.g., Gom, Kmplayer) (moreshown in Table 3).

Among all 1,165 processes (that have network communica-tions), our human-process-network correlation analysis mod-ule detects 490 processes that produce automatic networkconnections. Thus, these 490 processes are further investi-gated by other three modules. Process reputation analysismodule detects 2 processes as suspicious, system resourceexposure analysis module considers 14 processes suspicious,and network information trading analysis module detects 237processes as suspicious.

In the case of process reputation analysis module, 2 de-tected processes are browser processes and they visit someweb sites. One of the sites sells some Chinese software pro-

8

Functionality B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B11 B12 B13 B14 B15 B16 B17

Operate file P,N P,S P,S,N P,S,N P,S,N

Modify file P,S

Kill process S,N P,S P,S P,S,N

Capture Desktop P,S,N

Capture screen P,N P,S,N P,S,N P,S,N P,S

DDoS P,S,N

Flooding P,S,N

Create Shell P,S,N P,S,N

Download update P,S P,S

Steal key P,S,N S,N P,S,N

Spam Mail Send P,S,N

Network Scan P,S,N

Other Operation S,N S,N P,S P,S,N P,S

Table 6: Detection Results of All Modules (shaded cells represent functionalities provided by malware. Each“P”, “S”, and “N” denotes each process reputation analysis, system resource exposure analysis, and network informationtrading analysis module detect the functionalities, respectively.

grams and it is registered very recently (2011). In addition,several visited web sites are enlisted in blacklists and wefind some malicious keywords such as spam from web searchresults for the sites.The 14 processes detected by system resource exposure

analysis module are 5 Chrome browser processes, 3 Firefoxbrowser processes, 1 Internet Explorer browser process, 1 Sa-fari browser process, 2 Gom player (multimedia player) pro-cesses, 1 Seaport (MS search enhancement broker) process,and 1 MathType (mathematical equation editor). Most ofthe 10 browser processes are detected because they createnew processes and access system folders. Gom player pro-cesses try to access other users’ folders, Seaport process ac-cesses Windows system folders, and Mathtype process createnew processes. Thus, our system resource exposure analysismodule considers them suspicious.Network information trading analysis module detects 253

processes as suspicious, and most of the processes are browserand program update processes. They send some informationto remote hosts with gaining less (or no) information. Sincewe have not captured payload information of network traces,we could not understand why they send out more informa-tion. However, we infer that they might send information ofcurrent status of a host to remote servers or send query toremote mail servers to check new update.Even though each module misjudges some benign pro-

cesses as suspicious, it does not mean that our correlationanalyzer regards them as malicious. Since correlation ana-lyzer correlates all results from three modules, it determinesa process as malicious only if two or three modules mis-judge at the same time. In our test, we find 8 processesare misjudged by multiple modules, i.e., we have 8 falsepositives reported. Among them, 7 processes (4 Chromebrowser, 1 Internet Explorer, 1 Gom player, and 1 Seaportprocesses) are detected by system resource exposure analy-sis and network information trading analysis module, and1 process (Chrome browser process) is detected by systemresource exposure analysis and process reputation analysismodule. Thus, the false positive rate of our system is only0.68% (8 out of 1,165). We consider it very low that only8 false positives found in about a week of testing time onall 8 real-world heavily-used machines. If we can apply awhitelisting approach, our false positives could be easily sig-nificantly reduced.

4.5 Efficiency of EFFORTTo show the efficiency of EFFORT, we measure how many

processes are deeply/heavily investigated in real world. It

can be understood by measuring how many automatic con-nections there are because our system only focuses on pro-cesses generating automatic network connections. We an-alyze the number of automatic connections and processesproducing them in SET-2.

We find 231,613 connection trials in SET-2. Amongthem, 136,265 connections are automatically generated. Therate is quite high (58.83%). However we find that most ofthem are heading to hosts in local networks. For example,spoolsv.exe and taskeng.exe are the Windows system pro-cesses handling network or printing services and they con-tact network printers or sharing folders in local networks.In addition, synergyc.exe is a process to share mouse or key-board with multiple hosts. They generate a lot of automaticconnections to poll hosts or printers in the local networks.

We consider that we could ignore these automatic connec-tions contacting local trusted networks, because it is veryunlikely that a botmaster runs a server in the local network.There are 111,699 connections to local networks, and byremoving them we finally have 24,566 connections, which is10.6% of all connections. Among 24,566 connections, 10,154connections (more than 40% of them) head to well-knownweb sites such as Google.com and Yahoo.com. The connec-tions visiting to these domains could be ignored if a whitelistapproach is employed. To be conservative, we let EFFORTkeep investigating all the automatic connections.

4.6 Performance Overhead of EFFORTWe have measured the overhead of each module to fur-

ther verify the efficiency of EFFORT. In this measurement,we first show how our module in host (i.e., system resourceexposure analysis module) affects the system and other ap-plications, and then we will show the overhead of our net-work modules and demonstrate how they are scalable in real-world deployment.

To measure the performance overhead of the module inhost, we use two metrics: memory usage and program delay.The memory usage represents how our modules consumethe resources of memory and the program delay representshow our modules make the programs slow down when ourmodules are running. To measure the program delay, we se-lect three types of test programs: Internet Explorer whichproduces network connections frequently, Calculator whichmainly uses CPU, and Notepad which produces some diskoperations. We compare the running time of these programsbetween when our modules are running and not2. In the

2When we performed this test, we run a test program 5

9

case of the Internet Explorer, we simply visit one web site(Yahoo.com) and close. We divide some numbers using Cal-culator and read/edit/save a file using Notepad.We send queries of 100 domains, which are randomly se-

lected from SET-2, to whois server, blacklist servers, andsearch engines to measure the performance of process repu-tation analysis module and measure how long they take toget responses. We run this test 10 times (i.e., in each run,we send 100 randomly selected queries) and measure theaverage time to receive all responses.Overhead of Human-Process-Network Correlation

Analysis Module: As shown in Table 7, the overhead ofthis module is 1.35% at maximum and even 0% (i.e. ourmodule does not affect other programs at all). In addition,this module only consumes 1.81 MB of memory. Thus, webelieve that the overhead of this module is nearly ignorable.

Item w/o module with module overhead (%)

Internet Explorer 177 (ms) 179.4 (ms) 1.35%Notepad 4,206 (ms) 4,218 (ms) 0.29%Calculator 26 (ms) 26 (ms) 0%

Table 7: Overhead of Human-process-network Cor-relation Analysis Module.

Overhead of System Exposure Analysis Module:We expect this module will show relatively high overhead.Since it has to monitor a lot of system calls which are fre-quently called by a process, it is very hard to reduce theoverhead of this module.When we measure the overhead, we observe that it con-

sumes 9.18 MB memory and produces overhead around 6%at maximum and 1 % at minimum, as presented in Table 8.

Item w/o module with module overhead (%)

Internet Explorer 177 (ms) 185.1 (ms) 4.51%Notepad 4,206 (ms) 4,463 (ms) 6.12%Calculator 26 (ms) 26.3 (ms) 1.15%

Table 8: Overhead of System Exposure AnalysisModule.

The overhead seems to be not so high and even very lowin some case. In addition, our module does not need to mon-itor all processes all the time. The system exposure analysismodule only investigates a process when the process issuesautomatic connections to remote sites. In our real-worldtest on dataset SET-2, the automatic connections to re-mote sites happen very rarely, around 10% of all connec-tion trials. Moreover, since a process is likely to generatemultiple automatic connections (averagely 278 automatic-connections/process in the case of SET-2), the number ofprocesses that the module needs to investigate is very small.Hence, we also consider that the overhead from this moduleis low.Overhead of Process Reputation Module: When

we measure the time to collect information for this mod-ule, we find that collecting information related to whois andblacklist takes about 1.335 seconds per sending one query,and gathering information from search engine takes 0.149second per sending one query.

times and calculate the average value

The results seem reasonable. However one concern is thatis it scalable when there are many machines in the networkcommunicating with the Internet? We argue that we do notsend a query for every DNS request, because it is very likelythat this domain (particularly when it is normal, popular, orfrequently visited) is already queried before (either by thishost or any other host in the network) thus we already havethe cached information. To understand whether it is true ornot, we investigate DNS query patterns of a large campusnetwork, which consists of around 25,000 active unique hostsduring monitoring, as shown in Figure 2. We have capturedall DNS queries produced by the campus network users for aday. The number of captured DNS queries is around 200,000and they are targeted to around 50,000 domains. We findthat most DNS queries are targeted to a relatively smallstable set of domains. Among these 50,000 domains, top10% of them cover more than 70% of all DNS queries andtop 30% cover around 80% of all DNS queries. It is inferablethat process reputation analysis module does not need tocollect information for all DNS queries and the frequency ofinformation collection might be very low. Thus, even thoughthe number of hosts in a network can increase, we may notworry too much about the increased distinct DNS queries.

Figure 2: CDF Plot of Percentage of Queries toDomains (x-axis denotes each individual domain, y-axis denotes cumulative distribution of percentageof queries to a domain)

Overhead of Other Modules: Unlike the previousmodules, the other modules of the network information trad-ing analysis module, and the correlation analyzer exists inthe other host(s) and they mainly monitor light-weight net-work traffic or receive results from other modules. Thus theydo not affect the performance of the host/network directly.

5. RELATED WORKThere have been several approaches to detect bots at the

network level. They detect bots mainly based on networktraffic pattern analysis [17] or aggregating network flows [32]or through network behavior correlation analysis [11, 10, 12].Our work is different from the above, because we design bothof new network level sensors and host level sensors.

Detecting bots at the host level is also popular due to itseffectiveness. They employ several interesting techniques todetect bots such as tainting memory and system resources[4, 28] and examining the system call sequences/graphs [18].Although they detect malware accurately, they could causehigh overhead on the host. Our work designs several new

10

host level sensors without analyzing all running processes allthe time, but only investigating the process when necessary.There are also several interesting studies related to detect

bot malware. Liu et al. [21] proposed a host-based approachof executing malware in a virtual machine to detect bot-likebehaviors or characteristics such as making automatic net-work connections. EFFORT is a host-network cooperatedapproach to protect real-world user machines. In addition,our approach and major detection features/modules are dif-ferent. Lanzi et al. [20] proposed an approach of detectingmalware in the host by investigating the access patterns ofthe process. Our work differs from it because we use differ-ent features at host level (e.g., socket creation) and detectionmodels. Moreover, our work analyzes the process only whennecessary. Zeng et al. [33] also proposed to detect bots com-bining information from both host and network levels. Thiswork uses network sensors to trigger host analysis, thus itsuffers from the same limitations of previous network-baseddetection approaches. If a bot can evade the network levelmonitoring, it evades their detection system. EXPOSUREsystem has been proposed to detect malicious DNS based onseveral features [1]. Although some domain registration fea-tures of the Process reputation analysis module are similar toEXPOSURE, they are only small parts in our engine. Cui etal. provided an approach to detect malware by monitoringautomatically generated connections [6]. Our work differsfrom it in that we consider more factors and issues (such asforeground, helper process for DNS relaying) and we do notuse whitelisting. Moreover, this is only one module in ourwhole detection framework.In [30], Trestian et al. uses the Google search engine for

network traffic measurement, and our approach of identify-ing reputation of a process also employs search engines. Ourwork differs in its main goal (for security) and detection fea-tures/models. Also, while they only use IP address for theirquery, we use the process and domain name as well.

6. LIMITATIONS AND FUTURE WORKAs we all know that no detection system is perfect. Our

EFFORT system is no exception. For instance, a bot mightdecide to (mis)use benign domains (e.g., Google, Flickr) asa side channel[25, 3] for concealed C&C communication toevade our process reputation analysis module. However, theefficiency, bandwidth, or realtimeness of such C&C is likelyrestricted compared to regular C&C, which could downgradethe utility of bots. The bot can also choose not to accessunnecessary system resources to avoid the detection of oursystem resource exposure analysis module. However, thiswill essentially render the bot less useful or no profit to thebotmaster. The bot can also increase the incoming traffic ordecrease the outgoing traffic in order to evade our networkinformation trading analysis module. However, the former(increasing incoming traffic from C&C server) could increasethe exposure (and detection probability) of C&C server toregular security monitoring tools/IDSes that mostly mon-itor inbound traffic. And the later (decreasing outboundtraffic) will likely significantly decrease the utility of botsin performing malicious activities (information theft, spam-ming or DDoS). To evade our human-process-network corre-lation analysis module, the bot program should not produceautomatic network connections itself. Instead, it may forkanother process to do so or even consider compromising abenign program (may have frequent human interactions) to

do so. However, keep in mind that we do not use whitelistingand furthermore, there will be clear coordination and com-munication between these processes and they could be cor-related together in analysis to discovery the process group.In short, we do acknowledge these limitations but we thinkthe chance that a bot evades all our modules is very slimunless they sleep or behave like normal programs (in whichcase we still achieve the goal of deterrence because they arenot usable to botmasters then). Although not perfect, webelieve EFFORT raises a higher bar, and this is a timelyeffort and a right direction in the malware battle.

Our reputation module mainly assumes that bots will useDNS to contact their master. However not all bots may useDNS, some bots use IP address directly. Our reputationmodel is easily to be extended to handle IP address as well.

In the future, we will further improve the process reputa-tion analysis module with more robust features and intelli-gent context analysis. For instance, we plan to improve thecurrent context analysis in the case of malware keywordsappearing in the search results.

7. CONCLUSIONIn this paper, we study various features at network and

host levels and choose promising features that enable to de-tect bots both effectively and efficiently, a very challeng-ing research problem in the domain. We propose a novelhost-network cooperated detection approach with correla-tive and coordinated analysis and develop a first-of-its-kindprototype system EFFORT. In our extensive evaluation onreal world data, we show that our system can detect botsaccurately with very low false positive and low overhead.

8. REFERENCES[1] Leyla Bilge, Engin Kirda, Christopher Kruegel, and

Marco Balduzzi. Exposure: Finding malicious domainsusing passive dns analysis. In Proc of NDSS, 2011.

[2] Christopher J.C. Burges. A Tutorial on SupportVector Machines for Pattern Recognition. In Journalsof the Data Mining and Knowledge Discovery, 1998.

[3] Sam Burnett, Nick Feamster, and Santosh Vempala.Chipping away at censorship firewalls withuser-generated content. In Proceedings of the 19thUSENIX conference on Security, USENIX Security’10,2010.

[4] Juan Caballero, Pongsin Poosankam, ChristianKreibich, and Dawn Song. Dispatcher: EnablingActive Botnet Infiltration using Automatic ProtocolReverse-Engineering. In Proceedings of the 16th ACMConference on Computer and CommunicationSecurity, November 2009.

[5] Corinna Cortes and V. Vapnik. Support-VectorNetworks. In Journals of the Machine Learning, 1995.

[6] Weidong Cui, Randy H. Katz, and Wai tian Tan.BINDER: An Extrusion-based Break-In Detector forPersonal Computers. In University of BerkeleyTechnical Report No. UCB/CSD-4-1352, 2004.

[7] Detours. Software packaged for detouring win32 andapplication apis. http://research.microsoft.com/en-us/projects/detours/.

[8] EasyHook. Easyhook - the reinvention of windows apihooking. http://easyhook.codeplex.com/.

11

[9] Tal Garfinkel and Mendel Rosenblum. A virtualmachine introspection based architecture for intrusiondetection. In Proceedings of the 10th Annual Networkand Distributed System Security Symposium, February2003.

[10] Guofei Gu, Roberto Perdisci, Junjie Zhang, andWenke Lee. BotMiner: Clustering Analysis of NetworkTraffic for Protocol- and Structure-IndependentBotnet Detection. In Proceedings of the 17th USENIXSecurity Symposium (Security’08), July 2008.

[11] Guofei Gu, Phillip Porras, Vinod Yegneswaran,Martin Fong, and Wenke Lee. BotHunter: DetectingMalware Infection Through IDS-Driven DialogCorrelation. In Proceedings of the 16th USENIXSecurity Symposium (Security’07), August 2007.

[12] Guofei Gu, Junjie Zhang, and Wenke Lee. BotSniffer:Detecting Botnet Command and Control Channels inNetwork Traffic. In Proceedings of the 15th AnnualNetwork and Distributed System Security Symposium(NDSS’08), February 2008.

[13] Ramakrishna Gummadi, Hari Balakrishnan, PetrosManiatis, and Sylvia Ratnasamy. Not-a-Bot (NAB):Improving Service Availability in the Face of BotnetAttacks. In Proceedings of Symposium on NetworkedSystem Design and Implementation (NSDI), April2009.

[14] Thorsten Holz, Christian Gorecki, and Felix Freiling.Detection and Mitigation of Fast-Flux ServiceNetworks. In Proceedings of NDSS Symposium, Feb.2008.

[15] Nicholas Ianelli and Aaron Hackworth. Botnets as aVehicle for Online Crime. In Proceedings of 18thAnnual FIRST Conference, June 2006.

[16] Xuxian Jiang, Xinyuan Wang, and Dongyan Xu.Stealthy malware detection through vmm-basedout-of-the-box semantic view reconstruction. InProceedings of the 14th ACM conference on Computerand communications security, October 2007.

[17] Anestis Karasaridis, Brian Rexroad, and DavidHoeflin. Wide-scale botnet detection andcharacterization. In Proceedings of the first conferenceon First Workshop on Hot Topics in UnderstandingBotnets, April 2007.

[18] Clemens Kolbitsch, Paolo Milani Comparetti,Christopher Kruegel, Engin Kirda, Xiaoyong Zhou,and Xiaofeng Wang. Effective and efficient malwaredetection at the end host. In Proceedings of 18thUSENIX Security Symposium, August 2009.

[19] Taku Kudo. Tinysvm: Support vector machines.http://chasen.org/~taku/software/TinySVM/.

[20] Andrea Lanzi, Davide Balzarotti, ChristopherKruegel, Mihai Christodorescu, and Engin Kirda.AccessMiner: using system-centric models for malwareprotection. In Proceedings of 17th ACM conference onComputer and communications security, June 2010.

[21] Lei Liu, Songqing Chen, Guanhua Yan, and ZhaoZhang. BotTracer: Execution-Based Bot-Like MalwareDetection. In Proceedings of international conferenceon Information Security (ISC), 2008.

[22] MalwareDomains. Dns-bh malware domain blacklists.http:

//www.malwaredomains.com/wordpress/?p=1411.

[23] MicroSoft MSDN. Windows hook functions.http://msdn.microsoft.com/en-us/library/

ff468842(v=VS.85).aspx.

[24] B. Scholkopf, J.C. Platt, J.Shawe-Taylor, A.J. Smola,and R.C. Williamson. Estimating the support of ahigh-dimensional distribution. In Technical report,Microsoft Research, MSR-TR-99-87, 1999.

[25] Kapil Singh, Abhinav Srivastava, Jonathon T. Giffin,and Wenke Lee. Evaluating email’s feasibility forbotnet command and control. In Proceedings ofAnnual IEEE/IFIP International Conference onDependable Systems and Networks (DSN 2008), pages376–385, 2008.

[26] SPAMHAUS. The SPAMHAUS Project.http://www.spamhaus.org/.

[27] SRI-International. An analysis of Conficker C.http://mtc.sri.com/Conficker/addendumC/.

[28] Elizabeth Stinson and John C. Mitchell.Characterizing the Remote Control Behavior of Bots.In Proceedings of Detection of Intrusions and Malware& Vulnerability Assessment (DIMVA), 2007, July2007.

[29] Elizabeth Stinson and John C. Mitchell. Towardssystematic evaluation of the evadability of bot/botnetdetection methods. In WOOT’08: Proceedings of the2nd conference on USENIX Workshop on offensivetechnologies, Berkeley, CA, USA, 2008. USENIXAssociation.

[30] I. Trestian, S. Ranjan, A. Kuzmanovic, and A. Nucci.Unconstrained Endpoint Profiling (Googling theInternet). In Proceedings of ACM SIGCOMM 2008,Mayt 200.

[31] WinPcap. The industry-standard windows packetcapture library. http://www.winpcap.org/.

[32] T.-F. Yen and M. K. Reiter. Traffic aggregation formalware detection. In Proceedings of InternationalConference on Detection of Intrusions and Malware,and Vulnerability Assessment (DIMVA2008), July2008.

[33] Yuanyuan Zeng, Xin Hu, and Kang G. Shin. Detectionof Botnets Using Combined Host- and Network-LevelInformation. In Proceedings of Annual IEEE/IFIPInternational Conference on Dependable Systems andNetworks (DSN 2010), June 2010.

APPENDIXA. APPENDIX

A.1 SVM ClassifierFor the process reputation model, we use a SVM classifier.

Here we briefly talk about the SVM classifier.To start with the simplest case, we assume that there are

two classes and they can be separated by a linear function.More formally, given training examples xi and a classifieryi, if we assume that those two classes are denoted as 1 and−1 (i.e. yi ∈ {−1, 1}), the training examples which lie onthe hyperplane satisfy the following equation.

w · x+ b = 0, where w is a normal vector and b/||w|| is aperpendicular distance from the hyperplane to the origin.

From the above equation, we can find the hyperplanes

12

which separate the data with maximal margin by minimizing||w|| under the constrains of yi(xi ·w+ b)− 1 ≥ 0. To solvethis equation, we will apply a Lagrangian formulation, andthen we will have a primal form - Lp - of the Lagrangian [2].It is described as the following equations.

Lp ≡ 1

2||w||2 −

∑αiyi(xi · w + b) +

∑αi (1)

, where αi is a Lagrangian multiplier and αi ≥ 0.Now, we have to minimize Lp with respect w and b, and it

gives us two conditions of w =∑

αiyixi and∑

αiyi = 0. Inaddition, we can substitute these conditions into Lp, sincethey are equality in the dual formulation. Thus, we canget dual form - Ld - of the Lagrangian like the followingequation.

Ld =∑

αi −1

2

∑αiαjyiyjxi · xj (2)

Finally, we can get our SVM classifier through maximiz-ing Ld. If we can not separate the data by a linear func-tion, we have to extend the original set of training examplesxi into a high dimensional feature space with the mappingfunction Φ(x). Suppose that training examples xi ∈ Rd

are mapped into the euclidean space H by a mapping func-tion Φ : Rd → H, we can find a function K such thatK(xi, xj) = Φ(xi) · Φ(xj) (a.k.a. ”kernel function”). Wecan replace the inner-product of the mapping function bythe kernel function and solve the problem with similar ap-proach of the linearly separable case.

A.2 One­Class SVM (OCSVM)The One-Class SVM (OCSVM) has been proposed to cre-

ate a model with only one side of information [24]. In thispaper, we use OCSVM to build the system resource exposuremodel. The OCSVM maps training examples into a featurespace and finds a hyperplane which can best separate train-ing examples from the origin. For instance, given trainingexamples xi, if we assume that there are two planes denotedas “+” and “-” across an origin, the OCSVM will assign allknown examples xi into an one of the planes (i.e. “+” planeor “-” plane).Similar to generic multi-class SVM classifier, the OCSVM

needs to find a hyperplane with maximal geometric marginand it is described as solving the Lagrangian equations of (1)and (2) in Appendix A.1 (more details about the OCSVMcan be found in [24]). In this model, we will find a hy-perplane to assign all benign examples (i.e. the Yi featuresof the benign processes) into the “+” plane and anomalyexamples into the “-” plane.The weights for the correlation engine can also be deter-

mined by the OCSVM. To do this, we first collect detectionresults of other modules of benign processes and then weuse these results as features for the OCSVM classifier. Sim-ilar to the above approach, we try to find (a) hyperplane(s)which map(s) all (or most) benign processes to one plane.

13