Entropy and Malware Detection ITEC808 – Final Project Presentation
description
Transcript of Entropy and Malware Detection ITEC808 – Final Project Presentation
Entropy and Malware Detection
ITEC808 – Final Project Presentation
Vithushan SivalingamStudent No: 42413753
Supervisors: Prof. Vijay Varadharanjan & Dr Udaya Tupakula
11th November 2011
1/30
ContentsIntroductionProject AimsShannon’s Entropy ReviewMalware Entropy techniques with malware Analysis of the schemesDiscussionConclusion Future Works
2/30
Introduction
Entropy quantifies the uncertainty involved in predicting the value of a random variable.
The outcome of a fair coin flip (two equally likely outcomes)provides less information (lower entropy) than specifying the outcome from a roll of a dice (six equally likely outcomes).
In real world, most collections of data give the false information somewhere in between.
False Positive - Couldn’t be Identify the software to be malicious, but missed it and it is a malicious.
False Negative - Identify the software be malicious, but it doesn’t turn out to be.
3/30
Malware detection plays a significant role in protecting against attacks launched on a communication world.
Still malware detection tools cannot fully prevent against encrypted and packed malwares.
Explore improvement of malware detection through entropy techniques.
4/30
Project Aims
The main goal of this project was to investigate the development of suitable entropy techniques to detect malware.
ITEC808 Literature View Component are: Reviewing the Shannon’s entropy method. Identifying of malware attributes and functionality. Detailed understanding of entropy techniques and
malware detection. Study of entropy based malware detection schemes. Analysing and reasoning about the efficiency of the
proposed schemes.
5/30
Problems and SignificanceUnderstanding the entropy theorem.
Malware Growth & Identifying attributes and functionality.
Understanding on statistical variation in malware executables.
6/30
Investigate the development of suitable entropy techniques to detect malware.
Which could be helpful for security analysts to identify more efficiently malware samples (packed or encrypted).
7/30
Shannon’s Entropy Review
Point to Point Communication.
• Given two random variables, what can we say about one when we know the other? This is the central problem in information theory.
• Keywords : Choice, Uncertain and Entropy
8/30
The entropy of a random variable X is defined by
• X- information source
The entropy is non-negative. It is zero when the random variable is “certain” to be predicted.
9/30
𝐻 (𝑥)=−𝐾∑𝑖=1
𝑛
𝑃 𝑖 log 𝑃 𝑖 𝐻 (𝑥)=∑𝑖=1
𝑛
𝑃 𝑖 log1𝑃 𝑖
Flip Coin {0.5,0.5} ◦ H(x) = + ≈ 1 bit (Receive 1 bit of information)
Double headed {1}◦ H(x) = = 0 bit
Unfair Coin {0.75,0.25}◦ H(x) = + ≈ 0.811 bit
10/30
Fair distribution
Unfair distribution
Known distribution
11/30
H(X)
Bits
Probability
Fair distribution entropy reached the highest level (1 bit)
Known distribution, entropy getting 0 bits of information. ( P = 1 or 0)
Unfair distribution, the entropy lower than maximum. (not balanced)
Joint Entropy For two random variables X and Y , the joint entropy
is defined by H(X, Y) =
Conditional entropy Between two random variables X and Y are
dependent. The extra information X contains ones Y disclosed.
Continue with chain of entropy rules.
12/30
◦ H(X) - H(X|Y) = H(Y) - H (Y|X) ◦ H(X,Y) = H(X) +H(Y) (Independent)◦ H(X,Y) < H(X) +H(Y) (dependent)◦ H(X,Y) = H(X) + H(Y|X) = H(Y) + H (X|Y)
These entropy techniques helps to build the detection models.
13/30
Entropy
Joint Entropy
Conditional Entropy
Mutual Information(Information Gain)
MalwareMalware labelled by its attributes,
behaviours and attack patterns.
14/30
.
15/30
Reported that among 20, 000 malware samples more than 80% were packed by packers from 150 different families.
If the malware, modified in runtime encryption or compression, known as a packed malware.
This process compresses an executable file and modifies the file containing the code to decompress it at runtime
Packed executable is built with two main parts.
Initially, the original executable is compressed and kept in a packed executable as a file.
Secondly, a decompression section is added to the packed executable. (This section is used to reinstall the main executable. )
16/30
Entropy techniques with malware Entropy of packed information is higher than the
original information. Information is reduced by compression and a series of bits
becomes more unpredictable, which is equivalent to uncertainty.
◦ Packed Information Uncertainty Information Entropy
◦ Original Information. Uncertainty Information Entropy
False alarms play a big role. Possible that legitimately compressed and encrypted files
could trigger false positives.
17/30
But we can use entropy to determine whether it’s an anomaly or not.
Establish categories based on different entropies. If entropy over a threshold then we can categories to be
malicious and below that value all being not malicious.
That means, we can use the entropy as a measure to classify the software to be malware.
18/30
Not Malicious
Malicious
Analysis of the schemes In the Information-theoretic Measures for
Anomaly Detection.Objective
Provide theoretical foundation as well as useful tools that can facilitate the IDS development process and improve the effectiveness of ID technologies.
Experiments on University of New Mexico (UNM) sendmail system
call data MIT Lincoln Lab sendmail BSM data MIT Lincoln Lab tcpdump data
19/30
Approach: Entropy and conditional entropy: regularity
Determine how to build a model.
Joint (conditional) entropy: how the regularities between training and test datasets relate Determine the performance of a model on test
data.
A classification approach: Given the first k system calls, predict the k+1th
system call
20/30
Conditional Entropy of Training Data (UNM)
21/30
0
0.1
0.2
0.3
0.4
0.5
0.6
1 3 5 7 9 11 13 15 17
sliding window size
Con
ditio
nal E
ntro
py
bounce-1.int
bounce.int
queue.int
plus.int
sendmail.int
total
mean
• More information is included, the more regular the dataset.
Misclassification Rate: Training Data
22/30
• Misclassification means that the classification process classifies an item to be in class A while the actual class is B.
• The misclassification rate is used to measure anomaly detection performance.
05
1015
2025
3035
4045
50
1 3 5 7 9 11 13 15 17
sliding window size
Mis
clas
sific
atio
n Ra
tebounce-1.int
bounce.int
queue.int
plus.int
sendmail.int
total
mean
Conditional Entropy vs. Misclassification Rate
23/30
0
0.2
0.4
0.6
0.8
1
1.2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
sliding window size
cond
Ent a
nd m
isC
lass
rat
e
total-CondEnttotal-MisClassmean-CondEntmean-MisClass
• The movement of misclassification rate coincides with the movement of conditional entropy.
• Estimated movement of misclassification rate, to select a sequence length for the detection model.
• E.g. Length 6 is better than 4, and 14 is better than 6.
Misclassification Rate of Testing Data and Intrusion Data
24/30
05
1015
2025
3035
4045
50
1 3 5 7 9 11 13 15 17
sliding window size
Mis
clas
sific
atio
n Ra
te
bounce-1.int
bounce.int
queue.int
plus.int
sendmail.int
total
sm-10763.int
syslog-local-1.int
fwd-loops-1.int
fwd-loops-2.int
fwd-loops-3.int
fwd-loops-4.int
fwd-loops-5.int
• Misclassification rate used as a indicator to determine whether it is an abnormal trace or normal trace .
Other Schemes Objectives “Unpacking using Entropy Analysis” analysis, how to use
entropy to quickly and efficiently identify packed or encrypted malware executable and offer results from testing methodology.◦ - bintropy technique
“Estimation for real-time encrypted traffic identification” analysis Entropy and describes a novel approach to classify network traffic into encrypted and unencrypted traffic.◦ real-time encrypted traffic detector (RTETD)◦ The classifier is able to operate in real-time as only the first packet of each
flow is processed◦ Used encrypted Skype traffic
25 /30
DiscussionThrough studying the schemes and information
theory I was able to find the follows.
Entropy can be used to measure the regularity of reviewing datasets of mixture of records.
Conditional entropy can be used to measure the regularity on sequential dependencies of reviewing datasets of structured records.
Relative entropy can be used to measure the relationship between the regularity (consistency) measures of two datasets.
Information gain used to categorise the classifying data items.
26/30
ConclusionReview and Analyse of Shannon’s entropy
study, with Examples.
Research and identification of malware (packed) functionalities with characteristics and attributes.
Analysis of entropy based schemes.
These significant findings will be following up in future work.
27/30
28/30
Involving on the Investigation of entropy analysis for selected software samples.o Use the entropy techniques to compute the
entropy scores from the selected malware executable samples.
Identify the experimental tools. o We planed to analysis the malware samples
using commercial experiments tools. E.g. PECompact Executable Compressor
Reference1. C. E. Shannon. The Mathematical Theory of Communication. Reprinted with corrections
from The Bell System Technical Journal, Vol. 27, pp. 379–423, 623–656, July, October, 1948.
2. M. Morgenstern and A. Marx. Runtime packer testing experiences. In Proceedings of the 2nd International CARO Workshop, 2008.
3. *Lee, W., Xiang, D.: Information-theoretic Measures for Anomaly Detection. In: IEEE Symp. On Security and Privacy, Oakland, CA, pp. 130-143 (2001).
4. M. Morgenstern and Hendrik Pilz, AV-Test GmbH, Magdeburg, Germany, Useful and useless statistics about viruses and anti-virus programs, Presented at CARO 2010 Helsinki.
5. *Lyda, R., Hamrock, J.: Using Entropy Analysis to Find Encrypted and Packed Malware. In: Security & Privacy, IEEE Volume 5, Issue 2, pp. 40-45, Digital Object Identifier 10.1109/MSP.2007.48 (March-April 2007).
6. Guhyeon Jeong, Euijin Choo, Joosuk Lee, Munkhbayar Bat-Erdene, and Heejo Lee Generic, Unpacking using Entropy Analysis, Div. of Computer & Communication Engineering, Korea University, Seoul, Republic of Korea, 2010.
7. *Peter Dorfinger, Georg Panholzer, and Wolfgang John: Entropy estimation for real-time encrypted traffic identification: Salzburg Research, Salzburg, Austria, 2010.
29/30
Thank you.
30