HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.
![Page 1: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/1.jpg)
HUNTING FOR METAMORPHIC ENGINES
Mark Stamp&
Wing Wong
August 5, 2006
![Page 2: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/2.jpg)
Outline
I. Metamorphic software Both good and evil uses
II. Metamorphic virus construction kitsIII. How effective are metamorphic engines?
How to compare two pieces of code? Similarity within and between virus families Similarity to non-viral code
IV. Can we detect metamorphic viruses? Commercial virus scanners Hidden Markov models (HMMs) Similarity index
V. Conclusion
![Page 3: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/3.jpg)
PART I
Metamorphic Software
![Page 4: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/4.jpg)
What is Metamorphic Software?
Software is metamorphic provided All copies do the same thing Internal structure of copies differs
Today almost all software is cloned “Good” metamorphic software…
Mitigate buffer overflow attacks “Bad” metamorphic software…
Avoid virus/worm signature detection
![Page 5: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/5.jpg)
Metamorphic Software for “Good”?
Suppose program has a buffer overflow If we clone the program
One attack breaks every copy Break once, break everywhere (BOBE)
If instead, we have metamorphic copies Each copy still has a buffer overflow One attack does not work against every copy BOBE-resistant Analogous to genetic diversity in biology
A little metamorphism does a lot of good!
![Page 6: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/6.jpg)
Metamorphic Software for Evil?
Cloned virus/worm can be detected Common signature on every copy Detect once, detect everywhere (DODE)
If instead virus/worm is metamorphic Each copy has different signature Same detection does not work against every copy Provides DODE-resistance Analogous to genetic diversity in biology
But, effective metamorphism here is tricky!
![Page 7: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/7.jpg)
Virus Evolution
Viruses first appeared in the 1980s Fred Cohen
Viruses must avoid signature detection Virus can alter its “appearance”
Techniques employed encryption polymorphic metamorphic
![Page 8: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/8.jpg)
Virus Evolution - Encryption
Virus consists of decrypting module (decryptor) encrypted virus body
Different encryption key different virus body signature
Weakness decryptor can be detected
![Page 9: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/9.jpg)
Virus Evolution – Polymorphism
Try to hide signature of decryptor Can use code emulator to decrypt
putative virus dynamically Decrypted virus body is constant
Signature detection is possible
![Page 10: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/10.jpg)
Virus Evolution – Metamorphism
Change virus body Mutation
techniques: permutation of
subroutines insertion of
garbage/jump instructions
substitution of instructions
![Page 11: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/11.jpg)
PART II
Virus Construction Kits
![Page 12: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/12.jpg)
Virus Construction Kits – PS-MPC
According to Peter Szor:“… PS-MPC [Phalcon/Skism Mass-Produced Code generator] uses a generator that effectively works as a code-morphing engine…… the viruses that PS-MPC generates are not [only] polymorphic, but their decryption routines and structures change in variants…”
![Page 13: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/13.jpg)
Virus Construction Kits – G2
From the documentation of G2 (Second Generation virus generator):
“… different viruses may be generated from identical configuration files…”
![Page 14: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/14.jpg)
Virus Construction Kits - NGVCK
From the documentation for NGVCK (Next Generation Virus Creation Kit):
“… all created viruses are completely different in structure and opcode…… impossible to catch all variants with one or more scanstrings.…… nearly 100% variability of the entire code”
Oh, really?
![Page 15: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/15.jpg)
PART III
How Effective Are Metamorphic Engines?
![Page 16: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/16.jpg)
How We Compare Two Pieces of Code
Opcode sequences Score
0 call1 pop2 mov3 sub
… m-1 m-1
… score = n-1 jmp average
% match
0 push 0 n-1 0 n-11 mov2 sub3 and
…
…m-1 retn
Program X
Graph of real matches
Program Y Program Y
(lines with length > 5)(matching 3 opcodes)Assembly programs
Program X
Graph of matches
Program X
Program Y
![Page 17: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/17.jpg)
Virus Families – Test Data
Four generators, 45 viruses 20 viruses by NGVCK 10 viruses by G2 10 viruses by VCL32 5 viruses by MPCGEN
20 normal utility programs from the Cygwin DLL
![Page 18: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/18.jpg)
Similarity within Virus Families – Results
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 50 100 150 200
Comparison number
Similarity score
NGVCK viruses
Normal files
![Page 19: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/19.jpg)
Similarity within Virus Families – Results
Size of bubble = average similarity
NGVCK
G2VCL32 MPCGENNormal
0
0.2
0.4
0.6
0.8
1
1.2
-0.2 0 0.2 0.4 0.6 0.8
Minmum similarity score
Maximum similarity score
NGVCK
G2
VCL32
MPCGEN
Normal
![Page 20: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/20.jpg)
Similarity within Virus Families – Results
IDA_ NGVCK0-
IDA_ NGVCK8 (11.9%)
IDA_G4- IDA_G7 (75.2%)
![Page 21: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/21.jpg)
Similarity within Virus Families – Results
IDA_VCL0-
IDA_VCL9
(60.2%)
IDA_MPC1-
IDA_MPC3
(58.0%)
![Page 22: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/22.jpg)
NGVCK Similarity to Virus Families
NGVCK versus other viruses 0% similar to G2 and MPCGEN viruses 0 – 5.5% similar to VCL32 viruses (43
out of 100 comparisons have score > 0) 0 – 1.2% similar to normal files (only 8
out of 400 comparisons have score > 0)
![Page 23: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/23.jpg)
NGVCK Metamorphism/Similarity
NGVCK By far the highest degree of
metamorphism of any kit tested Virtually no similarity to other viruses
or normal programs Undetectable???
![Page 24: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/24.jpg)
PART IV
Can Metamorphic Viruses Be Detected?
![Page 25: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/25.jpg)
Commercial Virus Scanners
Tested three virus scanners eTrust version 7.0.405 avast! antivirus version 4.7 AVG Anti-Virus version 7.1
Each scanned 37 files 10 NGVCK viruses 10 G2 viruses 10 VCL32 viruses 7 MPCGEN viruses
![Page 26: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/26.jpg)
Commercial Virus Scanners
ResultseTrust and avast! detected 17
(G2 and MPCGEN)AVG detected 27 viruses (G2,
MPCGEN and VCL32)none of NGVCK viruses detected
![Page 27: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/27.jpg)
Detection with Hidden Markov Models
Use hidden Markov models (HMMs) to represent statistical properties of a set of metamorphic virus variants Train the model on family of
metamorphic viruses Use trained model to determine
whether a given program is similar to the viruses the HMM represents
![Page 28: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/28.jpg)
Detection with HMMs – Theory
A trained HMM maximizes the probabilities of
observing the training sequence assigns high probabilities to sequences
similar to the training sequence represents the “average” behavior if
trained on multiple sequences represents an entire virus family, as
opposed to individual viruses
![Page 29: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/29.jpg)
Detection with HMMs – Data
Data set 200 NGVCK viruses
Comparison set 40 normal exes from the Cygwin DLL 25 other “non-family” viruses (G2,
MPCGEN and VCL32) Many HMM models generated and
tested
![Page 30: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/30.jpg)
Detection with HMMs – Results
Test set 0, N = 2
-160
-140
-120
-100
-80
-60
-40
-20
0
0 10 20 30 40
File number
Score (LLPO)
family viruses
normal files
![Page 31: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/31.jpg)
Detection with HMMs – Results Detect some other viruses “for free”
Test set 0, N = 3
-180
-160
-140
-120
-100
-80
-60
-40
-20
0
0 10 20 30 40
File number
Score (LLPO)
familyviruses
non-familyviruses
normalfiles
![Page 32: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/32.jpg)
Detection with HMMs
Summary of experimental results All normal programs distinguished VCL32 viruses had scores close to
NGVCK family viruses With proper threshold, 17 HMM models
had 100% detection rate and 10 models had 0% false positive rate
No significant difference in performance between HMMs with 3 or more hidden states
![Page 33: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/33.jpg)
Detection with HMMs – Trained Models
Converged probabilities in HMM matrices may give insight into the features of the viruses it represents
We observe opcodes grouped into “hidden” states most opcodes in one state only
What does this mean? We are not sure…
![Page 34: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/34.jpg)
Detection via Similarity Index
Straightforward similarity index can be used as detector To determine whether a program belongs
to the NGVCK virus family, compare it to any randomly chosen NGVCK virus
NGVCK similarity to non-NGVCK code is small
Can use this fact to detect metamorphic NGVCK variants
![Page 35: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/35.jpg)
Detection with Similarity Index
Experiment compare 105 programs to one
selected NGVCK virus Results
100% detection, 0% false positive Does not depend on specific
NGVCK virus selected
![Page 36: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/36.jpg)
PART V
Conclusion
![Page 37: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/37.jpg)
Conclusion
Metamorphic generators vary a lot NGVCK has highest metamorphism
(10% similarity on average) Other generators far less effective (60%
similarity on average) Normal files 35% similar, on average
But, NGVCK viruses can be detected! NGVCK viruses too different from other
viruses and normal programs
![Page 38: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/38.jpg)
Conclusion
NGVCK viruses not detected by commercial scanners we tested
Hidden Markov model (HMM) detects NGVCK (and other) viruses with high accuracy
NGVCK viruses also detectable by similarity index
![Page 39: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/39.jpg)
Conclusion
All metamorphic viruses tested were detectable because High similarity within family and/or Too different from normal programs
Effective use of metamorphism by virus/worm requires A high degree of metamorphism and
similarity to other programs This is not trivial!
![Page 40: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/40.jpg)
The Bottom Line
Metamorphism for “good” For example, buffer overflow mitigation A little metamorphism does a lot of good
Metamorphism for “evil” For example, try to evade virus/worm
signature detection Requires high degree of metamorphism
and similarity to normal programs Not impossible, but not easy…
![Page 41: HUNTING FOR METAMORPHIC ENGINES Mark Stamp & Wing Wong August 5, 2006.](https://reader031.fdocuments.net/reader031/viewer/2022032522/56649d625503460f94a44827/html5/thumbnails/41.jpg)
References
X. Gao, “Metamorphic Software for Buffer Overflow Mitigation”, masters thesis, Department of Computer Science, San Jose State University, 2005
P. Szor, The Art of Computer Virus Research and Defense, Addison-Wesley, 2005
M. Stamp, Information Security: Principles and Practice, Wiley Interscience, 2005
W. Wong, “Analysis and Detection of Metamorphic Computer Viruses”, masters thesis, Department of Computer Science, San Jose State University, 2006