ISA 673 - George Mason Department of Computer Scienceastavrou/courses/ISA_673_S12/Malware...
Transcript of ISA 673 - George Mason Department of Computer Scienceastavrou/courses/ISA_673_S12/Malware...
ISA 673 Operating Systems’ Security
Topic: Malware Reverse Engineering Unpacking Binary Packers
Arnur Tokhtabayev, George Mason University!
Malware types (source: Securelist.com)
¨ Viruses and Worms (by propagation: net, e-mail, p2p, IM, IRC, USB)
¨ Trojan programs (by payload: backdoor, rootkit, clicker, keylogger - spyware, dropper, FakeAV, ransom, DDoS, downloader)
¨ Malware tools (by functionality: constructor, packer, hacktool)
¨ Riskware (by purpose: net-tool, monitor, proxy, fraud-tool, adware)
Malware reverse engineering scheme
1. Create isolated experimental environment (host only VM)
2. Submit malware to existing sandboxes (Anubis, ThreatExpert) • Inspect its high level behavior • Watch file droppers and created processes
3. Examine its imports, exports and strings n Based on import API, guess malware type (e.g.
SetWindowsHook)
4. Identify packer and unpack (manual or auto, PEiD)
Malware RE scheme (2)
5. Disassemble/decompile malware n Trace API usage (context, constant attributes) n Deduce variables, create structures, name them properly (semantics) n Deduce function semantics (what does it do, approx.) n Trace sensitive data path
6. Debug binary n Trigger conditions n Resolve implicit jumps (control flow) n Reveal actual values variables (dynamic API attributes)
7. Patch binary (optional) n Execute malicious payload (addresses limitations of step 2)
PE packer technologies
Code
Virtualization
Anti-crack (prevent debugging/dumping, resist
unpacking tools)
PE packers (multilayer packing) UPX, ASPack, PECompact, Boxed App
Armadillo, ASProtect, Themida
VMProtect, Enigma, Themida
Packer Distribution in Malware
Source: CSIS research team, CS, GMU 2012 (analyzed 7000 malware samples)
45%
22%
9%
6%
3%
2% 1% 1%
9%
Armadillo
UPX
BobSoft
ASPack
ASProtect
PECompact
Themida/WinLicense
UPX packing scheme (basic)
Compress
Copy resources
Empty space for an unpacked image
Compressed data
Unpacking code
Resources
Headers (new) Entry point Original executable
Packed executable
UPX operation scheme (basic)
Compress
Copy resources
Empty space for an unpacked image
Compressed data
Unpacking code
Resources
Headers (new) Entry point Original executable
Packed executable
Dynamic Unpacking Scheme Tool chain
Identify packer • PEiD
Trace (debug) unpacking
routine • OllyDbg
Define Original Entry
Point
• Signatures (semantics)
Dump unpacked
image • OllyDump
Rebuild import table
(section) • ImportRec
Anti-Debugging in Packers Debugger detection
¨ System API: check for the existence of a debugger using system information (IsDebugerPresent(), CheckRemoteDebuggerPresent()).
¨ System data: check debugger presence in Process/Thread global structures (NtGlobalFlag, PEB.BeingDebugged).
¨ Other: code integrity check (INT 3h instruction detection), SEH (INT 3h exception check), hardware breakpoints, execution latency
Anti-Debugging in Packers: Example (Source: http://www.openrce.org/reference_library/anti_reversing)
Note: FS – segment register (points to various items in TIB (Tread Information Block)) Example: FS[0] -> SEH FS[30h] -> Address of PEB (Process Environment Block) structure
CALL IsDebuggerPresent ; invoke API CMP EAX,1 JE @DebuggerDetected
Ssytem API:
MOV EAX,DWORD PTR FS:[18h] ; get TID entry address MOV EAX,DWORD PTR DS:[EAX+30h] ; get PEB entry address MOVZX EAX,BYTE PTR DS:[EAX+2h] CMP EAX,1 JE @DebuggerDetected
System data:
typedef struct _PEB { BOOLEAN InheritedAddressSpace; BOOLEAN ReadImageFileExecOptions; BOOLEAN BeingDebugged; BOOLEAN Spare; …
Contact info
Dr. Arnur Tokhtabayev, Center for Secure Information Systems George Mason University Computer Science Department Research I, Rm 435
E-mail: ¨ [email protected] ¨ [email protected]