Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis...

45
Transparent Malware Debugging on x86 and ARM Zhenyu Ning and Fengwei Zhang COMPASS Lab Wayne State University April 27, 2018 1

Transcript of Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis...

Page 1: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Transparent Malware Debugging on x86 andARM

Zhenyu Ning and Fengwei Zhang

COMPASS LabWayne State University

April 27, 2018

1

Page 2: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Outline

I Introduction

I BackgroundI Towards Transparent Malware Analysis

I MalT on x86 ArchitectureI Ninja on ARM Architecture

I Conclusions

2

Page 3: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Outline

I Introduction

I BackgroundI Towards Transparent Malware Analysis

I MalT on x86 ArchitectureI Ninja on ARM Architecture

I Conclusions

3

Page 4: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Transparency

What is transparent malware analysis?

4

Page 5: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Transparency

What is transparent malware analysis?

I Analyzing the malware without being aware.

I “Transparent” means that the malware cannotdetect it.

5

Page 6: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Transparency

Why transparency is important?

6

Page 7: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Evasive Malware

Computer System

Application Malware

7

Page 8: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Evasive Malware

Computer System

Application Malware

Analyzer

8

Page 9: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Evasive Malware

Computer System

Application Application

Analyzer

9

Page 10: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Malware Analysis

What is the current state of malwareanalysis systems?

10

Page 11: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Malware Analysis

Application

Operating System

Hypervisor/Emulator

App App Malware

11

Page 12: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Malware Analysis

Application

Operating System

Hypervisor/Emulator

App App Malware

Malware Analyzer

12

Page 13: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Malware Analysis

Application

Operating System

Hypervisor/Emulator

App App Malware

Malware Analyzer

I Unarmed to anti-virtualization or anti-emulation techniques.

I Large performance overhead.

13

Page 14: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Malware Analysis

Application

Operating System

Hypervisor/Emulator

App App Malware

Malware Analyzer

14

Page 15: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Malware Analysis

Application

Operating System

Hypervisor/Emulator

App App Malware

Malware Analyzer

I Unable to handle malware with high privilege (e.g. rootkits).

15

Page 16: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Transparency Requirements

What makes a transparent malwareanalysis system?

16

Page 17: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Transparency Requirements

I An Environment that provides the access to the states of thetarget malware.

I An Analyzer which is responsible for the further analysis ofthe states.

17

Page 18: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Transparency Requirements

I An Environment that provides the access to the states of thetarget malware.

I It is isolated from the target malware.

I It exists on an off-the-shelf (OTS) bare-metal platform.

I An Analyzer which is responsible for the further analysis ofthe states.

18

Page 19: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Transparency Requirements

I An Environment that provides the access to the states of thetarget malware.

I It is isolated from the target malware.

I It exists on an off-the-shelf (OTS) bare-metal platform.

I An Analyzer which is responsible for the further analysis ofthe states.

I It should not leave any detectable footprints to the outside ofthe environment.

19

Page 20: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Outline

I Introduction

I BackgroundI Towards Transparent Malware Analysis

I MalT on x86 ArchitectureI Ninja on ARM Architecture

I Conclusions

20

Page 21: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

System Management Mode

System Management Mode (SMM) [1] is special CPU modeexisting in x86 architecture, and it can be used as a hardwareisolated execution environment.

I Originally designed for implementing system functions (e.g.,power management)

I Isolated System Management RAM (SMRAM) that isinaccessible from OS

I Only way to enter SMM is to trigger a System ManagementInterrupt (SMI)

I Executing RSM instruction to resume OS (Protected Mode)

21

Page 22: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

TrustZone Technology

ARM TrustZone technology [2] divides the execution environmentinto a secure domain and a non-secure domain.

I The RAM is partitioned to secure and non-secure regions.

I The interrupts are assigned into the secure or non-securegroup.

I Secure-sensitive registers can only be accessed in securedomain.

I Hardware peripherals can be configured as secure access only.

22

Page 23: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

PMU and ETM

I The Performance Monitor Unit (PMU) [3, 4] leverages a setof performance counter registers to count the occurrence ofdifferent CPU events.

I The Embedded Trace Macrocell (ETM) [5] traces theinstructions and data of the system, and output the tracestream into pre-allocated buffers on the chip.

I The PMU exists in both x86 and ARM architecture while theETM is ARM special hardware.

23

Page 24: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Outline

I Introduction

I BackgroundI Towards Transparent Malware Analysis

I MalT on x86 Architecture [S&P’15]I Ninja on ARM Architecture [USENIX Security’17]

I Conclusions

24

Page 25: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Towards Transparent Malware Analysis

Application

Operating System

Hypervisor/Emulator

Hardware

App App Malware

MalT on x86 Ninja on ARM

25

Page 26: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

MalT on x86 Architecture

Debugging Client

GDB-likeDebugger

Debugging Server

SMIhandler

Debuggedapplication

1) Trigger SMI

2) Debug command

3) Response message

InspectapplicationBreakpoint

26

Page 27: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

MalT — Performance

I Testbed SpecificationI Motherboard: ASUS M2V-MX SE

I CPU: 2.2GHz AMD LE-1250

I Chipset: AMD k8 Northbridge + VIA VT 8237r Southbridge

I BIOS: Coreboot + SeaBIOS

27

Page 28: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

MalT — Performance

Table: SMM Switching and Resume (Time: µs)

Operations Mean STD 95% CI

SMM switching 3.29 0.08 [3.27, 3.32]

Command and BP checking 2.19 0.09 [2.15, 2.22]

Next SMI configuration 1.66 0.06 [1.64, 1.69]

SMM resume 4.58 0.10 [4.55, 4.61]

Total 11.72

28

Page 29: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

MalT — Limitation

I High performance overhead on mode switch.

I Unprotected modified registers.

I Vulnerable to external timing attack.

29

Page 30: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Ninja on ARM Architecture

Non-secure Domain

Rich OS

App

App

Malware

Secure Domain

Secure Interrupt Handler

TraceSubsystem

DebugSubsystem

RemoteDebugging

Client

Secure Interrupt

Secure PortERET

30

Page 31: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Ninja on ARM Architecture

I Use TrustZone as the isolated execution environment.

I The debug subsystem is similar to MalT while the tracesubsystem is immune to timing attacks.

I Modified registers are protected via hardware traps.

31

Page 32: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Ninja — Performance

I Testbed SpecificationI ARM Juno v1 development board

I A dual-core 800 MHZ Cortex-A57 cluster and a quad-core 700MHZ Cortex-A53 cluster

I ARM Trusted Firmware (ATF) [6] v1.1 and Android 5.1.1

32

Page 33: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Ninja — Performance

Table: Performance Scores Evaluated by CF-Bench [7]

Native Scores Java Scores Overall Scores

Mean Slowdown Mean Slowdown Mean Slowdown

Tracing Disabled 25380 18758 21407

Instruction Tracing 25364 1x 18673 1x 21349 1x

System call Tracing 25360 1x 18664 1x 21342 1x

Instruction Tracing 6452 4x 122 154x 2654 8x

33

Page 34: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Ninja — Performance

Table: Time consumption of domain switching (Time: µs)

ATF Enabled Ninja Enabled Mean STD 95% CI

× × 0.007 0.000 [0.007, 0.007]

X × 0.202 0.013 [0.197, 0.207]

X X 0.342 0.021 [0.334, 0.349]

34

Page 35: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Ninja — Limitation

I OS-related tracing requires software-based approach to fillsemantic gaps, which involves performance overhead.

I Malware may intentionally enable the ETM or PMU to detectthe analysis system.

I Hardware traps can only protect the system instruction accessto the registers.

35

Page 36: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Outline

I Introduction

I BackgroundI Towards Transparent Malware Analysis

I MalT on x86 ArchitectureI Ninja on ARM Architecture

I Conclusions

36

Page 37: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Conclusions

I We present MalT and Ninja, malware analysis systems in x86and ARM architectures aiming for higher transparency.

I We consider the hardware-based approach provides bettertransparency than software-based approaches.

I To build a fully transparent malware analysis system, we areseeking for more hardware support.

37

Page 38: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Related Papers

USENIX Security’17 Zhenyu Ning and Fengwei Zhang. Ninja: Towards Transparent Tracing and Debugging onARM. In Proceedings of The 26th USENIX Security Symposium, Vancouver, BC, Canada,August 2017.

S&P’15 Fengwei Zhang, Kevin Leach, Angelos Stavrou, and Haining Wang. Using Hardware Featuresfor Increased Debugging Transparency. In Proceedings of The 36th IEEE Symposium onSecurity and Privacy, San Jose, CA, May 2015.

38

Page 39: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

References I[1] Intel, “64 and IA-32 architectures software developer’s manual: Volume 3C,”

https://software.intel.com/sites/default/files/managed/a4/60/325384-sdm-vol-3abcd.pdf.

[2] ARM Ltd., “TrustZone Security Whitepaper,”http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.prd29-genc-009492c/index.html.

[3] ——, “ARMv8-A Reference Manual,”http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.k/index.html.

[4] Intel, “64 and IA-32 architectures software developer’s manual: Volume 3B,”https://software.intel.com/sites/default/files/managed/a4/60/325384-sdm-vol-3abcd.pdf.

[5] ARM Ltd., “Embedded Trace Macrocell Architecture Specification,”http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ihi0014q/index.html.

[6] ——, “ARM Trusted Firmware,” https://github.com/ARM-software/arm-trusted-firmware.

[7] Chainfire, “CF-Bench,” https://play.google.com/store/apps/details?id=eu.chainfire.cfbench.

39

Page 40: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Thank you!

[email protected] & [email protected]

http://compass.cs.wayne.edu

40

Page 41: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Hardware Traps

.......

MRS X0, PMCR EL0

MOV X1, #1

AND X0, X0, X1......

Non-secure Domain

41

Page 42: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Hardware Traps

.......

MRS X0, PMCR EL0

MOV X1, #1

AND X0, X0, X1......

Non-secure Domain

Analyzing the instruction

Secure Domain

MDCR EL3.TPM=1

42

Page 43: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Hardware Traps

.......

MRS X0, PMCR EL0

MOV X1, #1

AND X0, X0, X1......

Non-secure Domain

Analyzing the instruction

MOV X0, #0x41013000

Secure Domain

MDCR EL3.TPM=1

43

Page 44: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Hardware Traps

.......

MRS X0, PMCR EL0

MOV X1, #1

AND X0, X0, X1......

Non-secure Domain

Analyzing the instruction

MOV X0, #0x41013000

Modifying saved ELR EL3

Secure Domain

MDCR EL3.TPM=1

44

Page 45: Transparent Malware Debugging on x86 and ARM · I We present MalT and Ninja, malware analysis systems in x86 and ARM architectures aiming for higher transparency. I We consider the

Hardware Traps

.......

MRS X0, PMCR EL0

MOV X1, #1

AND X0, X0, X1......

Non-secure Domain

Analyzing the instruction

MOV X0, #0x41013000

Modifying saved ELR EL3

ERET

Secure Domain

MDCR EL3.TPM=1

45