StealthTest: Low Overhead Online Software Testing Using Transactional Memory Jayaram Bobba, Weiwei...

StealthTest: Low Overhead Online Software Testing

Using Transactional Memory

Jayaram Bobba, Weiwei Xiong*, Luke Yen†,Mark D. Hill, and David A. Wood

Multifacet Project (www.cs.wisc.edu/multifacet)Dept. of Computer Sciences

University of Wisconsin-Madison

*Dept. of Computer SciencesUniversity of Illinois at Urbana-

Champaign

† Currently at Advanced Micro Devices

http://www.cs.wisc.edu/multifacet

© 2009 Multifacet Project University of Wisconsin-Madison

2

Executive Summary (1/2)The Problem

• Software testing hard– Multithreading makes harder

• Online software testing can help– Run tests on deployed softwareE.g., Delta Execution for patch testing[Tucek et al., ASPLOS 2009]– Non-intrusive mechanisms• fork (existing)

Func

tiona

llyH

idde

n

LowOverhead

GoodScalingfork


3

Executive Summary (2/2)StealthTest

• Leverage Transactional Memory for online testing• Non-Intrusive?– transaction { test(); abort}– Fast TM mechanisms

• Demonstrate two uses• Delta Execution• In vivo Testing

Func

tiona

llyH

idde

n

LowOverhead

GoodScaling

StealthTest


4

Outline

• Online Software Testing– E.g., Patch Validation

• StealthTest: TM for online testing• Delta Execution using StealthTest• In vivo Testing using StealthTest


5

Online Patch Validation

• Bug fixes can introduce more bugs– Patches must be validated

• Online Validation [Nagaraja et al., OSDI 2004]– Increased resource usage– Lockstep execution

Testing

ProductionInput

Output

Diff


6

Delta Execution[Tucek et al., ASPLOS 2009]

• Online Patch Validation Most patches are small Patched and Un-patched executions similar

• Delta Execution– Run together except when they differ

Prior Work Delta ExecutionIncreased Resource Usage O PLockstep Execution O P


7

Delta Execution using forkPr

oduc

tion

Testi

ng

fork

Insta

ll D

dat

a

Unpat

ched

exec

ution

Patc

hed

exec

ution

Com

pute

D

dat

aIso

late

D d

ata

Mer

ged

exec

ution

Time


8

Multi-threading and forkPr

oduc

tion

Testi

ng

fork

Insta

ll D

dat

a

Unpat

ched

exec

ution

Patc

hed

exec

ution

Com

pute

D

dat

aIso

late

D d

ata

Mer

ged

exec

ution

Time Stop all threads to get a consistent memory snapshot

‘Park’ all other threads


9

fork

Poor Performance ~9.8ms for split/~106ms for merge [Tucek et al, ASPLOS 2009]

Poor ScalabilityWeb-server response rate reduced by 43%

Want an alternate mechanism


10

Outline




11

Delta Execution using StealthTest

Isolatepatched execution

Introspectpatched execution

Monitordelta data access

Delta Execution

TransactionalM

emory

transaction{…} Version ManagementTracks new/old values

Conflict DetectionMonitor accesses

StealthTest

fork

Execute on child process Page diffing mprotect


12

StealthTest Interface

Isolatepatched execution

Introspectpatched execution

Monitordelta data access

Delta Execution

transaction{…} Version ManagementTracks new/old values

Conflict DetectionMonitor accesses

StealthTest

ST_begin_transactionST_abort_transaction

ST_get_oldST_get_new

ST_protect_setST_protect_clear

TransactionalM

emory


13

Requirements from TM

• Strong Atomicity [Martin et al., CAL 2006]Transactions isolated from non-transactions=> Test transactions isolated from application code

• Flexible Conflict ResolutionCan abort transactions if necessary=> Abort tests if they block application

• Communication from within transactions=> Expose result of a test


14

Outline




15

Delta Execution using StealthTest

Prod

uctio

nTe

sting

fork

Install D data

Unpatchedexecution

Patchedexecution

Compute and Isolate D data

Mergedexecution

fork

Stea

lthTe

st

Prod

uctio

n

Insta

ll D da

ta

Unpatc

hed

exec

ution

Patch

edex

ecution

Compu

te an

d

Isolat

e D da

ta

Mer

ged

exec

ution

ST_a

bort_

tran…

ST_b

egin_

tran…

ST_p

rote

ct_se

t

transactionST

_get

_new

ST_g

et_o

ld

Intro

spec

t and

rollb

ack


16

Multi-threaded Delta ExecutionPr

oduc

tion

Testi

ng

fork

Install D data

Unpatchedexecution

Patchedexecution

Compute and Isolate D data

Mergedexecution

fork

Stea

lthTe

st

Orig

inal Ins

tall D

data

Unpatc

hed

exec

ution

Patch

edex

ecution

Compu

te an

d

Isolat

e D da

ta

Mer

ged

exec

ution

ST_a

bort_

tran…

ST_b

egin_

tran…

ST_p

rote

ct_se

t

transactionST

_get

_new

ST_g

et_o

ld

Intro

spec

t and

rollb

ack


17

Evaluation

(1) Effective? (2) Non-intrusive?• Workloads– Collection of multi-threaded server apps– Same as Tucek et al., ASPLOS 2009

• Pin-based TM Emulation• 2-way SMP with 2.4GHz Pentium 4 CPUs and 2.5GB RAM


18

(1) Effective?Program Description Patch

Description

Patch Verified?

fork StealthTest

Crafty Chess App Code refactoring P PRaytrace Raytracer Result reporting fix P PTar Archive Util Incremental archiving fix P PApache1 Web Server Buffer overflow fix P PApache2 Web Server Buffer overflow fix P PDNSCache DNS Cache Behavior Change P PMySQL5.0 DB Server Extra permission checks P OOpenSSL Security Lib Added bug in TLS handling P OSquid Web Cache Buffer overflow fix P OATPhttpd Web Server Buffer overflow fix P O

Memory allocation

Works

sockets


19

(2) Non-intrusive?Program Description

fork

ForkOverhead(%) PatchDuration(%)

Crafty Chess App 0.1 <0.1

Raytrace Raytracer 0.2 0.5

Tar Archive Util 41 7.3

Apache1 Web Server 2.8 0.1

Apache2 Web Server 12 0.1

DNSCache DNS Cache 65 0.1

MySQL5.0 DB Server 4.7 5.0

OpenSSL Security Lib 12 <0.1

Squid Web Cache 2.9 0.2

ATPhttpd Web Server 65 0.8


20

Outline




21

In vivo Testing[Murphy et al. TR 2007, Chu et al. ICST 2008]

• Run unit tests on deployed software+ More testing+ More realistic Catch bugs early


22

In vivo Testingusing StealthTest

ST_begin_transaction();try {

test(); ST_begin_escape(); fprintf(log, “…”, success); ST_end_escape();} catch/except() { ST_begin_escape(); fprintf(log, “…”, fail); ST_end_escape();}ST_abort_transaction(NO_RETRY);


23

Evaluation

• Workloads– Bugbench

• Server Workloads

– STAMP• Transactional Memory benchmarks

• Implementation– Intel STM

• Language-Based TM

– TL2 STM• Library-Based TM

• Quad-core workstation with RHEL5


24

(1) Effective?

Program Description Size (LOC)

Bug Type ErrorDetected?

NCOM file compress 1.9K Stack Smash Yes

POLY file “unixier” 0.7K Stack Smash Yes

GZIP file compress 8.2K Buffer Overflow Yes

MAN documentation 4.7K Buffer Overflow Yes

BC calculator 17.0K Buffer Overflow Yes

HTPD1 web server 224K Atomicity Yes

SQUD proxy cache 93.5K Buffer Overflow Possible

CVS version control 114.5K Double Free Possible

MSQL2 DBMS 514K Atomicity Possible

MSQL3 DBMS 1028K Atomicity Possible Uns

uppo

rted

Li

brar

y C

alls

Wor

ks

• Built on Intel STM.• Run tests on Bugbench applications


25

(2) Non-intrusive?

Genome Intruder Vacation Yada0

0.5

1

1.5

2

2.5

3

3.5

4

NoneforkStealthTest

Nor

mal

ized

Exe

cutio

n Ti

me

• Built on TL2 STM.• Run tests on STAMP applications (1000 tests per min)


26

Conclusions and Future Work

• StealthTest can provide non-intrusive online testing• Synergistic benefit for TM and testing

Future• More testing frameworks– Band-aid patch testing [Sidiroglou et al., 2007]

• Multi-threaded tests


27

Executive Summary

• Software testing hard• Online software testing can help– Existing mechanisms inadequate

• StealthTest leverages TM for non-intrusive online testing• Demonstrate two uses– Delta Execution – In vivo Testing

Func

tiona

llyH

idde

n

LowOverhead

GoodScaling

StealthTest


28


29


30

Atomicity Violation Bugs?


31

Degree-2 Transactions

• Isolate only writes.

• Implementation– Reads in escape action– Early Release– Add new type of transaction to TM


32

StealthTest Wish List

• Hardware Support• System Calls within Transactions• Interaction between Locks and Transactions

StealthTest: Low Overhead Online Software Testing Using Transactional Memory Jayaram Bobba, Weiwei...

Documents

Transcript of StealthTest: Low Overhead Online Software Testing Using Transactional Memory Jayaram Bobba, Weiwei...