Testing Multithreaded Java Applications for Synchronization Problems, ISTA 2011

www.istabg.orgwww.vmware.com www.istabg.orgwww.vmware.com

Testing Multithreaded Java Applications for

Synchronization Problems

Vassil Popovski

www.istabg.orgwww.vmware.com

Why multithreading?

A modern CPU

…core 1

cache

core N

cache

shared memory


Why multithreading?

…Thread 1 Thread N

Java heap

A modern JavaApplication

synchronization

thread 1 thread N

Java heap


• Non-determinism

• New types of defects exist such as deadlocks, livelocks and race conditions

New concept -> new problems


Thread 2Thread 1

Thread interleaving

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2


T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2 T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2


Number of different thread interleavings


Q: What is the number of all interleavings for 3 threads with 2 blocks each?

QUIZ#1


• safety: nothing bad happens

• liveness: something good eventually happens

What to test for?


Image source: http://www.doc.ic.ac.uk/~jnm/book/ppt/ch7.ppt

Example 1: Single Lane Bridge

• safety:– no car crash

• liveness:– every car eventually get an opportunity to cross the bridge

http://www.doc.ic.ac.uk/~jnm/book/ppt/ch7.ppt


Example 2: BoundedBuffer

Producer 1

Producer 2

Producer 3

Producer N

Consumer 1

Consumer 2

Consumer M…

…

• safety:– If empty, must not allow Get(); If full, must not allow Put()

• liveness (quiz#2):– Put() on empty and Get() on full will eventually be allowed


How to test for synchronization issues?Why

• NASA’s Remote Agent

• Therac-25


How to test for synchronization issues?

• Load/Stress testing (blackbox, whitebox)• Specific interleavings testing (whitebox)• All interleavings testing (whitebox)

• Instrumentation (blackbox)


• A lot of threads and operations to exercise different interleavings

Load/Stress testing


Demo!


Deadlock example

Thread 2Thread 1

synchronized ( A ) {

synchronized ( B ) {

synchronized ( B ) {

synchronized ( A ) {

Deadlock !!!


• Tools– ExecutorService -

http://download.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html#invokeAll(java.util.Collection)

– TestNG - http://testng.org/doc/documentation-main.html#annotations

– JUnit - http://www.junit.org/apidocs/junit/extensions/ActiveTestSuite.html

• JUnitPerf - http://testng.org/doc/documentation-main.html#annotations

– Custom Threading– GroboUtils -

http://groboutils.sourceforge.net/testing-junit/using_mtt.html

Load/Stress testing



http://testng.org/doc/documentation-main.html#annotations

http://www.junit.org/apidocs/junit/extensions/ActiveTestSuite.html

http://www.junit.org/apidocs/junit/extensions/ActiveTestSuite.html

http://testng.org/doc/documentation-main.html#annotations

http://groboutils.sourceforge.net/testing-junit/using_mtt.html


• Deterministic and repeatable tests

Specific interleavings testing


• Based on internal clock

– Tick increases when all threads are blocked

– waitForTick(tick) – blocks the thread until tick reaches certain value

– assertTick(tick) – asserts the current tick

MultithreadedTC


MultithreadedTC

Thread 2Thread 1

put 42

get 42

put 17(blocks)

BoundedBuffer with size = 1

get 17


MultithreadedTC

Thread 2Thread 1

put 42

get 42

BoundedBuffer with size = 1

get 17

Tick 0

put 17(blocks)

Tick 1

waitForTick(1)

assertForTick(1)


Demo!


• Events based

– IMUnit.fireEvent(“event1”)

– @Schedule(“event1 -> event2”)

– @Schedule(“[event1] -> event2”)

IMUnit


Demo!


• Two threads only – Main and Secondary

• By default: For each line of Main thread (T1) -> block and execute fully Secondary thread (T2)

ThreadWeaver

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2

T1: Block A

T1: Block B

T2: Block 1

T2: Block 2


• Two threads only – Main and Secondary

• By default: For each line of Main thread (T1) -> block and execute fully Secondary thread (T2)

• Powerful Breakpoints

ThreadWeaver


Demo!


• Tools– MultithreadedTC -

http://code.google.com/p/multithreadedtc/• Enhanced version for Junit 4 -

http://code.google.com/p/multithreadedtc-junit4/

– IMUnit - http://mir.cs.illinois.edu/imunit/– ThreadWeaver -

http://code.google.com/p/thread-weaver/– Awaitility - http://code.google.com/p/awaitility/

Specific interleavings testing

http://code.google.com/p/multithreadedtc/

http://code.google.com/p/multithreadedtc-junit4/

http://mir.cs.illinois.edu/imunit/

http://code.google.com/p/thread-weaver/

http://code.google.com/p/awaitility/


• Exercise all possible interleavings

All interleavings testing


JavaPathFinder

Traditional testing

OK Code

Testing

error Image source: http://javapathfinder.sourceforge.net/events/JPF-workshop-050108/tutorial.ppt


JavaPathFinder

Model Checking with JavaPathFinder

Image source: http://javapathfinder.sourceforge.net/events/JPF-workshop-050108/tutorial.ppt

OKCode

properties

Model Checking

error trace

Line 5: …Line 12: ……Line 41:…Line 47:…


Program testing can be used to show the presence of bugs, but never to show their absence!

--Edsger Dijkstra

JavaPathFinder


Demo!


• Tools– JavaPathFinder (JPF) -

http://babelfish.arc.nasa.gov/trac/jpf

• JavaRaceFinder - http://www.cise.ufl.edu/research/JavaRacefinder/Java_RaceFinder/JRF_Home.html

All interleavings testing


• Instrument the code to catch problems easier

Instrumentation


AspectJ

Original code.class or .jar

AspectJ Compiler

Instrumented .class or .jar

Aspect Definition.aj


Demo!


• Tools– AspectJ - www.eclipse.org/aspectj/

• RacerAJ - http://www.bodden.de/tools/raceraj/

– CalFuzzer - http://srl.cs.berkeley.edu/~ksen/calfuzzer/– (commercial) Flashlight / Jsure -

http://www.surelogic.com/concurrency-tools.html

Instrumentation


• CHESS (native DLLs & managed executables) - http://research.microsoft.com/en-us/projects/chess/

• MoonWalker (.NET)- http://code.google.com/p/moonwalker/

Other tools


Recommended books


Thank you!


Q & A


• org.ista2011.multithreaded.common– Several implementations of AccountManager, Counter, BounderBuffer and

MultipleReadersSingleWriter

• org.ista2011.multithreaded.executorservice– Load/stress tests using ExecutorService

• org.ista2011.multithreaded.testng– Load/stress tests using TestNG

• org.ista2011.multithreaded.mtc– MultithreadedTC tests

• org.ista2011.multithreaded.imunit– IMUnit tests

• org.ista2011.multithreaded.threadweaver– ThreadWeaver tests

Source code packages (1)


• org.ista2011.multithreaded.javapathfinder– JavaPathFinder examples

• ca.mcgill.sable.racer – RacerAJ source code

• org.ista2011.multithreaded.aspectj– Custom aspect that increase the chance of hitting multithreading problem

during testing

• org.ista2011.multithreaded.blockbox.*– Rest based AccountManager client/test and server (in .server package)

Source code packages (2)

www.istabg.org

Testing Multithreaded Java Applications for Synchronization Problems, ISTA 2011

Software

Transcript of Testing Multithreaded Java Applications for Synchronization Problems, ISTA 2011