Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research.

Taming Concurrency:A Program Verification Perspective

Shaz Qadeer

Microsoft Research

Reliable concurrent software?• Correctness problem

– does program behave correctly for all inputs and all interleavings?

• Bugs due to concurrency are insidious – non-deterministic, timing dependent– data corruption, crashes– difficult to detect, reproduce, eliminate

P satisfies S

Undecidable problem!

Why is verification of concurrent programs more difficult than verification of sequential programs?

P satisfies S

Assertions: Provide contracts to decompose problem into a collection of decidable problems• pre-condition and post-condition for each procedure• loop invariant for each loop

Abstractions: Provide an abstraction of the program for which verification is decidable• Finite-state systems

• finite automata• Infinite-state systems

• pushdown automata, counter automata, timed automata, Petri nets

Interference

int t;

t := x;

t := t + 1;

x := t;

Correct

pre x = 0;

post x = 1;

Interference

int t;

t := x;

t := t + 1;

x := t;

Incorrect!

pre x = 0;

post x = 2;

int t;

t := x;

t := t + 1;

x := t;

A B

Controlling interference

int t;

acquire(l);

t := x;

t := t + 1;

x := t;

release(l);

Correct!

pre x = 0;

post x = 2;

int t;

acquire(l);

t := x;

t := t + 1;

x := t;

release(l);

A B

Interference makes program verification difficult

• Annotation explosion with the assertion approach

• State explosion with the abstraction approach

Annotation explosion

• For sequential programs– assertion for each loop – assertion refers only to variables in scope

• For concurrent programs– assertion for each control location– assertion may need to refer to private state of

other threads

State explosion

Sequential Concurrent

Finite-statesystems

Pushdownsystems

P.G

(P.G)3

P = # of program locationsG = # of global states

PSPACEcomplete

Undecidable

n = # of threads

Taming interference

• Atomicity for combating annotation explosion

• Interference-bounding for combating state explosion

Critical_Section l;/*# guarded_by l */int balance;

Bank account

/*# atomic */void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l);}

/*# atomic */int read( ) { int r; acquire(l); r = balance; release(l); return r;}

/*# atomic */void withdraw(int x) { int r = read(); acquire(l); balance = r – x; release(l);}

Atomicity violation in StringBuffer (Flanagan-Q, PLDI 03)

public final class StringBuffer { private int count; private char[ ] value; . . public synchronized StringBuffer append (StringBuffer sb) { if (sb == null) sb = NULL; int len = sb.length( ); int newcount = count + len; if (newcount > value.length) expandCapacity(newcount); sb.getChars(0, len, value, count); //use of stale len !! count = newcount; return this; }

public synchronized int length( ) { return count; }

public synchronized void getChars(. . .) { . . . }}

Inadequate atomicity is a good predictor ofundesirable interference!• compared to data races

Numerous tools for detecting atomicity violations• static (Type systems, ESPC, QED)• dynamic (Atomizer, AVIO, AtomAid, Velodrome, …)

Significant effort cutting across communities • architecture and operating systems• programming languages and compilers • testing and formal methods

Definition of atomicity

• deposit is atomic if for every non-serialized execution, there is a serialized execution with the same behavior

acq(l) r=bal bal=r+n rel(l)x y z

Serialized execution of deposit



Non-serialized executions of deposit

S0 S1 S2 S3 S4 S7S6S5


S0 T1 T2 T3 S4 S7T6S5

r=bal bal=r+n rel(l)x y zacq(l)

S0 T1 S2 T3 S4 S7S6S5

y r=bal bal=r+n rel(l)x acq(l) z

S0 T1 T2 T3 S4 S7S6S5


S0 S1 S2 T3 S4 S7S6S5

acq(l) r=bal bal=r+n rel(l)y zx

Reduction (Lipton, CACM 75)

B: both right + left commutes– variable access holding lock

A: atomic action, non-commuting – access unprotected variable

Four atomicities

L: left commutes– lock release

R: right commutes– lock acquire

S0. S5

R* A L*x Y. . .

S0. S5

R* A L*x Y. . .

Sequential composition Theorem: Sequence (R+B)*;(A+); (L+B)* is atomic

CCCCCCCCCAAACCCLLLCARARRCARLBBCARLB;

; A

R;A;L ; R;A;L

C

A

; A

R; B ; A; L

A

R


Bank account



/*# atomic */void withdraw(int x) { int r = read(); acquire(l); balance = r – x; release(l);}

RBBL

A RBLB

A

ARBL

C

Incorrect!


Bank account



/*# atomic */void withdraw(int x) { acquire(l); int r = balance; balance = r – x; release(l);}

RBBL

A RBLB

A

RBBL

A

Correct!

Taming interference

• Atomicity for combating annotation explosion

• Interference-bounding for combating state explosion

Interference bounding

• An approach to the state-explosion problem

• Explore all executions with a bounded amount of interference– increase the interference bound iteratively

• Good idea if low interference bound– can be exploited by algorithms– is enough to expose bugs

Context-bounded verification [Wu-Q, PLDI 04] [Rehof-Q, TACAS 05]

Context Context Context

Context switch Context switch

• Interference proportional to number of context switches

• Explore all executions with few context switches• Unbounded computation within each context

– Different from bounded model checking

Context-bounding today

• CHESS [Musuvathi-Q, PLDI 07]• JMoped [Suwimonteerabuth et al., SPIN

08]• SPIN for multithreaded C programs [Zaks-

Joshi, SPIN 08]• CBA [Lal-Reps, CAV 08]• Static Driver Verifier

Testing concurrent programs is HARD

• Bugs hidden in rare thread interleavings

• Today, concurrency testing = stress testing– Poor coverage of interleavings– Unpredictable coverage results in

“Heisenbugs”

• The mark of reliability of the system still remains its ability to withstand stress

CHESS in a nutshell

• Replace the OS scheduler with a demonic scheduler

• Systematically explore all scheduling choices

ConcurrentProgram

Win32 API

Kernel SchedulerDemonicScheduler

CHESS architecture

Kernel: Threads, Scheduler, Synchronization Objects

While(not done) { TestScenario()}

TestScenario() { …}

Program

CHESSCHESS runs the scenario in a loop • Each run takes different interleaving• Each run is repeatable

Win32 API

Intercept synchronization and threading calls• To control and introduce

nondeterminismDetect• Assertion violations• Deadlocks• Dataraces• Livelocks

CHESS methodology generalizes

• CHESS works for– Unmanaged programs (written in C, C++)– Managed programs (written in C#,…)– Singularity applications

• With appropriate wrappers, can work for Java, Linux applications

.NETProgram

.NET CLR

CHESSWin32

Program

Win32 / OS

CHESSSingularityProgram

Singularity

CHESS

CHESS: Systematic testing for concurrency

Kernel: Threads, Scheduler, Synchronization Objects

While(not done){ TestScenario()}

TestScenario(){ …}

Program

CHESS

Win32 API

CHESS runs the scenario in a loop • Each run is a different interleaving• Each run is repeatable

x = 1; … … … … … x = k;

State-space explosion

x = 1; … … … … …x = k;

…

n threads

k steps each

• Number of executions

= O( nnk )

• Exponential in both n and k– Typically: n < 10 k > 100

• Limits scalability to large programs

Goal: Scale CHESS to large programs (large k)

Thread 1 Thread n

x = 1;if (p != 0) { x = p->f;}

Preemption-boundingPrioritize executions with small number of preemptions

Two kinds of context switches: Preemptions – forced by the scheduler

e.g. Time-slice expiration

Non-preemptions – a thread voluntarily yieldse.g. Blocking on an unavailable lock, thread end

x = p->f;}

x = 1;if (p != 0) {

p = 0;

preemption

non-preemption

Thread 1 Thread 2

Preemption-bounding in CHESS• The scheduler has a budget of c preemptions

– Nondeterministically choose the preemption points

• Resort to non-preemptive scheduling after c preemptions

• Once all executions explored with c preemptions– Try with c+1 preemptions

x = 1; … … … …

x = 1; … … …

…x = k;

Property 1: Polynomial bound• Terminating program with fixed inputs and deterministic threads

– n threads, k steps each, c preemptions

• Number of executions <= nkCc . (n+c)!

= O( (n2k)c. n! )

Exponential in n and c, but not in k

x = 1; … … … … …x = k;

x = 1; … … … … … x = k;

… …

x = k;

• Choose c preemption points

• Permute n+c atomic blocks

Thread 1 Thread 2

Property 2: Simple error traces• Finds smallest number of preemptions to the

error

• Number of preemptions better metric of error complexity than execution length

Property 3: Coverage metric• If search terminates with preemption-

bound of c, then any remaining error must require at least c+1 preemptions

• Intuitive estimate for– The complexity of the bugs remaining in

the program– The chance of their occurrence in practice

Property 4: Many bugs with 2 preemptions

Program Lines of code

Bugs

Work Stealing Q 4K 4

CDS 6K 1

CCR 9K 3

ConcRT 16K 4

Dryad 18K 7

APE 19K 4

STM 20K 2

TPL 24K 9

PLINQ 24K 1

Singularity 175K 2

37 (total)

Acknowledgement: testers from PCP team

Coverage vs. Preemption-bound

CHESS status

• Being used by testers in many Microsoft product groups

• Demo and session at Professional Developer Conference 2008

• CHESS for Win32 – http://research.microsoft.com/projects/chess

• CHESS for .NET– Coming soon

So far …

• Interference makes program verification difficult

• Taming interference– atomicity – interference-bounding

Whither next?

• Concurrent programs as a composition of modules implementing atomic actions

• Inteference-bounding for general concurrent systems

• Symbolic context-bounded verification

Structuring concurrent programs• In addition to atomicity, we need

– behavioral abstraction analogous to pre/post-conditions for sequential code fragments

– intuitive and simple contract language

• A calculus of atomic actions (Elmas-Tasiran-Q, POPL 08)– QED verifier

Interference-bounding for general concurrent systems

• Task = unit-of-work• Shared-memory systems

– Task (usually) corresponds to a thread• Message-passing systems

– Task corresponds to a sequence of messages• Need linguistic support for the task

abstraction in software models and implementations

Context-bounded reachability is NP-complete

Unbounded Context-bounded

Finite-statesystems

Pushdownsystems

PSPACEcomplete

NP-complete

Undecidable NP-complete

P = # of program locationsG = # of global states

n = # of threadsc = # of context switches

Symbolic context-bounded verification

• The transition relations of tasks are usually encoded symbolically– reachability analysis is PSPACE-complete even for a

single task

• Scalable tools for single symbolic task– Bebop, Moped, Getafix

• Can we extend these techniques to deal with multiple tasks?– Lal and Reps (TACAS 2008, CAV 2008)– Suwimonteerabuth et al. (SPIN 2008)

Questions?

Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research.

Documents

Transcript of Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research.