Finding Code That Explodes Under Symbolic Evalua

Finding Code That Explodes Under Symbolic Evalua<on

James Bornholt Emina Torlak University of Washington

unsat.org

Automated reasoning tools help us solve hard programming problems


Does my program s8ll work a:er the file system

crashes? [ASPLOS’16]👷 Verifica8on



crashes? [ASPLOS’16]👷

How do I compile codefor this weird new

architecture? [PLDI’14] "

Verifica8on

Synthesis



crashes? [ASPLOS’16]👷



How do I teach kids the rules of algebra

effec8vely? [VMCAI’18]#

Verifica8on

Synthesis

“Programs”

Symbolic evaluatorsDoes my program s8ll

work a:er the file system crashes? [ASPLOS’16]👷







Interpreter for file system opera8ons

Interpreter for new architecture instruc8ons





Symbolic evaluator Sketch, RoseWe, …








Verifica8on SynthesisAngelic

Execu8on for free!



Symbolic evaluators: no free lunchDoes my program s8ll




Execu8on for free!




How do you make these tools scale?



Execu8on for free!




Searching all paths through the interpreter

Searching all paths through the interpreter

How do you make these tools scale?



Execu8on for free!


Symbolic profiling iden<fies performance issues in symbolic evalua<on


Symbolic profiling Data structures and analyses



Symbolic evalua8on an8-paWerns Common issues and source-level repairs0

1020

0 1500



Symbolic evalua8on an8-paWerns Common issues and source-level repairs

Empirical results 300× speedup on real-world tools

01020

0 1500



Symbolic evalua8on an8-paWerns Common issues and source-level repairs

Empirical results 300× speedup on real-world tools

01020

0 1500

Symbolic evalua8on All-paths execu8on of programs

if (…) { … }

∀x. φ(…, x)

Symbolic evalua<onAll-paths execu8on of programs

Symbolic evalua<on executes all paths through a program

#lang rosette

(define (first-k-even lst k) (define xs (filter even? lst)) (take xs k))


#lang rosette


Inputs are unknown (trying to find values

that violate spec)


(filter even? ‘(x0 x1))

#lang rosette



that violate spec)



‘() ‘(x0)

¬(even? x0) (even? x0)

#lang rosette



that violate spec)



‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)


(even? x1)¬(even? x1) (even? x1)¬(even? x1)

#lang rosette



that violate spec)



‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)



#lang rosette


‘() ‘()

k=0


that violate spec)

‘(x1) ‘() ‘(x0) ‘() ‘(x0) ‘(x0 x1)

k=0 k=1 k=0 k=1 k=0k=1

k=2



‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)



#lang rosette


‘() ‘()

k=0


that violate spec)

take runs 22 8mes

‘(x1) ‘() ‘(x0) ‘() ‘(x0) ‘(x0 x1)

k=0 k=1 k=0 k=1 k=0k=1

k=2



‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)



#lang rosette


‘() ‘()

k=0


that violate spec)

take runs 22 8mes

because filter ran on a list of size 2

‘(x1) ‘() ‘(x0) ‘() ‘(x0) ‘(x0 x1)

k=0 k=1 k=0 k=1 k=0k=1

k=2

Blaming filter even though it’s not the slowest

Symbolic profilingData structures and metrics

Two data structures to summarize symbolic evalua<on

‘()

‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)



(even? x0)

∧

¬

∧∧ ∧

¬

(even? x1)

Symbolic evalua<on graph Reflects the evaluator’s strategyfor all-paths execu8on of the program

Symbolic heap Shape of all symbolic valuescreated by the program

Any symbolic evalua<on technique can be summarized by these two data structures

The symbolic evalua4on graph summarizes branching and merging

Symbolic evalua<on graph • Nodes are program states • Edges are transi8ons

between states


‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)



‘() ‘()

k=0

‘(x1) ‘() ‘(x0) ‘() ‘(x0) ‘(x0 x1)

k=0 k=1 k=0 k=1 k=0k=1

k=2



‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)





‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)



Symbolic execu8on



‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)



Symbolic execu8on(filter even? ‘(x0 x1))

‘() ‘(x0)


Bounded model checking



‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)




‘() ‘(x0)


ys0

ys0 = (ite (even? x0) ‘() ‘(x0))




‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)




‘() ‘(x0)


(even? x1)¬(even? x1)

ys0

ys0 ys1

ys0 = (ite (even? x0) ‘() ‘(x0))ys1 = (append ys0 ‘(x1))




‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)




‘() ‘(x0)



ys0

ys0 ys1

ys2

ys0 = (ite (even? x0) ‘() ‘(x0))ys1 = (append ys0 ‘(x1))ys2 = (ite (even? x1) ys1 ys0)




‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)




‘() ‘(x0)



ys0

ys0 ys1

ys2



More states, but more concrete

Fewer states but less concrete

The symbolic heap shows how symbolic values are used

(even? x0)

∧

¬

∧∧ ∧

¬

(even? x1)

Symbolic execu8on

Symbolic heap • Nodes are symbolic terms • Edges are sub-terms


(even? x0)

∧

¬

∧∧ ∧

¬

(even? x1)

(even? x0)

(even? x1)

ite

‘(x0)‘()

‘(x1)append

ite

ys0 =

ys1 =

ys2 =

Symbolic execu8on Bounded model checking



Only condi8ons in the heap

Condi8ons and values (lists etc.)

in the heap

(even? x0)

∧

¬

∧∧ ∧

¬

(even? x1)

(even? x0)

(even? x1)

ite

‘(x0)‘()

‘(x1)append

ite

ys0 =

ys1 =

ys2 =

Symbolic execu8on Bounded model checking


Analyzing symbolic data structures

For each procedure, measure metrics that summarize the evolu8on of the

symbolic evalua8on graph and symbolic heap


For each procedure, measure metrics that summarize the evolu8on of the

symbolic evalua8on graph and symbolic heap

Summarize metrics as a score to rank procedures in the program


Symbolic evalua<on an<-paJernsCommon issues and repairs

Common an<-paJerns and repairs in symbolic evalua<onAlgorithmic mismatch

Algorithms or op8miza8ons poorly suited to symbolic evalua8on



(define (list-set lst idx val) (match lst [(cons x xs) (if (= idx 0) (cons val xs) (cons x (list-set xs (- idx 1) val))] [_ lst]))

Terminates early once idx is found



(define (list-set lst idx val) (match lst [(cons x xs) (cons (if (= idx 0) val x) (list-set xs (- idx 1) val))]

[_ lst]))




[_ lst])) Always recurse to the end of lst




[_ lst])) Always recurse to the end of lst

Tim

e (s

ec)

0

5

10

15

20

Length0 500 1000 1500 2000

OriginalRepaired



Irregular representa8on Data structures of different shapes create different paths

Missed concre8za8on Lost opportuni8es to exploit concrete values

Empirical resultsCase studies and evalua8on

Three symbolic profilersWe developed two implementa8ons:

• The RoseJe solver-aided language (Racket)

• The Jalangi dynamic analysis framework (JavaScript)

Since publica8on, based on our work:

• The Crucible symbolic simula8on library (C, Java, …) by Galois

Three symbolic profilersWe developed two implementa8ons:

• The RoseJe solver-aided language (Racket)

• The Jalangi dynamic analysis framework (JavaScript)

Since publica8on, based on our work:

• The Crucible symbolic simula8on library (C, Java, …) by Galois

Today

Ac<onable: real-world bugs

Tool SpeedupType system soundness checker [POPL’18] 1.35×

Refinement type checker for Ruby [VMCAI’18] 6×

File-system crash consistency verifier [ASPLOS’16] 24×

Cryptographic protocol verifier [FM’18] 29×

SQL query verifier [CIDR’17] 75×

Safety-cri8cal radiotherapy system verifier [CAV’16] 290×

Mul8ple patches accepted by developers

Case studies on published RoseWe-based tools

Ac<onable: real-world bugs

Tool SpeedupType system soundness checker [POPL’18] 1.35×

Refinement type checker for Ruby [VMCAI’18] 6×

File-system crash consistency verifier [ASPLOS’16] 24×

Cryptographic protocol verifier [FM’18] 29×

SQL query verifier [CIDR’17] 75×

Safety-cri8cal radiotherapy system verifier [CAV’16] 290×

Mul8ple patches accepted by developers

Used in produc8on at the UW Medical Center

Case studies on published RoseWe-based tools

Explainable: study real usersSmall user study: 8 RoseWe users, asked to find known performance bug in 4 programs

Users solved every task more quickly when they had access to symbolic profiling

6 failures without symbolic profiling, none with

Qualita8ve feedback: “gave insight into what RoseWe is doing” “even more useful on my own code”

Does my program work on all inputs?

Verifica8on$

Is there a program that does what I want?

Synthesis👷

https://unsat.org


raco symprofile file.rkt

https://unsat.org

Finding Code That Explodes Under Symbolic Evalua

Documents

Transcript of Finding Code That Explodes Under Symbolic Evalua