UC Berkeley GuidedSampler: EECS Department Rafael Dutra...
Transcript of UC Berkeley GuidedSampler: EECS Department Rafael Dutra...
GuidedSampler:Coverage-guided Samplingof SMT SolutionsRafael Dutra, Jonathan Bachrach, Koushik SenEECS DepartmentUC Berkeley
Formal Methods in Computer-Aided DesignOctober 25, 2019
A SMT solver can generate one solution:
Constraint Sampling
2
mem[0] mem[1]
σ0 1 0 0 0 1 0 0 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
Input: SMT formula φ
Goal: Generate many solutions to φ
Constraint Sampling
3
mem[0] mem[1]
σ0
σ1
σ2
σ3
σ4
σ5
1 0 0 0 1 0 0 0
0 0 0 0 1 0 1 0
1 1 0 0 1 0 0 1
1 0 0 0 0 1 0 0
0 1 0 0 1 0 1 1
0 0 0 0 0 1 1 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
Input: SMT formula φ
● Synthesis● Symbolic execution
Motivation: Sampling Solutions
● Thoroughly exercising some target functionality● Constrained-Random Verification
4
int4 x, y, z, w;int4 mem[4] = {x, y, z, w};for (int4 i = 0; i < 4; ++i) {
mem[mem[i]] *= -1;}
i < 4
mem[0] < 0∨ mem[0] ≥ 4
SMT: Satisfiability Modulo Theories
5
SMT formula φ
∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])
mem ∈ Array(BV[4], BV[4])
mem ∈ Array(BV[4], BV[4])
SMT: Satisfiability Modulo Theories
6
SMT formula φBit-vector
∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])
SMT: Satisfiability Modulo Theories
7
SMT formula φBit-vector
Array
mem ∈ Array(BV[4], BV[4])
∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])
State of the art
● SMTSampler (our prior work)○ Efficient generation of solutions for SMT formulas
● Markov Chain Monte Carlo (MCMC)○ Works for linear constraints and can generate biased solutions
● Constraint solver heuristics○ Can be expensive, requiring one solver call per solution
● Universal hashing○ Expensive, but can guarantee exact distribution of solution
● Weighted Sampling○ Literal-weighted distributions: WAPS
8
Goal: Generate solutions to φ
SMTSampler
9
mem[0] mem[1]
σ0
σ1
σ2
σ3
σ4
σ5
1 0 0 0 1 0 0 0
0 0 0 0 1 0 1 0
1 1 0 0 1 0 0 1
1 0 0 0 0 1 0 0
0 1 0 0 1 0 1 1
0 0 0 0 0 1 1 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
Input: SMT formula φ
Goal: Generate solutions to φ
Coverage-guided Sampling
10
mem[0] mem[1]
σ0
σ1
σ2
σ3
σ4
σ5
1 0 0 0 1 0 0 0
0 0 0 0 1 0 1 0
1 1 0 0 1 0 0 1
1 0 0 0 0 1 0 0
0 1 0 0 1 0 1 1
0 0 0 0 0 1 1 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
mem’[1] < 0
Input: SMT formula φ
Input: Coverage predicates
mem’[1] ≥ 4 mem’[0] < 0
ψ1 ψ2 ψ3
Goal: Generate solutions to φ such that the predicates ψ1, ψ2, …, ψn are covered uniformly
Coverage-guided Sampling
11
mem[0] mem[1] ψ1 ψ2 ψ3
σ0
σ1
σ2
σ3
σ4
σ5
1 0 0 0 1 0 0 0
0 0 0 0 1 0 1 0
1 1 0 0 1 0 0 1
1 0 0 0 0 1 0 0
0 1 0 0 1 0 1 1
0 0 0 0 0 1 1 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
mem’[1] < 0
Input: SMT formula φ
0 1 0
0 1 0
0 0 1
1 1 0
0 1 0
1 1 0
mem’[1] ≥ 4 mem’[0] < 0
Input: Coverage predicates
ψ1 ψ2 ψ3
Goal: Generate solutions to φ such that the predicates ψ1, ψ2, …, ψn are covered uniformly
Coverage-guided Sampling
12
mem[0] mem[1] ψ1 ψ2 ψ3
σ0
σ1
σ2
σ3
σ4
σ5
1 0 0 0 1 0 0 0
0 0 0 0 1 0 1 0
1 1 0 0 1 0 0 1
1 0 0 0 0 1 0 0
0 1 0 0 1 0 1 1
0 0 0 0 0 1 1 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
mem’[1] < 0
Input: SMT formula φ
0 1 0
0 1 0
0 0 1
1 1 0
0 1 0
1 1 0
mem’[1] ≥ 4 mem’[0] < 0
Input: Coverage predicates
ψ1 ψ2 ψ3
Uniformity over Coverage Classes
13
Uniformity over Coverage Classes
14
Uniformity over Coverage Classes
15
Challenges
● Coverage of the formula might still not be ideal even using state-of-the-art approaches, such as SMTSampler
● User might be interested in a specific notion of coverage for the produced solutions
16
GuidedSampler
17
GuidedSampler
Our goals:
● Sample solutions from a formula φ, but have the distribution determined by the coverage predicates ψ1, ψ2, …, ψn
● Uniformly sample solutions from the different coverage classes
● Uniformly sample within each coverage class
Our approach:
● Compute simple mutations that can be applied to one solution to generate another solution from a different class
● Combine those mutations together to generate a large number of new solutions
18
19
Formula φ
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
Coverage Predicates
...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1
ψ2
ψ3
20
Formula φx = mem[0]y = mem[1]
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
21
Formula φx = mem[0]y = mem[1]
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])ψ1 ψ2 ψ3
0 1 1RandomClass
...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
22
Solution σ 1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])ψ1 ψ2 ψ3
0 1 1
ψ1 ψ2 ψ3
0 1 0...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
RandomClass
23
Solution σ 1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
ψ1 ψ2 ψ3
0 1 0
MAX-SMT ...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
24
Solution σ 1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
Hard constraints● φ● ψ1 ≠ 0
Soft constraints● ψ2 = 1● ψ3 = 0● x1 = 0● x2 = 0● x3 = 0● y0 = 1● y1 = 0● y2 = 0● y3 = 0
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
ψ1 ψ2 ψ3
0 1 0
MAX-SMT ...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
σ1
25
Solution σ
0 0 0 0 1 0 1 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
ψ1 ψ2 ψ3
0 1 0
1 1 0
...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
σ1
26
Solution σ
0 0 0 0 1 0 1 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
ψ1 ψ2 ψ3
0 1 0
MAX-SMT ...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
σ1
27
Solution σ
0 0 0 0 1 0 1 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
σ1
28
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ2
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0
0 0 0
...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
σ1
29
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ2
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
σ1
30
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ2
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
Coverage Predicates
ψ1
ψ2
ψ3
σ1
31
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ2 σ3
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
0 0 1
Coverage Predicates
ψ1
ψ2
ψ3
σ1
32
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ2 σ3
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ1
33
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
MAX-SMT
= σ ⊕ σ1 = σ ⊕ σ2 = σ ⊕ σ3
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
δ12
σ1
34
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
MAX-SMT
= δ1 ∨ δ2
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ12
δ12
σ1
35
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
= σ ⊕ δ12
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1 ψ2 ψ3
1 0 0
...
Coverage Predicates
ψ1
ψ2
ψ3
σ12
δ12
σ1
36
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
= σ ⊕ δ12
Why does it work?● δ1 and δ2 are a minimal set of bits that can be
flipped and preserve the satisfiability of the formula● It’s likely that the formula has some clauses
establishing a relation between those bits● Those clauses will likely still be satisfied when
flipping both the bits in δ1 and δ2
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1 ψ2 ψ3
1 0 0
...
Coverage Predicates
ψ1
ψ2
ψ3
σ12
δ12
σ1
37
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
= σ ⊕ δ12
⇐ And new sample σ12 is likelyfrom a new coverage class
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1 ψ2 ψ3
1 0 0
...
Coverage Predicates
ψ1
ψ2
ψ3
σ12
δ12
σ1
38
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
= σ ⊕ δ12
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1 ψ2 ψ3
1 0 0
...
Coverage Predicates
ψ1
ψ2
ψ3
σ12
δ12
σ1
39
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
δ131 0 0 0 1 1 1 0
= δ1 ∨ δ3
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ12
δ12 δ13
σ13
σ1
40
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
1 0 0 0 1 1 1 0
0 0 0 0 0 1 1 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
= σ ⊕ δ13
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1 ψ2 ψ3
1 0 1
...
Coverage Predicates
ψ1
ψ2
ψ3
σ12
δ12 δ13 δ23
σ13
σ1
41
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
1 0 0 0 1 1 1 0
0 0 0 0 0 1 1 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
0 1 0 0 1 1 0 1
MAX-SMT
= δ2 ∨ δ3
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ12
δ12 δ13 δ23
σ13
σ1
42
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
1 0 0 0 1 1 1 0
0 0 0 0 0 1 1 0
σ2 σ3
δ31 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
0 1 0 0 1 1 0 1
MAX-SMT
σ231 1 0 0 0 1 0 1
= σ ⊕ δ23
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1 ψ2 ψ3
0 0 1
...
Coverage Predicates
ψ1
ψ2
ψ3
σ12 σ13
σ1
43
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ12 σ13
σ1
44
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
17 / 18 valid solutions
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ12 σ13
σ1
45
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
σ1230 1 0 0 0 1 1 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1 ψ2 ψ3
1 0 1
...
Coverage Predicates
ψ1
ψ2
ψ3
σ12 σ13
σ1
46
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
σ1230 1 0 0 0 1 1 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
ψ1 ψ2 ψ3
1 0 1 Repeated class
...
Coverage Predicates
ψ1
ψ2
ψ3
σ12 σ13
σ1
47
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ12 σ13
σ1
48
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
At most ngenerate atomic mutations
=O(n6)mutations: NO MAX-SMT ( ) n
6 samples by combining
MAX-SMT calls to
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ12 σ13
σ1
49
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
At most 50generate atomic mutations
15 890 700mutations: NO MAX-SMT
samples by combining
MAX-SMT calls to
MAX-SMT
ψ1 ψ2 ψ3
0 1 0......
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
σ12 σ13
σ1
50
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0...
Coverage Predicates
ψ1
ψ2
ψ3
...
mem’[1] < 0
mem’[1] ≥ 4
mem’[0] < 0
σ12 σ13
σ1
51
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 0 0 0 0 1 1 0
σ2 σ3
0 1 0 0 1 0 1 1
MAX-SMT
σ231 1 0 0 0 1 0 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
MAX-SMT
ψ1 ψ2 ψ3
0 1 0...
Coverage Predicates
ψ1
ψ2
ψ3
Random assignment
σ′
52
Key Ideas
σ
Random assignment
Base solution
σ′
53
Key Ideas
● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values
σ2
σ1
σ
Random assignment
Base solution
Closest solutions
σ′
54
σ3σ4
Key Ideas
● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values
● M1: Find neighboring solutions that flip coverage predicates
Random assignment
Base solution
Closest solutions
Generated samples
σ2 σ12
σ1
σσ′
55
σ3σ4
Key Ideas
● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values
● M1: Find neighboring solutions that flip coverage predicates
● M2: Whenever generating a new sample, discard it if it’s from a repeated coverage class
σ2 σ12
σ1
σ
Random assignment
Base solution
Closest solutions
Generated samples
σ′
56
σ3σ4
Key Ideas
● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values
● M1: Find neighboring solutions that flip coverage predicates
● M2: Whenever generating a new sample, discard it if it’s from a repeated coverage class
σ2 σ12
σ1
σ
Random assignment
Base solution
Closest solutions
Generated samples
σ′
57
σ3σ4
Key Ideas
● M3: In the MAX-SMT query to generate σ, set coverage predicates to random values
● M1: Find neighboring solutions that flip coverage predicates
● M2: Whenever generating a new sample, discard it if it’s from a repeated coverage class
Implementation
● Implemented in C++ using Z3 as the constraint solver● https://github.com/RafaelTupynamba/GuidedSampler
58
Experiments on SMT-LIB
We evaluated GuidedSampler on 213 industrial benchmarks from 22 classes.
59
Benchmark Class Average # Nodes Average # Bits
QF_AUFBV/ecc 179 1931
QF_ABV/bmc-arrays 855 53
QF_ABV/stp_samples 1139 192
QF_BV/bmc-bv-svcomp14 7518 7607
QF_BV/tacas07 8812 16620
QF_BV/sage/app8 978 1047
Experiments
We compared 6 approaches for SMT sampling:● BH: Baseline with hard constraints● BS: Baseline with soft constraints● S0: SMTSampler [1]● S1 = S0 + M1 (flipping coverage predicates to generate neighboring solutions)● S2 = S0 + M1 + M2 (discarding solutions from repeated classes)● S3 = S0 + M1 + M2 + M3: GuidedSampler (randomize class of base solution)
[1] Rafael Dutra, Jonathan Bachrach and Koushik Sen. 2018. SMTSampler: Efficient Stimulus Generation from Complex SMT Constraints. In ICCAD 2018. 60
Coverage Predicates
● Internal Predicates○ Look at values of internal nodes○ Analogous to internal wires in a circuit○ General notion of coverage from the
formula itself
● Random Predicates○ Random formulas generated from a
grammar including variables of φ○ Problem-specific notion of coverage
61
Experiments: Unique Coverage Classes
62
Higher is better
GuidedSampler vs. BH Baseline
Number of uniquecoverage classesper time
Experiments: Unique Coverage Classes
63
Higher is better
GuidedSampler vs. BS Baseline
Number of uniquecoverage classesper time
Experiments: Unique Coverage Classes
64
Higher is better
GuidedSampler vs. SMTSampler
Number of uniquecoverage classesper time
Experiments: Unique Coverage Classes
65
Higher is better
S3 = GuidedSamplerS0 = SMTSamplerBS, BH: baselines
Number of uniquecoverage classesper time
Experiments: Uniformity over Coverage Classes
66
S3 = GuidedSamplerS0 = SMTSamplerBS, BH: baselines
→ GuidedSampler generated > 100 000 classes
Discussion
● The most important modification is M1, which allows covering 3.1 times more classes in average
● M1 and M2 are also essential for producing a more uniform distribution over coverage classes.
● Similar results for internal predicates and random predicates
67
Modifications:
● M1: Flipping coverage predicates to compute neighboring solutions
● M2: Discarding new solutions that repeat a previously seen coverage class
● M3: Randomizing coverage class of initial base solution
Conclusion
● Generating lots of solutions efficiently given an SMT formula
● Generate millions of solutions with tens of solver calls
● Achieve better coverage of the constraint space, even for user-defined coverage classes
68
σ12 = σ ⊕ δ12
δ12 = δ1 ∨ δ2
σ1
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
0 1 0 0 0 0 0 1
σ2
1 0 0 0 0 0 1 0 δ2δ1
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
ψ1 ψ2 ψ3
0 1 0
ψ1 ψ2 ψ3
1 0 0
MAX-SMT
x0 x1 x2 x3 y0 y1 y2 y3