Very Large Scale Integration II - VLSI II Synthesis Using RTL Compiler Hayri U ğur UYANIK
Program Synthesis using Deduction-Guided Reinforcement...
Transcript of Program Synthesis using Deduction-Guided Reinforcement...
![Page 1: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/1.jpg)
Program Synthesis using Deduction-Guided Reinforcement Learning
Yanju Chen, Chenglong Wang, Osbert Bastani, Isil Dillig and Yu Feng
![Page 2: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/2.jpg)
Program Synthesis 2
Find a program ! that satisfies all the specifications ".specifications # synthesizer program $
table transformation string transformationdata visualization HPC
![Page 3: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/3.jpg)
Program Synthesis 3
Deductive Reasoning
Statistical ReasoningDISCONNECTION
NEO(Feng et al. 2018)
BLAZE(Wang et al. 2018)
DEEPCODER(Balog et al. 2017)
AutoPandas(Bavishi et al. 2019)
MORPHEUS(Feng et al. 2017)
The feedback of deduction can not be seamlessly used by the statistical model.
![Page 4: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/4.jpg)
4
Can we bridge the statistical and deductive approaches in program synthesis in a seamless way?
There's a fundamental disconnection between program synthesis using pure statistical and deductive methods, so...
![Page 5: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/5.jpg)
5
CONCORDutilizes fine-grained feedbacks from deductive reasoning
to update statistical reasoning on the fly
![Page 6: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/6.jpg)
Our Approach: CONCORD(An Overview)
6
Take Action Deduce
UpdatePolicy
initial policy !"
!specification #
$
program
DSL
![Page 7: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/7.jpg)
A Running Example(Koala Habitat Selectivity Study*)
u (goal) aggregate two koala factor tracking lists:u (input1) factor 1 - GPS last seen locations (reversed order): [152,140,102,58,35]u (input2) factor 2 - tree conditions: [78,55,89,40,33]
u (output) model computes cumulative factors: [43,75,120,122,143]
7
koalaFactor::list->list->list
35 58 102 140 152
78
55
89
40
33
0
*Synthetic data inspired by Davies, N., Gramotnev, G., Seabrook, L. et al. Movement patterns of an arboreal marsupial at the edge of its range: a case study of the koala. Mov Ecol 1, 8 (2013).
![Page 8: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/8.jpg)
Sample Domain Specific Language(A DSL for List Processing)
u koalaFactor::list->list->listu koalaFactor([152,140,102,58,35],[78,55,89,40,33])=[43,75,120,122,143]
u solution program:u v1 <- reverse input1
u v2 <- sumUpTo input2u v3 <- sub v2,v1
u which is: sub(sumUpTo(input2), reverse(input1))
8
S -> N|LN -> 0|...|10|input1|inpu2L -> input1|input2|take(L,N)|drop(L,N)|sort(L)
|reverse(L)|add(L,L)|sub(L,L)|sumUpTo(L)
sample DSL
sumUpTo reverse
sub
input2 input1
solution ast
![Page 9: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/9.jpg)
Deductive Approach(Solving koalaFactor in Variant of CEGIS* Loop, Step1)
9
Decide Deduce
#1: add(sort(input1),take(input2,N))#2: add(sort(input1),drop(input2,N))#3: add(sort(input1),sumUpTo(input2))...#20: sub(sumUpTo(input2),reverse(input1))#21: add(sumUpTo(input1),take(input2,N))...
fixed candidate list
!
KBDSL
input1 = [152,140,102, 58, 35]input2 = [ 78, 55, 89, 40, 33]output = [ 43, 75,120,122,143]
*CEGIS: Counter-Example-Guided Inductive Synthesis
![Page 10: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/10.jpg)
Deductive Approach(Solving koalaFactor in Variant of CEGIS Loop, Step2)
10
Decide Deduce
#1: add(sort(input1),take(input2,N))#2: add(sort(input1),drop(input2,N))#3: add(sort(input1),sumUpTo(input2))...#20: sub(sumUpTo(input2),reverse(input1))#21: add(sumUpTo(input1),take(input2,N))...
fixed candidate list
!
KBDSL
*CEGIS: Counter-Example-Guided Inductive Synthesis
input1 = [152,140,102, 58, 35]input2 = [ 78, 55, 89, 40, 33]output = [ 43, 75,120,122,143]
![Page 11: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/11.jpg)
Deductive Approach(Solving koalaFactor in Variant of CEGIS Loop, Step3)
11
Decide Deduce
#1: add(sort(input1),take(input2,N))#2: add(sort(input1),drop(input2,N))#3: add(sort(input1),sumUpTo(input2))...#20: sub(sumUpTo(input2),reverse(input1))#21: add(sumUpTo(input1),take(input2,N))...
fixed candidate list
!
KBDSL
*CEGIS: Counter-Example-Guided Inductive Synthesis
input1 = [152,140,102, 58, 35]input2 = [ 78, 55, 89, 40, 33]output = [ 43, 75,120,122,143]
![Page 12: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/12.jpg)
Deductive Approach(Solving koalaFactor in Variant of CEGIS Loop, Step4)
12
Decide Deduce
#1: add(sort(input1),take(input2,N))#2: add(sort(input1),drop(input2,N))#3: add(sort(input1),sumUpTo(input2))...#20: sub(sumUpTo(input2),reverse(input1))#21: add(sumUpTo(input1),take(input2,N))...
fixed candidate list
• rich and accurate deduction feedback• efficient pruning of search space
Pros
• incorporating deductive feedback into existing statistical model is difficult
Cons
!
KBDSL
*CEGIS: Counter-Example-Guided Inductive Synthesis
![Page 13: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/13.jpg)
Statistical Approach(Solving koalaFactor using Data-Driven Machine Learning, Step1)
13
Infer Check#1: add(sort(input1),take(input2,N))#2: add(sort(input1),drop(input2,N))#3: add(sort(input1),sumUpTo(input2))...#20: sub(sumUpTo(input2),reverse(input1))#21: add(sumUpTo(input1),take(input2,N))...
candidate list
!"
#DSL
![Page 14: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/14.jpg)
14
#1: add(sort(input1),take(input2,N))#2: add(sort(input1),drop(input2,N))#3: add(sort(input1),sumUpTo(input2))...#20: sub(sumUpTo(input2),reverse(input1))#21: add(sumUpTo(input1),take(input2,N))...
candidate list
! (if in reinforcement learning setting)
Infer Check
Statistical Approach(Solving koalaFactor using Data-Driven Machine Learning, Step2)
!"
#DSL
![Page 15: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/15.jpg)
15
Infer Check#1: add(sort(input1),take(input2,N))#2: add(sort(input1),drop(input2,N))#3: add(sort(input1),sumUpTo(input2))...#20: sub(sumUpTo(input2),reverse(input1))#21: add(sumUpTo(input1),take(input2,N))...
candidate list
Statistical Approach(Solving koalaFactor using Data-Driven Machine Learning, Step3)
!"
#DSL
![Page 16: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/16.jpg)
16
Infer Check
#1: add(sort(input1),take(input2,N))#2: add(sort(input1),drop(input2,N))#3: add(sort(input1),sumUpTo(input2))...#20: sub(sumUpTo(input2),reverse(input1))#21: add(sumUpTo(input1),take(input2,N))...
candidate list
• data-driven candidate list• can update policy seamlessly
Pros
• feedback from checker is relatively less informative than that from deduction
Cons
Statistical Approach(Solving koalaFactor using Data-Driven Machine Learning, Step4)
!"
#DSL
![Page 17: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/17.jpg)
Our Approach: CONCORD(An Overview)
17
Take Action Deduce
UpdatePolicy
!
"
program
initial policy !#specification $
DSL
![Page 18: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/18.jpg)
CONCORD: Formalization(Program Synthesis as Markov Decision Process)
18
sumUpTo reverse
sub
input2 input1
initial state
final state +1
sumUpTo
sub
L
L
sub
L L
S
sub 0.5
sumUpTo 0.3
input2 0.4
reverse 0.2
input1 0.3
...
... ...
action probability S working node instantiated nodeuninstantiated node?state reward
maximize rewardsObjective
![Page 19: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/19.jpg)
CONCORD: Running Example(Solving koalaFactor using Deduction-Guided RL, Step1)
19
Take Action Deduce
#1: add(sort(input1),take(L,N))#2: add(sort(input1),drop(L,N))#3: add(sort(input1),sumUpTo(L))...#20: sub(sumUpTo(input2),reverse(L))#21: add(sumUpTo(input1),take(L,N))...
candidate list
UpdatePolicy
!
!"#
DSL
![Page 20: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/20.jpg)
CONCORD: Running Example(Solving koalaFactor using Deduction-Guided RL, Step2)
20
Take Action Deduce
#1: add(sort(input1),take(L,N))#2: add(sort(input1),drop(L,N))#3: add(sort(input1),sumUpTo(L))...#20: sub(sumUpTo(input2),reverse(L))#21: add(sumUpTo(input1),take(L,N))...
candidate list
UpdatePolicy
!add(sort(L),take(L,N))add(sort(L),drop(L,N))add(sumUpTo(L),take(L,N))...
sampled infeasible programs
!"#
DSL
![Page 21: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/21.jpg)
CONCORD: Running Example(Solving koalaFactor using Deduction-Guided RL, Step3)
21
Take Action Deduce
updated candidate list
UpdatePolicy
!add(sort(L),take(L,N))add(sort(L),drop(L,N))add(sumUpTo(L),take(L,N))...
sampled infeasible programs
...#20: sub(sumUpTo(input2),reverse(L))
#2: add(sort(input1),drop(L,N))#3: add(sort(input1),sumUpTo(L))
#21: add(sumUpTo(input1),take(L,N))...
#1: add(sort(input1),take(L,N))!"#
DSL
![Page 22: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/22.jpg)
CONCORD: Running Example(Solving koalaFactor using Deduction-Guided RL, Step3)
22
Take Action Deduce
#1: sub(sumUpTo(input2),reverse(L))#2: add(sort(input1),sumUpTo(L))...#24: add(sumUpTo(input1),take(L,N))#25: add(sort(input1),take(L,N))#26: add(sort(input1),drop(L,N))...
updated candidate list
UpdatePolicy
!add(sort(L),take(L,N))add(sort(L),drop(L,N))add(sumUpTo(L),take(L,N))...
sampled infeasible programs
!"#
DSL
![Page 23: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/23.jpg)
CONCORD: Running Example(Solving koalaFactor using Deduction-Guided RL, Step4)
23
Take Action Deduce
#1: sub(sumUpTo(input2),reverse(input1))#2: add(sort(input1),sumUpTo(input2))...#24: add(sumUpTo(input1),take(input2,N))#25: add(sort(input1),take(input2,N))#26: add(sort(input1),drop(input2,N))...
updated candidate list
UpdatePolicy
!add(sort(L),take(L,N))add(sort(L),drop(L,N))add(sumUpTo(L),take(L,N))...
sampled infeasible programs• feedback from deduction flows seamlessly to the policy update• not only prune the search space, but also promote good candidates
What's the difference?
!"#
DSL
![Page 24: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/24.jpg)
CONCORD: Synthesis Algorithm 24
DeductionEngine
infeasibleSampler
add(sort(input1),take(L,N))partial program
feasible empty set ∅
add(sort(L),take(L,N))add(sort(L),drop(L,N))add(sumUpTo(L),take(L,N))...
sampled infeasible programsDeduceTake
ActionUpdatePolicy
"
![Page 25: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/25.jpg)
Deduce
CONCORD: Synthesis Algorithm 25
DeductionEngine
infeasibleSampler
add(sort(input1),take(L,N))partial program
feasible empty set ∅
add(sort(L),take(L,N))add(sort(L),drop(L,N))add(sumUpTo(L),take(L,N))...
sampled infeasible programs
?(?(L),take(L,N))?(?(L),drop(L,N))...
logical constraints
![Page 26: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/26.jpg)
Deduce
CONCORD: Synthesis Algorithm(Deduction-Guided Reinforcement Learning)
26
DeductionEngine
infeasibleSampler
add(sort(input1),take(L,N))partial program
feasible empty set ∅
add(sort(L),take(L,N))add(sort(L),drop(L,N))add(sumUpTo(L),take(L,N))...
sampled infeasible programs
Take Action
UpdatePolicy
"
weighted future rewards
importance weighting
Off-Policy Sampling
program distribution of policy
program distribution of Sampler
![Page 27: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/27.jpg)
Evaluations(Research Questions and Experiment Settings)
u Research Questions:u How does Concord compare against existing synthesis tools?
u How effective is the off-policy RL algorithm compared to standard policy gradient?
u Experiment Settingsu Deduction Engine: NEO's (Feng et al. 2018) deduction engineu Policy: Gated Recurrent Unit (GRU)
u Benchmarks: DEEPCODER benchmarks used in NEO
u 100 challenging list processing problems
u Comparison between:u NEO (Feng et al. 2018)
u DEEPCODER (Balog et al. 2017)
27
The architecture of the policy network used
![Page 28: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/28.jpg)
Evaluations(Experiment Results and Analysis)
28
tool solved timeCONCORD 82% 36s
NEO 71% 99sDEEPCODER 32% 205s
tool solved speedup over NEO
CONCORD 82% 8.71xCONCORD
(StandardPG)65% 2.88x
• Concord tightly couples statistical and deductive reasoning based on reinforcement learning.• The off-policy reinforcement learning technique is effective.
Take-Away Message
![Page 29: Program Synthesis using Deduction-Guided Reinforcement ...yanju/publications/cav20_concord_slides.p… · u How does Concord compare against existing synthesis tools? u How effective](https://reader036.fdocuments.net/reader036/viewer/2022071502/612277d87f900259960a9946/html5/thumbnails/29.jpg)
Thank you!Questions?
29