Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of...
Transcript of Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of...
![Page 1: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/1.jpg)
Wenchao Li
Dissertation Talk
EECS Department, UC Berkeley
Thesis Committee: Sanjit A. Seshia (Advisor),
Andreas Kuehlmann, Francesco Borrelli
![Page 2: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/2.jpg)
2
Acknowledgement
• Prof. Sanjit A. Seshia
• Prof. Andreas Kuehlmann, Prof. Francesco Borrelli
• Dr. Alessandro Forin (MSR, Redmond)
• Dr. Natarajan Shankar, Dr. Shalini Ghosh (SRI International, Menlo
Park)
• GigaScale Research Center
• TerraSwarm Research Center
• Daniel Holcomb, Bryan Brady, Susmit Jha, Alexandre Donze,
Ruediger Ehlers, Indranil Saha, Jonathan Kotker, Rohit Sinha, Dorsa
Sadigh, Zach Wasson, Wei Yang Tan, Garvit Juniwal, Ankush Desai,
Nishant Totla, Daniel Fremont
• Colleagues in the DOP center
• Friends
• Nuo Zhang
![Page 3: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/3.jpg)
![Page 4: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/4.jpg)
4
Cost of Bugs
• Human loss: Pacemakers,
Aircraft, Nuclear reactor
controllers, Car engine
management system, etc.
• Financial Loss: 1994
Pentium FDIV costs $475
million, Mars Rover, North
America Blackout, etc.
Reality Check:
o Writing assertion is a time-consuming manual process and is
perceived as “difficult”.
o “During the first formal verification runs of a new hardware
design, typically 20% of the formulas are found to be
trivially valid.” [IBM Haifa]
![Page 5: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/5.jpg)
5
Verification and Synthesis
Model 𝑀
An execution should
never reach an error
state.
Specification 𝜓
Check 𝑀 ⊨ 𝜓
Verification
Find 𝑀 s.t. 𝑀 ⊨ 𝜓
Synthesis
err
![Page 6: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/6.jpg)
6
Specification:
1. Every request should be
eventually granted.
2. Never reach an error
state.
Assertion-based
Verification
Pass? Debug No Yes Done
Coverage Good
Poor
Need More
Specifications
Cex
Verification is as Good as Specification
![Page 7: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/7.jpg)
7
Specification:
1. Every request should be
eventually granted.
2. Never reach an error
state.
Assertion-based
Verification
Pass? Debug No Yes Done
Coverage Good
Poor
Need More
Specifications
Cex
Verification is as Bad as Specification
“Not a bug!”
Missing
Assumptions
![Page 8: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/8.jpg)
8
Formalisms
Temporal specifications can be mined systematically both
from observed and counteracting behaviors, and are useful
for automating difficult tasks in verification and synthesis
such as localizing bugs and finding missing assumptions.
Algorithms
Applications
![Page 9: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/9.jpg)
9
![Page 10: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/10.jpg)
Linear Temporal Logic
10
p p p p p p p p …
p …
p p p …
G p
F p
p q q p …
G F p
G (p F q)
Formal specification: behavior description
supported by logic-based languages
![Page 11: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/11.jpg)
11
Specification Mining with Templates
Example specification :
req reset grant req grant grant
3 cycles 2 cycles
7 cycles
(1) every a is followed by a b within 3 cycles; (2) every two as are separated by at least 7 cycles.
![Page 12: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/12.jpg)
Part I Requirement Generation and Error Localization
![Page 13: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/13.jpg)
13
M
Static: Infer specification directly from the description of
the design, e.g. synthesis of interface specification for Java
classes [Alur et. al., 2005]
Dynamic: Infer likely specification from simulation
/execution traces, e.g. DAIKON [Ernst et. al., 2000]
Requirement Generation
Common behaviors as
likely specifications Dynamic
Static
Automata-based [DAC’10] Sparse Coding [RV’12]
![Page 14: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/14.jpg)
An Automata-based Monitoring Approach
Specification Mining:
![Page 15: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/15.jpg)
15
Traces
Specification
Mining
Engine User Event
Definitions
Library of
Temporal
Patterns
Ranking
Module Mined
Assertions
Most Relevant
Assertions
Mining Temporal Properties
[Li et al., Scalable Specification Mining for Verification and Diagnosis. DAC 2010]
![Page 16: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/16.jpg)
16
All possible mappings
0 1
err Challenges:
Solutions:
![Page 17: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/17.jpg)
17
Design |Τ| |Τ∆m| nm |S| |Smerged| Runtime (s)
eMIPS 5 mil 5408 108 2079 1028 51
Router 0.23 mil 12420 28 120 74 13
I2C 1.6 mil 20904 33 389 308 9
CAN 26 mil 36100 175 3272 1356 71
Requirement Generation:
eMIPS - 278 modules and more than 20,000 signals
Summary:
• Industrial-size designs;
• Traces of millions of cycles;
• Mine relevant temporal properties efficiently.
![Page 18: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/18.jpg)
19
010101010101
011011010101
010111111010
10101
Where?
Post-Si Challenges: • Limited observability
• Long error detection latency
• Transient and hard-to-reproduce bugs
Expensive: $1 million to redesign the masks [Ying et al., 2005];
3:1 headcount for design vs. post-Si validation [Patra et al., 2007]; post-Si validation
consumes 35% of chip development time on average [Abramovici et al., 2006]
Research Question
![Page 19: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/19.jpg)
20
Proposed Solution
Normal
Traces
Diagnosis
Error
Trace
Candidate
Ranking
Candidate
Fault
Locations
Assertion
Miner
Assertion
Miner
![Page 20: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/20.jpg)
21
Type of Fault Fault Coverage % Time Localization % Module
Localization %
Stuck-at 100 - 100
Erroneous
Transition 100 - 100
Erroneous
Assignment 100 - 57
Transient 100 81 56
Error Localization:
CMP router; localize to within 15 cycles for transient faults
Summary:
• eMIPS: effectively localize different design bugs.
• CMP router: effectively localize transient bugs also.
• Mining simple distinguishing patterns can help to
localize complex bugs.
![Page 21: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/21.jpg)
22
Research Question
0.8 * + 0.3 * + 0.5 * 0.8 * + 0.3 * + 0.5 *
Sparse Coding:
Sparsity helps to uncover latent structure
e.g. finding edge detectors in an unsupervised setting
Specification formalism: Express each subtrace as a
Boolean combination of a few “basis subtraces”– a
(sparsity-constrained) Boolean matrix factorization problem.
[Li and Seshia. Sparse Coding for Specification Mining and Error Localization. RV 2012]
![Page 22: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/22.jpg)
A Sparse Coding Approach
Specification Mining:
![Page 23: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/23.jpg)
24
Problem Formulation
1 1 0 0 1
0 0 1 0 1
1 1
1 0
0 0
0 1
= ○
basis coefficient
Multiplication as “AND”
Addition as “OR”
columns are sparse
Subtrace
t
![Page 24: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/24.jpg)
25
Given a data matrix 𝑋 ∈ 𝑩𝒎×𝒏 and a positive integer 𝐶, the
sparsity-constrained Boolean factorization problem is to find 𝑘,
𝐵 = 𝑩𝒎×𝒌 and 𝑆 = 𝑩𝒌×𝒏 such that
𝑋 = 𝑩 ∘ 𝑺
and 𝑺∙,𝒊 𝟏≤ 𝑪, ∀𝒊
(and 𝑺𝒊,𝒋𝒋𝒊 is maximized).
𝑋 𝐵 𝑆
C = 2
Sparsity-Constrained Boolean Factorization
![Page 25: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/25.jpg)
• Observe that the data matrix X can be viewed as the adjacency
matrix for a bipartite graph.
• Idea: factorization → biclique cover (biclique ↔ basis subtrace)
26
v
u
Algorithm Idea
![Page 26: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/26.jpg)
30
• Error localization and explanation based on reconstruction:
A subtrace has an error if it cannot be
reconstructed from the basis subtraces
• A subtrace is error-free if
𝑋∙,𝑖⨁(𝐵 ∘ 𝑆∙,𝑖) 1= 0
0 1 0 1 1 0 … …
1 0 0 1 1 1 … …
0 1 0 0 1 0 … …
1 0 0 1 0 1 … …
𝑋∙,1
Minimize 𝑋∙,𝑖⨁(𝐵 ∘ 𝑆∙,𝑖) 1
Subject to 𝑆∙,𝑖 ≤ 𝐶
𝑋∙,2
𝑆∙,𝑖
Error Localization
All subtraces
Space
spanned by
the learned
basis
Correct
subtraces
Error
![Page 27: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/27.jpg)
• Chip Multiprocessor Router:
– Observe 14 control signals
– Subtrace width: 2 cycles
– Learn the basis from a single error-free
trace of 1000 cycles: 0.243 seconds to
obtain 189 basis subtraces from 93
distinct subtraces
31
• Error Localization:
– Inject a single bit flip at a random cycle for each of 99 error traces
– Localize the error to the subtrace (out of 999) where it was injected
• Comparisons:
– Baseline approach (1): hash all distinct subtraces – report error even
before an error is injected for the 99 traces
– Baseline approach (2): use unit basis – 0% localization
– Sparse Coding: 55.6% localization
A CMP
Router
in a NoC
Experimental Results
[Source: Daniel Holcomb]
![Page 28: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/28.jpg)
32
Automata-based: [Li et al., 2010]
• An efficient algorithm for mining temporal properties from
traces of digital designs.
• Effective algorithm for localizing bugs in hardware using
distinguishing patterns.
Sparse Coding: [Li et al., 2012]
• A novel formalism of specification based on the notion of
basis subtraces.
• An unsupervised algorithm for learning basis subtraces.
• An effective way of using basis subtraces to localize bugs.
![Page 29: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/29.jpg)
33
![Page 30: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/30.jpg)
Part II Assumption Mining for LTL Synthesis
![Page 31: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/31.jpg)
Temporal Logic Synthesis
35
Automatically construct an implementation that is
guaranteed to satisfy its behavioral description.
Behavioral Description Implementation
Synthesis
![Page 32: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/32.jpg)
36
Digital Circuit
PSL
“Writing a complete formal specification for the arbiter was not trivial. Many
aspects of the arbiter are not defined in ARM’s standard.”
![Page 33: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/33.jpg)
A Counterstrategy-Guided Approach
Assumption Mining:
![Page 34: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/34.jpg)
GR(1) Specifications
38
![Page 35: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/35.jpg)
39
![Page 36: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/36.jpg)
40
![Page 37: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/37.jpg)
41
![Page 38: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/38.jpg)
42
GR(1) Synthesis ~ Games
![Page 39: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/39.jpg)
43
![Page 40: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/40.jpg)
44
![Page 41: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/41.jpg)
45
Mining with Templates
![Page 42: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/42.jpg)
49
Iterative Search
GR(1)
Synthesis
Realizable
Unrealizable
Templates Compute
Mine
![Page 43: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/43.jpg)
Search Optimizations
…
…
50
![Page 44: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/44.jpg)
52
![Page 45: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/45.jpg)
Experimental Evaluation
53
![Page 46: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/46.jpg)
Summary of Contributions
54
• First counterstrategy-guided synthesis framework
• An efficient algorithm with theoretical guarantees for
assumption generation – a key problem in correct-by-
construction synthesis from temporal logic.
![Page 47: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/47.jpg)
Synthesizing Human-in-the-Loop Controllers
Assumption Mining:
58
![Page 48: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/48.jpg)
Many safety-critical systems interact with humans. The correctness
of such systems depend on both the correctness of autonomous
controller, actions of the human and their interaction.
59
![Page 49: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/49.jpg)
“Vehicles at this level of automation enable the driver to cede
full control of all safety-critical functions under certain traffic
or environmental conditions and in those conditions to rely
heavily on the vehicle to monitor for changes in those
conditions requiring transition back to driver control. The
driver is expected to be available for occasional control, but
with sufficiently comfortable transition time.”
Level 0: No Automation: Driver is in complete control
Level 1: Function Specific Automation Pre-charged Brakes
Level 2: Combined Function Automation Cruise Control
Lane Keeping
Level 3: Limited Self Driving Automation
Level 4: Full Self Driving Automation
Source: National Highway Traffic Safety Administration. Preliminary
statement of policy concerning automated vehicles, May 2013.
60
![Page 50: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/50.jpg)
61
Research Question
MIT Cornell Crash during DARPA Urban Challenge, 2007
![Page 51: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/51.jpg)
62
Human-in-the-Loop Controllers
Low probability of human
control needed
Notify danger ahead of time
Safe until human takes over
control
Determine control switch
based on monitored
information
[Li et al., Synthesis of Human-in-the-Loop Controllers. UCB Tech. Report 2013]
![Page 52: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/52.jpg)
63
Controller Synthesis
Autonomous
Controller
Advisory Controller
Human-in-
the-Loop
Controller
System
Specification
Environment
Assumption
Temporal Logic
Synthesis
Realizable
Unrealizable?
Compute
Counterstrategy
![Page 53: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/53.jpg)
64
Theoretical Guarantees
[Li et al., Synthesis of Human-in-the-Loop Controllers. UCB Tech. Report 2013]
![Page 54: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/54.jpg)
65
A Car Following Example
Autonomous car: A
Environment cars: B & C
Objective: A follows B, and
when this is not achievable,
switches control to the human
driver with sufficient time for
her to respond.
Follow := move to a square
where A can still sense B
Given specs encoding
movement rules and T = 1.
![Page 55: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/55.jpg)
66
![Page 56: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/56.jpg)
67
![Page 57: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/57.jpg)
68
A Car Following Example
[Li et al., Synthesis of Human-in-the-Loop Controllers. UCB Tech. Report 2013]
![Page 58: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/58.jpg)
69
Assumption Mining: [Li et al., 2011]
• First counterstrategy-guided approach for synthesis from
temporal logic.
• An efficient algorithm with theoretical guarantees for
mining assumptions for GR(1) synthesis.
Human-in-the-Loop Controllers: [Li et al., 2013]
• A novel formalism of human-in-the-loop controllers.
• Identify criteria with application to driving automation.
• An algorithm for synthesizing human-in-the-loop controllers
that automatically satisfy these criteria, from temporal logic
specifications.
![Page 59: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/59.jpg)
70
![Page 60: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/60.jpg)
CrowdMine: Gamification and Crowdsourcing
Human Inputs:
71
![Page 61: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/61.jpg)
72
Counterexample
[URL: http://verifun.eecs.berkeley.edu/crowdmine2/]
CrowdMine
Two Sampled Subtraces
Traces
Selected Patterns → LTL Formulas → Model Checker
Spec. Found
No Cex
![Page 62: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/62.jpg)
73
Preliminary Results
• Circuit: I/O traces from a 2-input 2-output arbiter.
• Top ranked patterns:
1
0
0
1
“When r1 is high
and there is no
competing r0, g1
is high at the
same cycle.
Arbiter r1
g0
g1
r0
g1
g0
r1
r0
0
1
1
0
0
0
0
0
![Page 63: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/63.jpg)
74
• What are humans good at?
– Visual recognition?
Most frequently identified common patterns correspond to desired behaviors of the circuit.
– Randomness?
165 different patterns out of 283 hits (mostly EECS students) Top rank patterns have counts 31, 16 and 7.
• What problems do we crowdsource?
Problems that require human input and insight, or ones that are hard to formally define.
E.g. specification, diagnosis, repair.
Not purely computationally intractable problems.
Related Work: FunSAT/Human EDA [DeOrio and Bertacco, DAC 2009]
Discussion
![Page 64: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/64.jpg)
Mapping Natural Language to Temporal Logic
Human Inputs:
75
![Page 65: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/65.jpg)
76
Natural Language → LTL Specification
Intermediate
Format
Formula
Generation
Type Dependency
Parser (STDP)
Requirements
in NL LTL
Formulas
Result highlights:
• FAA-Isolette requirements from NL to LTL.
• Assumption mining discovered a missing assumption.
Source: D. L. Lempia and S. P. Miller. Requirements engineering management handbook.
Final Report DOT/FAA/AR-08/32, Federal Aviation Administration, June 2009.
![Page 66: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/66.jpg)
77
Conclusion
Formal specifications can be mined in a systematic way to
improve the effectiveness of verification and synthesis.
![Page 67: Wenchao Li - Ptolemy Project · Specification Mining User Event Engine Definitions Library of Temporal Patterns Ranking Module Mined Assertions Most Relevant Assertions Mining Temporal](https://reader035.fdocuments.net/reader035/viewer/2022063000/5f101e9c7e708231d4478ae6/html5/thumbnails/67.jpg)
78
Future Work
• Combine automata-based and sparse coding-based approaches
for mining specifications.
• Improve the scalability of the sparse coding-based approach.
• Mining assumptions in contract-based synthesis.
• Evaluate human-in-the-loop controller synthesis in real setting.
• Human studies of CrowdMine for large designs.
• More robust NL→LTL techniques.